| *** mriedem has joined #openstack-infra | 00:00 | |
| mriedem | anyone noticed this? http://logs.openstack.org/04/256304/1/gate/gate-grenade-dsvm-multinode/ee8293a/console.html#_2015-12-12_20_02_51_724 | 00:01 |
|---|---|---|
| mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message:%5C%22bash:%20/opt/stack/new/devstack/functions-common:%20No%20such%20file%20or%20directory%5C%22%20AND%20tags:%5C%22console%5C%22 | 00:01 |
| *** davideagnello has joined #openstack-infra | 00:01 | |
| *** apuimedo has quit IRC | 00:03 | |
| *** apuimedo has joined #openstack-infra | 00:04 | |
| *** davideagnello has quit IRC | 00:06 | |
| *** jerryz has quit IRC | 00:08 | |
| *** salv-orlando has joined #openstack-infra | 00:08 | |
| *** unicell has joined #openstack-infra | 00:09 | |
| *** unicell has quit IRC | 00:09 | |
| *** markvoelker_ has quit IRC | 00:10 | |
| *** apuimedo has quit IRC | 00:14 | |
| *** vilobhmm11 has quit IRC | 00:15 | |
| *** apuimedo has joined #openstack-infra | 00:15 | |
| *** tzn has joined #openstack-infra | 00:16 | |
| *** flepied has joined #openstack-infra | 00:18 | |
| *** apuimedo has quit IRC | 00:20 | |
| *** apuimedo has joined #openstack-infra | 00:21 | |
| *** tzn has quit IRC | 00:21 | |
| *** flepied1 has quit IRC | 00:22 | |
| *** apuimedo has quit IRC | 00:25 | |
| *** apuimedo has joined #openstack-infra | 00:27 | |
| *** sdake has joined #openstack-infra | 00:27 | |
| *** rfolco has joined #openstack-infra | 00:29 | |
| *** apuimedo has quit IRC | 00:32 | |
| *** apuimedo has joined #openstack-infra | 00:33 | |
| *** rfolco has quit IRC | 00:36 | |
| *** gildub has joined #openstack-infra | 00:38 | |
| *** jcoufal has quit IRC | 00:41 | |
| *** apuimedo has quit IRC | 00:43 | |
| *** apuimedo has joined #openstack-infra | 00:44 | |
| *** thorst has joined #openstack-infra | 00:45 | |
| *** chlong has joined #openstack-infra | 00:48 | |
| *** changbl has joined #openstack-infra | 00:49 | |
| *** apuimedo has quit IRC | 00:49 | |
| *** apuimedo has joined #openstack-infra | 00:49 | |
| *** thorst has quit IRC | 00:51 | |
| *** thorst has joined #openstack-infra | 00:51 | |
| *** thorst has quit IRC | 00:52 | |
| *** salv-orl_ has joined #openstack-infra | 00:53 | |
| *** apuimedo has quit IRC | 00:54 | |
| *** apuimedo has joined #openstack-infra | 00:55 | |
| *** salv-orlando has quit IRC | 00:55 | |
| *** apuimedo has quit IRC | 00:59 | |
| *** apuimedo has joined #openstack-infra | 01:00 | |
| *** salv-orl_ has quit IRC | 01:02 | |
| *** otter768 has joined #openstack-infra | 01:03 | |
| *** mmedvede has joined #openstack-infra | 01:05 | |
| *** Qiming has joined #openstack-infra | 01:07 | |
| *** davideagnello has joined #openstack-infra | 01:10 | |
| *** tjones has joined #openstack-infra | 01:10 | |
| *** markvoelker has joined #openstack-infra | 01:10 | |
| *** davideagnello has quit IRC | 01:14 | |
| *** markvoelker has quit IRC | 01:15 | |
| *** unicell has joined #openstack-infra | 01:22 | |
| *** apuimedo has quit IRC | 01:26 | |
| *** tjones has left #openstack-infra | 01:26 | |
| *** apuimedo has joined #openstack-infra | 01:28 | |
| *** apuimedo has quit IRC | 01:32 | |
| *** ParsectiX has joined #openstack-infra | 01:33 | |
| *** ParsectiX has quit IRC | 01:33 | |
| *** apuimedo has joined #openstack-infra | 01:33 | |
| *** ParsectiX has joined #openstack-infra | 01:33 | |
| *** jyuso1 has joined #openstack-infra | 01:38 | |
| *** apuimedo has quit IRC | 01:38 | |
| *** apuimedo has joined #openstack-infra | 01:39 | |
| *** zhenguo has joined #openstack-infra | 01:42 | |
| *** apuimedo has quit IRC | 01:43 | |
| tonyb | Is it possible to filter by branch in ci-watch ? | 01:44 |
| *** apuimedo has joined #openstack-infra | 01:44 | |
| *** dewsday has joined #openstack-infra | 01:45 | |
| *** zhurong has joined #openstack-infra | 01:46 | |
| *** thorst has joined #openstack-infra | 01:50 | |
| *** thorst has quit IRC | 01:50 | |
| *** thorst has joined #openstack-infra | 01:51 | |
| *** apuimedo has quit IRC | 01:51 | |
| *** apuimedo has joined #openstack-infra | 01:52 | |
| *** apuimedo has quit IRC | 01:57 | |
| *** apuimedo has joined #openstack-infra | 01:57 | |
| pabelanger | ianw: around? | 01:59 |
| ianw | pabelanger: yep! | 01:59 |
| *** thorst has quit IRC | 01:59 | |
| pabelanger | ianw: Hey, question about python signal and nodepool | 01:59 |
| pabelanger | trying to do kill -s SIGHUP `pidof nodepoold` but it's not working as expected | 02:00 |
| pabelanger | looking at https://github.com/openstack-infra/nodepool/blob/master/nodepool/cmd/nodepoold.py#L116 | 02:00 |
| pabelanger | don't we need to add os._exit(0) | 02:00 |
| pabelanger | ? | 02:00 |
| pabelanger | cause looking at strace, the signal.pause() never releases | 02:00 |
| pabelanger | I see the pool.stop() | 02:01 |
| pabelanger | that completes as expected | 02:01 |
| ianw | hmm | 02:02 |
| clarkb | pabelanger no, the idea is all other threads are daemon thrsads so you exit cleanly | 02:02 |
| clarkb | except they arent so it continues to run | 02:03 |
| clarkb | and thats not the handler for hup | 02:03 |
| pabelanger | I should note, I am also starting like nodepoold -d | 02:04 |
| clarkb | thats not related, in python daemon threads are threads that dont force interpreter to run if they are running | 02:04 |
| *** apuimedo has quit IRC | 02:04 | |
| pabelanger | so, the only 2 things I see once I send SIGHUP | 02:05 |
| pabelanger | http://pastebin.com/xzQcHx4K | 02:05 |
| clarkb | yes | 02:05 |
| ianw | pabelanger: so yeah, I don't think there is a sighup handler, it's usr1 & usr2 if you're doing a reload | 02:06 |
| *** apuimedo has joined #openstack-infra | 02:06 | |
| ianw | we've had discussions about this before ... was it that we aliased sigint to graceful shutdown? | 02:06 |
| clarkb | ianw ya | 02:06 |
| clarkb | and sigterm to forceful | 02:06 |
| pabelanger | and strace http://pastebin.com/pNMHvZup | 02:06 |
| clarkb | the issue is ehat I said | 02:06 |
| clarkb | we dont need a strace | 02:06 |
| clarkb | if you sigusr2 the log should show you what the offending threads are | 02:07 |
| clarkb | I think hashar may have a patch | 02:07 |
| *** salv-orlando has joined #openstack-infra | 02:07 | |
| clarkb | though looking at it it has a -1 and I am not sure its the correct way to handle it | 02:09 |
| *** otter768 has quit IRC | 02:11 | |
| *** salv-orlando has quit IRC | 02:11 | |
| *** otter768 has joined #openstack-infra | 02:12 | |
| pabelanger | okay, so stick with SIGUSR1 for now? | 02:12 |
| *** apuimedo has quit IRC | 02:12 | |
| clarkb | well hup isnt a thing so avoid it. and reading the comment there sigint is what should be used, sigusr1 is backward compat | 02:13 |
| *** apuimedo has joined #openstack-infra | 02:13 | |
| pabelanger | Oh, wait.. ya, not SIPHUP, sorry. I was sending SIGINT | 02:13 |
| pabelanger | my bad | 02:14 |
| pabelanger | let me try this all again, and see what happens. | 02:14 |
| clarkb | it us broken | 02:14 |
| clarkb | but its a bug in nodepool that needs fixing | 02:14 |
| pabelanger | clarkb: idea where I should be looking? | 02:15 |
| pabelanger | can try and figure it out | 02:15 |
| clarkb | its the daemon thread thing, we have non daemon threads so after you stop the main thrwad nodepool keeps running | 02:16 |
| clarkb | after you sigint you can sigusr2 to get a list of threads and tracebacks in the debug log | 02:16 |
| clarkb | then its a matter of either making them daemon threads or forcefully killing them on a stop | 02:16 |
| pabelanger | there we go: http://pastebin.com/bXJeNn0R | 02:16 |
| clarkb | you might need it to quiesce a but more | 02:17 |
| clarkb | but thats basically it | 02:17 |
| pabelanger | okay, will hack around. | 02:18 |
| *** davideagnello has joined #openstack-infra | 02:18 | |
| *** tzn has joined #openstack-infra | 02:18 | |
| clarkb | (I say that because looks like main thread is still running but the handler should stopnat least that one | 02:19 |
| clarkb | the Thread: NodePool one | 02:19 |
| *** tjones has joined #openstack-infra | 02:21 | |
| *** tjones has left #openstack-infra | 02:21 | |
| *** davideagnello has quit IRC | 02:22 | |
| *** tzn has quit IRC | 02:23 | |
| pabelanger | clarkb: that one cleared after the sleep, missing from dump now | 02:23 |
| *** apuimedo has quit IRC | 02:25 | |
| *** apuimedo has joined #openstack-infra | 02:25 | |
| *** Piet has joined #openstack-infra | 02:26 | |
| kitsuneninetails | Can someone please take a look at https://review.openstack.org/#/c/254489/ It's code review +2'd and is verified +1. It's just waiting the last step. Thanks! | 02:26 |
| *** apuimedo has quit IRC | 02:30 | |
| *** Sukhdev has joined #openstack-infra | 02:31 | |
| *** apuimedo has joined #openstack-infra | 02:31 | |
| *** sabeen1 has joined #openstack-infra | 02:32 | |
| *** apuimedo has quit IRC | 02:36 | |
| *** baoli has joined #openstack-infra | 02:37 | |
| *** apuimedo has joined #openstack-infra | 02:37 | |
| *** apuimedo has quit IRC | 02:47 | |
| *** rguillebert has quit IRC | 02:47 | |
| *** apuimedo has joined #openstack-infra | 02:47 | |
| *** zz_dimtruck is now known as dimtruck | 02:49 | |
| *** otter768 has quit IRC | 02:51 | |
| *** apuimedo has quit IRC | 02:52 | |
| *** apuimedo has joined #openstack-infra | 02:53 | |
| *** otter768 has joined #openstack-infra | 02:56 | |
| *** Piet has quit IRC | 02:57 | |
| *** apuimedo has quit IRC | 02:57 | |
| *** thorst has joined #openstack-infra | 02:58 | |
| *** apuimedo has joined #openstack-infra | 02:58 | |
| *** davideagnello has joined #openstack-infra | 02:59 | |
| *** baoli has quit IRC | 03:00 | |
| *** dimtruck is now known as zz_dimtruck | 03:00 | |
| *** thorst has quit IRC | 03:04 | |
| *** davideagnello has quit IRC | 03:04 | |
| *** apuimedo has quit IRC | 03:05 | |
| *** apuimedo has joined #openstack-infra | 03:05 | |
| *** salv-orlando has joined #openstack-infra | 03:09 | |
| *** markvoelker has joined #openstack-infra | 03:11 | |
| *** salv-orlando has quit IRC | 03:14 | |
| *** apuimedo has quit IRC | 03:16 | |
| *** markvoelker has quit IRC | 03:16 | |
| *** apuimedo has joined #openstack-infra | 03:18 | |
| *** baoli has joined #openstack-infra | 03:18 | |
| *** tzn has joined #openstack-infra | 03:19 | |
| *** mriedem has quit IRC | 03:20 | |
| *** apuimedo has quit IRC | 03:22 | |
| *** apuimedo has joined #openstack-infra | 03:23 | |
| *** tzn has quit IRC | 03:24 | |
| *** apuimedo has quit IRC | 03:28 | |
| *** apuimedo has joined #openstack-infra | 03:29 | |
| *** otter768 has quit IRC | 03:33 | |
| *** apuimedo has quit IRC | 03:33 | |
| *** MarkAtwood has quit IRC | 03:33 | |
| *** ParsectiX has quit IRC | 03:33 | |
| *** otter768 has joined #openstack-infra | 03:34 | |
| *** apuimedo has joined #openstack-infra | 03:34 | |
| *** zz_dimtruck is now known as dimtruck | 03:35 | |
| pabelanger | ianw: okay, I think I go it but I still think I see an issue | 03:35 |
| pabelanger | isn't https://github.com/openstack-infra/nodepool/blob/master/nodepool/cmd/nodepoold.py#L137 wrong? | 03:35 |
| pabelanger | meaning, once a signal is received, it will continue, looping back to signal.pause() for ever? | 03:36 |
| pabelanger | and never breaking the main thread? | 03:36 |
| pabelanger | unless SIGTERM is called | 03:36 |
| pabelanger | maybe that is the trick | 03:36 |
| pabelanger | I need to call both | 03:36 |
| pabelanger | SIGINT and SIGTERM | 03:36 |
| pabelanger | not just SIGINT | 03:36 |
| *** tsg has joined #openstack-infra | 03:37 | |
| *** dewsday has quit IRC | 03:37 | |
| *** apuimedo has quit IRC | 03:38 | |
| *** apuimedo has joined #openstack-infra | 03:39 | |
| *** Liuqing has joined #openstack-infra | 03:39 | |
| *** apuimedo has quit IRC | 03:43 | |
| *** apuimedo has joined #openstack-infra | 03:45 | |
| *** Sukhdev has quit IRC | 03:47 | |
| *** cfarquhar has quit IRC | 03:50 | |
| *** jianghuaw has joined #openstack-infra | 03:51 | |
| *** apuimedo has quit IRC | 03:52 | |
| *** apuimedo has joined #openstack-infra | 03:53 | |
| *** yamamoto has joined #openstack-infra | 03:55 | |
| *** unicell1 has joined #openstack-infra | 04:00 | |
| *** unicell has quit IRC | 04:01 | |
| *** regXboi has joined #openstack-infra | 04:01 | |
| *** thorst has joined #openstack-infra | 04:02 | |
| *** kambiz has quit IRC | 04:02 | |
| *** lyz has quit IRC | 04:02 | |
| *** grantbow has quit IRC | 04:02 | |
| *** apuimedo has quit IRC | 04:04 | |
| *** apuimedo has joined #openstack-infra | 04:04 | |
| *** regXboi has quit IRC | 04:06 | |
| *** davideagnello has joined #openstack-infra | 04:07 | |
| *** sdake has quit IRC | 04:08 | |
| *** sdake_ has joined #openstack-infra | 04:08 | |
| *** vilobhmm11 has joined #openstack-infra | 04:09 | |
| *** thorst has quit IRC | 04:09 | |
| *** vilobhmm111 has joined #openstack-infra | 04:11 | |
| *** davideagnello has quit IRC | 04:11 | |
| *** wolsen_ is now known as wolsen | 04:12 | |
| *** Guest45133 is now known as jgriffith | 04:12 | |
| *** otter768 has quit IRC | 04:12 | |
| *** vilobhmm11 has quit IRC | 04:13 | |
| *** apuimedo has quit IRC | 04:14 | |
| *** MarkAtwood has joined #openstack-infra | 04:14 | |
| *** apuimedo has joined #openstack-infra | 04:15 | |
| *** sdake_ has quit IRC | 04:15 | |
| *** baoli has quit IRC | 04:18 | |
| *** mahatic has quit IRC | 04:18 | |
| *** cfarquhar has joined #openstack-infra | 04:19 | |
| *** cfarquhar has quit IRC | 04:19 | |
| *** cfarquhar has joined #openstack-infra | 04:19 | |
| *** apuimedo has quit IRC | 04:19 | |
| *** tzn has joined #openstack-infra | 04:20 | |
| *** dims has joined #openstack-infra | 04:20 | |
| *** apuimedo has joined #openstack-infra | 04:21 | |
| *** cfarquhar has quit IRC | 04:22 | |
| *** tzn has quit IRC | 04:24 | |
| *** apuimedo has quit IRC | 04:25 | |
| *** apuimedo has joined #openstack-infra | 04:26 | |
| *** Qiming has quit IRC | 04:29 | |
| *** fawadkhaliq has joined #openstack-infra | 04:29 | |
| *** cfarquhar has joined #openstack-infra | 04:29 | |
| *** cfarquhar has quit IRC | 04:29 | |
| *** cfarquhar has joined #openstack-infra | 04:29 | |
| *** unicell1 has quit IRC | 04:30 | |
| *** unicell has joined #openstack-infra | 04:31 | |
| *** flepied1 has joined #openstack-infra | 04:32 | |
| *** ramineni has joined #openstack-infra | 04:34 | |
| *** flepied has quit IRC | 04:35 | |
| *** sabeen1 has quit IRC | 04:35 | |
| *** Sukhdev has joined #openstack-infra | 04:36 | |
| *** mcarden has quit IRC | 04:36 | |
| *** sabeen has joined #openstack-infra | 04:36 | |
| *** apuimedo has quit IRC | 04:36 | |
| *** erikmwilson has quit IRC | 04:36 | |
| *** mgagne has quit IRC | 04:36 | |
| *** jbryce has quit IRC | 04:36 | |
| *** erikmwilson has joined #openstack-infra | 04:37 | |
| *** apuimedo has joined #openstack-infra | 04:37 | |
| *** mcarden has joined #openstack-infra | 04:37 | |
| *** jbryce has joined #openstack-infra | 04:37 | |
| *** mgagne has joined #openstack-infra | 04:37 | |
| *** mgagne is now known as Guest73233 | 04:37 | |
| *** apuimedo has quit IRC | 04:41 | |
| *** apuimedo has joined #openstack-infra | 04:43 | |
| *** MarkAtwood has quit IRC | 04:44 | |
| *** apuimedo has quit IRC | 04:48 | |
| *** MarkAtwood has joined #openstack-infra | 04:49 | |
| *** apuimedo has joined #openstack-infra | 04:49 | |
| *** ig0r_ has quit IRC | 04:50 | |
| *** apuimedo has quit IRC | 04:54 | |
| *** ig0r_ has joined #openstack-infra | 04:55 | |
| *** apuimedo has joined #openstack-infra | 04:55 | |
| *** maishsk has joined #openstack-infra | 05:07 | |
| *** thorst has joined #openstack-infra | 05:08 | |
| *** apuimedo has quit IRC | 05:12 | |
| *** markvoelker has joined #openstack-infra | 05:12 | |
| *** apuimedo has joined #openstack-infra | 05:13 | |
| *** tsg has quit IRC | 05:13 | |
| *** thorst has quit IRC | 05:14 | |
| *** davideagnello has joined #openstack-infra | 05:15 | |
| *** markvoelker has quit IRC | 05:17 | |
| *** baoli has joined #openstack-infra | 05:19 | |
| *** davideagnello has quit IRC | 05:20 | |
| *** maishsk has quit IRC | 05:20 | |
| *** tzn has joined #openstack-infra | 05:21 | |
| *** Qiming has joined #openstack-infra | 05:21 | |
| *** apuimedo has quit IRC | 05:22 | |
| *** apuimedo has joined #openstack-infra | 05:24 | |
| *** tzn has quit IRC | 05:25 | |
| *** maishsk has joined #openstack-infra | 05:26 | |
| *** sabeen3 has joined #openstack-infra | 05:27 | |
| stevemar | is hound not called hound any longer? | 05:28 |
| stevemar | http://hound.openstack.org/ 404s for me | 05:28 |
| *** hdd has joined #openstack-infra | 05:28 | |
| *** sabeen has quit IRC | 05:30 | |
| *** maishsk has quit IRC | 05:31 | |
| *** MarkAtwood has quit IRC | 05:32 | |
| *** maishsk has joined #openstack-infra | 05:32 | |
| greghaynes | stevemar: I think it is now http://codesearch.openstack.org/ | 05:36 |
| *** zhurong has quit IRC | 05:37 | |
| *** Liuqing has quit IRC | 05:37 | |
| *** abregman has joined #openstack-infra | 05:38 | |
| stevemar | greghaynes: i liked the name 'hound' better :) | 05:38 |
| stevemar | greghaynes: but thank you sir | 05:38 |
| *** maishsk has quit IRC | 05:40 | |
| *** kushal has joined #openstack-infra | 05:43 | |
| *** mrmartin has joined #openstack-infra | 05:44 | |
| *** apuimedo has quit IRC | 05:44 | |
| *** apuimedo has joined #openstack-infra | 05:45 | |
| *** maishsk has joined #openstack-infra | 05:45 | |
| nibalizer | hound is the software and codesearch is the service | 05:47 |
| *** Qiming has quit IRC | 05:47 | |
| *** ParsectiX has joined #openstack-infra | 05:47 | |
| *** ParsectiX has quit IRC | 05:48 | |
| *** ParsectiX has joined #openstack-infra | 05:48 | |
| *** Liuqing has joined #openstack-infra | 05:49 | |
| *** apuimedo has quit IRC | 05:49 | |
| *** apuimedo has joined #openstack-infra | 05:51 | |
| *** maishsk has quit IRC | 05:52 | |
| *** dims has quit IRC | 05:53 | |
| *** melwitt has quit IRC | 05:55 | |
| *** dimtruck is now known as zz_dimtruck | 05:55 | |
| *** apuimedo has quit IRC | 05:56 | |
| *** apuimedo has joined #openstack-infra | 05:56 | |
| *** sabeen3 has quit IRC | 05:57 | |
| *** apuimedo has quit IRC | 06:01 | |
| *** apuimedo has joined #openstack-infra | 06:02 | |
| *** fabo_ is now known as fabo | 06:04 | |
| *** fabo has quit IRC | 06:04 | |
| *** fabo has joined #openstack-infra | 06:04 | |
| *** binbincong has joined #openstack-infra | 06:06 | |
| *** Ramanjaneya has joined #openstack-infra | 06:08 | |
| *** apuimedo has quit IRC | 06:09 | |
| *** melwitt has joined #openstack-infra | 06:10 | |
| *** apuimedo has joined #openstack-infra | 06:10 | |
| *** yuanying has joined #openstack-infra | 06:11 | |
| *** ig0r_ has quit IRC | 06:12 | |
| *** thorst has joined #openstack-infra | 06:13 | |
| *** otter768 has joined #openstack-infra | 06:13 | |
| *** mrmartin has quit IRC | 06:15 | |
| *** hdd has quit IRC | 06:16 | |
| *** otter768 has quit IRC | 06:18 | |
| *** thorst has quit IRC | 06:19 | |
| *** apuimedo has quit IRC | 06:20 | |
| *** ParsectiX has quit IRC | 06:20 | |
| *** kitsune9tails has joined #openstack-infra | 06:20 | |
| *** kitsuneninetails has quit IRC | 06:21 | |
| *** apuimedo has joined #openstack-infra | 06:21 | |
| *** tzn has joined #openstack-infra | 06:21 | |
| *** tzn has quit IRC | 06:26 | |
| *** apuimedo has quit IRC | 06:26 | |
| *** apuimedo has joined #openstack-infra | 06:26 | |
| *** aopincar has joined #openstack-infra | 06:28 | |
| *** _nadya_ has joined #openstack-infra | 06:30 | |
| *** apuimedo has quit IRC | 06:31 | |
| *** amotoki has joined #openstack-infra | 06:31 | |
| *** apuimedo has joined #openstack-infra | 06:31 | |
| *** ParsectiX has joined #openstack-infra | 06:32 | |
| *** ParsectiX has quit IRC | 06:32 | |
| *** ParsectiX has joined #openstack-infra | 06:32 | |
| *** apuimedo has quit IRC | 06:36 | |
| *** davideagnello has joined #openstack-infra | 06:37 | |
| *** apuimedo has joined #openstack-infra | 06:38 | |
| *** salv-orlando has joined #openstack-infra | 06:39 | |
| *** AJaeger has joined #openstack-infra | 06:39 | |
| AJaeger | lifeless: could you review dims' duplicate detection change again, please? https://review.openstack.org/#/c/238871/ | 06:40 |
| *** davideagnello has quit IRC | 06:42 | |
| *** salv-orlando has quit IRC | 06:43 | |
| *** apuimedo has quit IRC | 06:45 | |
| *** apuimedo has joined #openstack-infra | 06:45 | |
| *** gildub has quit IRC | 06:49 | |
| *** MarkAtwood has joined #openstack-infra | 06:49 | |
| kitsune9tails | ``I will change job | 06:51 |
| kitsune9tails | oops, nvm, wrong channel :) | 06:51 |
| *** apuimedo has quit IRC | 06:55 | |
| *** zhurong has joined #openstack-infra | 06:55 | |
| *** apuimedo has joined #openstack-infra | 06:56 | |
| AJaeger | infra-root, it seems that we lost our bots, I see no bots here. could you bring them back, please? | 06:58 |
| jhesketh | AJaeger: I'll take a looksee | 06:59 |
| AJaeger | thanks, jhesketh | 06:59 |
| *** mjb has quit IRC | 07:00 | |
| *** apuimedo has quit IRC | 07:01 | |
| *** apuimedo has joined #openstack-infra | 07:02 | |
| *** rcernin has joined #openstack-infra | 07:02 | |
| *** mjb has joined #openstack-infra | 07:05 | |
| *** maishsk has joined #openstack-infra | 07:05 | |
| *** openstackstatus has joined #openstack-infra | 07:05 | |
| *** ChanServ sets mode: +v openstackstatus | 07:05 | |
| *** apuimedo has quit IRC | 07:06 | |
| *** vilobhmm111 has quit IRC | 07:06 | |
| jhesketh | AJaeger: so that statusbot had its connections closed... a restart seems to have brought that back | 07:06 |
| jhesketh | AJaeger: and the meetbot was still here | 07:07 |
| jhesketh | need to track down the gerritbotr | 07:07 |
| *** apuimedo has joined #openstack-infra | 07:07 | |
| AJaeger | I didn't check the meetbot ;/, only missed the other two ;) | 07:07 |
| AJaeger | it was too silent everywhere ;) | 07:08 |
| *** boris-42_ has joined #openstack-infra | 07:09 | |
| jhesketh | AJaeger: looks like the gerritbot is not connected and it's reconnect method not implemented http://paste.openstack.org/show/481770/ | 07:09 |
| jhesketh | I'll try restarting | 07:09 |
| *** openstackgerrit has joined #openstack-infra | 07:10 | |
| jhesketh | hello openstackgerrit | 07:10 |
| *** salv-orlando has joined #openstack-infra | 07:11 | |
| AJaeger | welcome, openstackgerrit ! | 07:12 |
| *** jamielennox is now known as jamielennox|away | 07:12 | |
| *** nmagnezi has joined #openstack-infra | 07:13 | |
| *** miqui has quit IRC | 07:13 | |
| *** markvoelker has joined #openstack-infra | 07:13 | |
| AJaeger | and I see first notifications again, thanks jhesketh ! | 07:16 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: yum-minimal: Don't create eth0 & eth1 https://review.openstack.org/257172 | 07:16 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: yum-minimal: leave behind dummy /etc/resolv.conf https://review.openstack.org/257173 | 07:16 |
| *** thorst has joined #openstack-infra | 07:17 | |
| *** davideagnello has joined #openstack-infra | 07:17 | |
| *** markvoelker has quit IRC | 07:17 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/gerritbot: Fix reconnection method https://review.openstack.org/257174 | 07:18 |
| jhesketh | AJaeger: and the bot picked up the fix to itself ^ | 07:18 |
| AJaeger | ;) | 07:18 |
| *** apuimedo has quit IRC | 07:19 | |
| *** apuimedo has joined #openstack-infra | 07:20 | |
| *** tzn has joined #openstack-infra | 07:22 | |
| *** davideagnello has quit IRC | 07:23 | |
| *** thorst has quit IRC | 07:24 | |
| *** apuimedo has quit IRC | 07:25 | |
| *** apuimedo has joined #openstack-infra | 07:25 | |
| *** tzn has quit IRC | 07:27 | |
| *** apuimedo has quit IRC | 07:32 | |
| *** apuimedo has joined #openstack-infra | 07:33 | |
| *** egonzalez has joined #openstack-infra | 07:34 | |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: Add check-requirements to i18n project https://review.openstack.org/257123 | 07:35 |
| *** Fillip_k has joined #openstack-infra | 07:36 | |
| Fillip_k | hi all | 07:36 |
| *** apuimedo has quit IRC | 07:38 | |
| *** lyz has joined #openstack-infra | 07:38 | |
| *** dims has joined #openstack-infra | 07:39 | |
| *** scheuran has joined #openstack-infra | 07:39 | |
| *** apuimedo has joined #openstack-infra | 07:39 | |
| *** Daisy has joined #openstack-infra | 07:39 | |
| lifeless | AJaeger: sure | 07:39 |
| *** kitsune9tails is now known as kitsuneninetails | 07:43 | |
| *** egonzalez has quit IRC | 07:44 | |
| *** chlong has quit IRC | 07:46 | |
| AJaeger | Nakato: could you review some constraints jobs, please? https://review.openstack.org/#/c/255992/ and https://review.openstack.org/#/c/256391/ https://review.openstack.org/256163 https://review.openstack.org/256702 . I gave already a few -1s but would like you to check these as well... | 07:46 |
| stevemar | looks like httpretty broke things | 07:47 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Revert "Experimental grenade jobs for oslo.messaging" https://review.openstack.org/257192 | 07:47 |
| *** sdake has joined #openstack-infra | 07:48 | |
| *** apuimedo has quit IRC | 07:48 | |
| *** apuimedo has joined #openstack-infra | 07:50 | |
| AJaeger | stevemar: it's since over 6 months not in requirements, there aren't many repos using it... | 07:50 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Grenade jobs for testing libraries (esp. oslo.messaging) https://review.openstack.org/256467 | 07:51 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Revert "Experimental grenade jobs for oslo.messaging" https://review.openstack.org/257192 | 07:51 |
| dims | AJaeger : i lined up the revert and the new job in sequence. thanks! | 07:51 |
| stevemar | AJaeger: https://github.com/openstack/ospurge/blob/master/test-requirements.txt | 07:51 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: Add check-requirements to i18n project https://review.openstack.org/257123 | 07:52 |
| stevemar | AJaeger: ospurge seems to be using | 07:53 |
| stevemar | as well as: http://codesearch.openstack.org/?q=httpretty&i=nope&files=&repos= | 07:55 |
| *** irenab has joined #openstack-infra | 07:55 | |
| AJaeger | dims: great, worthy a double +2 ;) | 07:55 |
| *** MarkAtwood has quit IRC | 07:55 | |
| AJaeger | stevemar: as I said: not many;) | 07:55 |
| irenab | hello! Would appreciate the guidance on grenade-dvsm-neutron failure for this patch: https://review.openstack.org/#/c/253928/ | 07:56 |
| *** ihrachys has joined #openstack-infra | 07:56 | |
| stevemar | AJaeger: looks like ospurge, python-group-based-policy-client, cloudcafe, watcherclient and cloud-init | 07:56 |
| AJaeger | stevemar: it's also interesting to read some of the comments about httpretty | 07:56 |
| AJaeger | "Not all packages properly build wheels (httpretty for example" | 07:57 |
| stevemar | nod | 07:57 |
| stevemar | "Convert httpretty tests to requests-mock" | 07:57 |
| AJaeger | ""httpretty doesn't work as expected..." | 07:57 |
| *** apuimedo has quit IRC | 07:57 | |
| AJaeger | yep... | 07:57 |
| dims | AJaeger : haha | 07:57 |
| stevemar | "Removed httpretty as suggested in other projects" | 07:57 |
| *** ramineni has quit IRC | 07:57 | |
| stevemar | the SDK folks just commented out a bunch of tests :) | 07:57 |
| *** apuimedo has joined #openstack-infra | 07:58 | |
| AJaeger | not the best way to do this... | 07:58 |
| *** davideagnello has joined #openstack-infra | 07:59 | |
| *** dtardivel has joined #openstack-infra | 07:59 | |
| stevemar | AJaeger: some projects haven't had a successful build in almost 3 months: https://review.openstack.org/#/q/status:open+project:openstack/tuskar,n,z | 07:59 |
| stevemar | big tent ftw! | 08:00 |
| *** bexelbie_ has joined #openstack-infra | 08:00 | |
| *** Daisy has quit IRC | 08:00 | |
| *** bexelbie_ is now known as bexelbie | 08:01 | |
| stevemar | i dont understand the point of creating a project if it's not going to be maintained :( | 08:01 |
| *** jcoufal has joined #openstack-infra | 08:01 | |
| irenab | ihrachys: hi | 08:02 |
| *** ihrachys has quit IRC | 08:02 | |
| *** davideagnello has quit IRC | 08:03 | |
| AJaeger | stevemar: want some more fun? Check https://review.openstack.org/#/q/status:open++topic:stackforge-retirement,n,z | 08:04 |
| *** egonzalez has joined #openstack-infra | 08:04 | |
| AJaeger | Those are all projects that wanted to move from stackforge to openstack namespace - but didn't fix their gitreview file yet ;( | 08:04 |
| AJaeger | For some the change did not even build ;( | 08:05 |
| stevemar | oh gosh, there's "next" | 08:05 |
| AJaeger | my email provoked 5 merges so far - all to fuel-main | 08:06 |
| AJaeger | http://lists.openstack.org/pipermail/openstack-dev/2015-December/082134.html | 08:06 |
| AJaeger | stevemar: yeah ;( | 08:06 |
| stevemar | oh nice, i wanted to update "sticks" and i think it's in your affects projects AJaeger | 08:06 |
| *** _nadya_ has quit IRC | 08:06 | |
| stevemar | yep... | 08:07 |
| stevemar | i just tried to push a change | 08:07 |
| stevemar | so i'm trying not to break people in the next release of KSC | 08:07 |
| *** apuimedo has quit IRC | 08:07 | |
| stevemar | but they are making it very difficult since their project doesn't build | 08:07 |
| stevemar | and i can't fix one small change for them | 08:07 |
| AJaeger | Last change in August ;( | 08:08 |
| *** apuimedo has joined #openstack-infra | 08:08 | |
| *** ramineni has joined #openstack-infra | 08:08 | |
| *** MarkAtwood has joined #openstack-infra | 08:09 | |
| *** arxcruz has joined #openstack-infra | 08:09 | |
| stevemar | AJaeger: what's the point of: https://github.com/openstack/group-based-policy/blob/master/gbpservice/neutron/tests/unit/services/grouppolicy/test_resource_mapping.py#L17 | 08:11 |
| nmagnezi | hi guys, any known issues in CI currently? I'm trying to get proper tests for https://review.openstack.org/#/c/162260/27 and each time another gate fails for reasons which are not related to my patch (gates fail to run etc) | 08:11 |
| *** salv-orlando has quit IRC | 08:12 | |
| AJaeger | stevemar: no idea ;( | 08:12 |
| stevemar | AJaeger: their requirements file is nearly empty | 08:13 |
| stevemar | AJaeger: how do you do this?! | 08:13 |
| stevemar | i'm frustrated after 10 mintues | 08:13 |
| AJaeger | there're areas I'm not going to touch ;) | 08:13 |
| gsagie | I thank god every day for sending us AJaeger :) | 08:13 |
| AJaeger | LOL | 08:14 |
| AJaeger | thanks, gsagie | 08:14 |
| AJaeger | nmagnezi: no current known issues | 08:14 |
| *** otter768 has joined #openstack-infra | 08:14 | |
| *** apuimedo has quit IRC | 08:15 | |
| AJaeger | nmagnezi: you better dig into the log files and see whether it's a known issue and if not, open a bug | 08:15 |
| nmagnezi | AJaeger, https://jenkins01.openstack.org/job/gate-grenade-dsvm-neutron/2828/console https://jenkins01.openstack.org/job/gate-grenade-dsvm-neutron/2828/console | 08:15 |
| *** apuimedo has joined #openstack-infra | 08:16 | |
| nmagnezi | AJaeger, looks fishy :-) | 08:16 |
| *** markus_z has joined #openstack-infra | 08:16 | |
| irenab | nmagnezi: I also have the problem with grenade-dvsm-neutron job | 08:18 |
| *** gchamoul- is now known as gchamoul | 08:18 | |
| AJaeger | nmagnezi: I'm not an expert for these jobs and can't help further - but there's nothing obvious infra related that I can see. If you see something, please give a detailed reference and let's see whether others can help. | 08:19 |
| *** Fillip_k has quit IRC | 08:19 | |
| *** otter768 has quit IRC | 08:19 | |
| stevemar | AJaeger: you'll love this one | 08:19 |
| AJaeger | yeah, the last 8 jobs failed on jenkins01 ;( | 08:19 |
| stevemar | steve@/opt/stack/distil$ tox -e pep8 -> ERROR: toxini file 'tox.ini' not found | 08:20 |
| stevemar | i guess they don't use gerrit? | 08:20 |
| stevemar | their git review is wrong :( Trying again with ssh://stevemar@gerrit.dmz.wgtn.cat-it.co.nz:29418/openstack-artifice.git | 08:21 |
| AJaeger | stevemar: gate-distil-pep8 is voting ;( | 08:21 |
| stevemar | this has gotta be my saddest day in openstack | 08:21 |
| *** mpavone has joined #openstack-infra | 08:22 | |
| *** mpavone has quit IRC | 08:22 | |
| *** thorst has joined #openstack-infra | 08:22 | |
| AJaeger | stevemar: send an email flwang@catalyst.net.nz and CC openstack-dev ;) | 08:22 |
| nmagnezi | AJaeger, how about + echo 'ERROR: the main setup script run by this job failed - exit code: 1' | 08:22 |
| AJaeger | nmagnezi: so, investigate the log files for the main setup script... | 08:23 |
| AJaeger | nmagnezi: that's a normal error if one of the tests fails | 08:23 |
| *** tzn has joined #openstack-infra | 08:23 | |
| AJaeger | stevemar: do you have some handy gate debugging references for nmagnezi before it gets a sad day for me as well ;) | 08:24 |
| *** unicell1 has joined #openstack-infra | 08:25 | |
| AJaeger | stevemar: we should consider decommissing distil ;(, so ask flwang whether they still need a repo that was created a year ago and has no single change since then ;( | 08:25 |
| *** shardy has joined #openstack-infra | 08:25 | |
| *** unicell has quit IRC | 08:26 | |
| *** apuimedo has quit IRC | 08:27 | |
| *** tzn has quit IRC | 08:27 | |
| *** apuimedo has joined #openstack-infra | 08:29 | |
| *** thorst has quit IRC | 08:30 | |
| *** sdake has quit IRC | 08:30 | |
| *** fawadkhaliq has quit IRC | 08:30 | |
| *** fawadkhaliq has joined #openstack-infra | 08:31 | |
| *** fawadkhaliq has quit IRC | 08:31 | |
| *** bexelbie has quit IRC | 08:31 | |
| *** fawadkhaliq has joined #openstack-infra | 08:32 | |
| *** hichihara has quit IRC | 08:32 | |
| *** Fillip_k has joined #openstack-infra | 08:32 | |
| *** apuimedo has quit IRC | 08:33 | |
| *** apuimedo has joined #openstack-infra | 08:34 | |
| AJaeger | igorbelikov: could you check http://lists.openstack.org/pipermail/openstack-dev/2015-December/082134.html and https://review.openstack.org/#/q/status:open++topic:stackforge-retirement+project:%255Eopenstack/.*fuel.*,n,z and inform the fuel folks about it, please? | 08:34 |
| AJaeger | stevemar: have you given above on distil or what are you going to do? | 08:36 |
| stevemar | AJaeger: you mean have i sent the note yet? | 08:37 |
| stevemar | errr email yet? | 08:37 |
| *** matrohon has joined #openstack-infra | 08:37 | |
| *** MarkAtwood has quit IRC | 08:37 | |
| *** irenab has left #openstack-infra | 08:38 | |
| AJaeger | I wanted to say "given up on" ;( | 08:38 |
| AJaeger | sad day, I can't write anymore. | 08:38 |
| *** irenab has joined #openstack-infra | 08:38 | |
| AJaeger | yeah, wondering what you're going to do... | 08:39 |
| *** Sukhdev has quit IRC | 08:39 | |
| stevemar | hehe, i was in the process of gathering data about distil and then got distracted by a simple OSC bug that i could fix | 08:40 |
| stevemar | https://bugs.launchpad.net/python-openstackclient/+bug/1525805 | 08:40 |
| openstack | Launchpad bug 1525805 in python-openstackclient "container show does not filter meta properties properly" [Medium,In progress] - Assigned to Steve Martinelli (stevemar) | 08:40 |
| stevemar | i wanted an easy win! | 08:40 |
| AJaeger | ;) | 08:40 |
| AJaeger | let me not stop you from that ;) | 08:40 |
| stevemar | oh its done now | 08:40 |
| stevemar | only took 10 mintues :P | 08:40 |
| *** jlanoux has joined #openstack-infra | 08:41 | |
| *** sdake has joined #openstack-infra | 08:41 | |
| *** binbincong_ has joined #openstack-infra | 08:43 | |
| *** fhubik has joined #openstack-infra | 08:43 | |
| AJaeger | great | 08:43 |
| AJaeger | a fix less than an hour after reporting, great | 08:44 |
| *** ifarkas has joined #openstack-infra | 08:45 | |
| *** jordanP has joined #openstack-infra | 08:46 | |
| *** apuimedo has quit IRC | 08:46 | |
| *** apuimedo has joined #openstack-infra | 08:47 | |
| *** bmwiedemann has left #openstack-infra | 08:47 | |
| *** fhubik is now known as fhubik_brb | 08:48 | |
| *** sdake has quit IRC | 08:49 | |
| *** fawadkhaliq has quit IRC | 08:49 | |
| *** fawadkhaliq has joined #openstack-infra | 08:49 | |
| *** Fillip_k has quit IRC | 08:50 | |
| *** apuimedo has quit IRC | 08:51 | |
| *** shardy has quit IRC | 08:52 | |
| *** apuimedo has joined #openstack-infra | 08:53 | |
| *** fhubik_brb is now known as fhubik | 08:53 | |
| *** egonzalez has quit IRC | 08:54 | |
| *** sdake has joined #openstack-infra | 08:54 | |
| *** tangchen has joined #openstack-infra | 08:54 | |
| tangchen | Hi all, Seems that http://stackalytics.com/ stopped working since Dec 11. | 08:55 |
| tangchen | The data is not the latest. | 08:55 |
| *** samuelBartel has joined #openstack-infra | 08:55 | |
| *** Daisy has joined #openstack-infra | 08:56 | |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack/requirements: Bump oslo.utils to 3.2.0 https://review.openstack.org/257225 | 08:57 |
| *** apuimedo has quit IRC | 08:59 | |
| *** fhubik is now known as fhubik_brb | 09:00 | |
| *** binbincong_ has quit IRC | 09:00 | |
| *** binbincong has quit IRC | 09:00 | |
| AJaeger | tangchen: somebody from mirantis maintains that, not sure what's the best way to contact them. ;( | 09:00 |
| *** apuimedo has joined #openstack-infra | 09:00 | |
| SergeyLukjanov | tangchen, AJaeger, hey folks, I'll ping the maintainers | 09:01 |
| stevemar | AJaeger: i sent something | 09:01 |
| *** Daisy has quit IRC | 09:01 | |
| AJaeger | thanks, SergeyLukjanov and stevemar ! | 09:04 |
| *** lezbar has joined #openstack-infra | 09:04 | |
| *** marekd has joined #openstack-infra | 09:05 | |
| *** ociuhandu has quit IRC | 09:05 | |
| *** dingyichen has quit IRC | 09:06 | |
| *** fhubik_brb is now known as fhubik | 09:06 | |
| *** ParsectiX has quit IRC | 09:07 | |
| *** davideagnello has joined #openstack-infra | 09:07 | |
| *** zeih has joined #openstack-infra | 09:08 | |
| *** sshnaidm has joined #openstack-infra | 09:09 | |
| *** apuimedo has quit IRC | 09:10 | |
| *** rossella_s has joined #openstack-infra | 09:10 | |
| *** apuimedo has joined #openstack-infra | 09:11 | |
| *** davideagnello has quit IRC | 09:12 | |
| *** jistr has joined #openstack-infra | 09:12 | |
| *** binbincong_ has joined #openstack-infra | 09:12 | |
| *** Fillip_k has joined #openstack-infra | 09:13 | |
| *** binbincong has joined #openstack-infra | 09:13 | |
| *** markvoelker has joined #openstack-infra | 09:14 | |
| *** jaypipes has joined #openstack-infra | 09:15 | |
| *** apuimedo has quit IRC | 09:15 | |
| *** apuimedo has joined #openstack-infra | 09:16 | |
| *** rm_you has joined #openstack-infra | 09:17 | |
| *** markvoelker has quit IRC | 09:18 | |
| *** shardy has joined #openstack-infra | 09:19 | |
| *** mrmartin has joined #openstack-infra | 09:19 | |
| *** mrmartin has quit IRC | 09:20 | |
| *** tzn has joined #openstack-infra | 09:24 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: Enable network.service with systemd https://review.openstack.org/257230 | 09:25 |
| *** ociuhandu has joined #openstack-infra | 09:26 | |
| *** Guest58914 is now known as BobBall | 09:26 | |
| *** fhubik is now known as fhubik_brb | 09:27 | |
| *** ilyashakhat has quit IRC | 09:27 | |
| *** thorst has joined #openstack-infra | 09:27 | |
| *** tzn has quit IRC | 09:28 | |
| *** apuimedo has left #openstack-infra | 09:28 | |
| *** Hal has joined #openstack-infra | 09:28 | |
| *** Hal is now known as Guest65746 | 09:29 | |
| *** admin0 has joined #openstack-infra | 09:30 | |
| *** xylan_kong is now known as lane_kong | 09:30 | |
| openstackgerrit | Vincent Françoise proposed openstack-infra/project-config: i18n - Add translation jobs https://review.openstack.org/257236 | 09:30 |
| *** amotoki has quit IRC | 09:31 | |
| *** yamahata has quit IRC | 09:32 | |
| *** jianghuaw has quit IRC | 09:33 | |
| *** tzn has joined #openstack-infra | 09:33 | |
| *** thorst has quit IRC | 09:34 | |
| *** vgridnev has joined #openstack-infra | 09:38 | |
| *** ig0r_ has joined #openstack-infra | 09:39 | |
| *** acruz_ has joined #openstack-infra | 09:41 | |
| *** acruz_ has quit IRC | 09:41 | |
| *** _skolekonov is now known as skolekonov | 09:41 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config: Remove OVERRIDE_ENABLED_SERVICES from neutron yaml files https://review.openstack.org/254441 | 09:43 |
| *** fhubik_brb is now known as fhubik | 09:46 | |
| *** salv-orlando has joined #openstack-infra | 09:47 | |
| *** davideagnello has joined #openstack-infra | 09:48 | |
| *** zeljkof has left #openstack-infra | 09:49 | |
| *** davideagnello has quit IRC | 09:52 | |
| *** placko_ has joined #openstack-infra | 09:53 | |
| *** bharathm has quit IRC | 09:53 | |
| *** placko_ has quit IRC | 09:53 | |
| *** bharathm has joined #openstack-infra | 09:54 | |
| *** cdent has joined #openstack-infra | 09:54 | |
| *** dizquierdo has joined #openstack-infra | 09:54 | |
| *** zhurong has quit IRC | 09:55 | |
| *** vgridnev has quit IRC | 09:55 | |
| *** vgridnev has joined #openstack-infra | 09:56 | |
| *** Liuqing has quit IRC | 09:57 | |
| openstackgerrit | Shuquan Huang proposed openstack-infra/zuul: Replace assertEqual(None, *) with assertIsNone in tests https://review.openstack.org/257251 | 09:57 |
| *** _nadya_ has joined #openstack-infra | 09:58 | |
| *** rguillebert has joined #openstack-infra | 10:00 | |
| *** MarkAtwood has joined #openstack-infra | 10:00 | |
| *** dtantsur|afk is now known as dtantsur | 10:00 | |
| openstackgerrit | Evgeny Antyshev proposed openstack-infra/zuul: Warning if .ssh_wrapper not found https://review.openstack.org/255956 | 10:02 |
| *** zhenguo has quit IRC | 10:04 | |
| *** MarkAtwood has quit IRC | 10:05 | |
| *** fawadkhaliq has quit IRC | 10:05 | |
| *** jtomasek has joined #openstack-infra | 10:06 | |
| openstackgerrit | Szymon Wróblewski proposed openstack/requirements: Bump tooz version to 1.28.0 https://review.openstack.org/254748 | 10:06 |
| *** ifarkas has quit IRC | 10:06 | |
| *** bharathm has quit IRC | 10:10 | |
| *** claudiub has joined #openstack-infra | 10:12 | |
| *** maishsk has quit IRC | 10:14 | |
| *** maishsk has joined #openstack-infra | 10:15 | |
| *** otter768 has joined #openstack-infra | 10:15 | |
| *** christiana_ has joined #openstack-infra | 10:17 | |
| *** electrofelix has joined #openstack-infra | 10:19 | |
| *** otter768 has quit IRC | 10:19 | |
| *** zeus` has quit IRC | 10:23 | |
| openstackgerrit | Merged openstack-infra/storyboard: Deprecated tox -downloadcache option removed https://review.openstack.org/256757 | 10:23 |
| *** zeih has quit IRC | 10:23 | |
| *** fawadkhaliq has joined #openstack-infra | 10:25 | |
| *** _nadya_ has quit IRC | 10:25 | |
| *** rcernin has quit IRC | 10:25 | |
| *** sambetts-afk is now known as sambetts | 10:27 | |
| *** davideagnello has joined #openstack-infra | 10:28 | |
| *** jaypipes has quit IRC | 10:29 | |
| *** pleia2 has quit IRC | 10:32 | |
| *** jaypipes has joined #openstack-infra | 10:32 | |
| *** thorst has joined #openstack-infra | 10:33 | |
| *** davideagnello has quit IRC | 10:33 | |
| *** Ramanjaneya has quit IRC | 10:33 | |
| *** samuelBartel has quit IRC | 10:38 | |
| *** thorst has quit IRC | 10:40 | |
| *** claudiub|2 has joined #openstack-infra | 10:42 | |
| *** dizquierdo has quit IRC | 10:44 | |
| *** claudiub has quit IRC | 10:45 | |
| *** jaypipes has quit IRC | 10:46 | |
| *** Fillip_k has quit IRC | 10:47 | |
| *** sdake has quit IRC | 10:56 | |
| *** mrmartin has joined #openstack-infra | 10:58 | |
| *** bookwar has quit IRC | 11:00 | |
| *** maishsk has quit IRC | 11:00 | |
| *** rfolco has joined #openstack-infra | 11:03 | |
| *** mrmartin has quit IRC | 11:03 | |
| *** e0ne has joined #openstack-infra | 11:05 | |
| *** fhubik is now known as fhubik_brb | 11:06 | |
| openstackgerrit | Merged openstack-infra/project-config: Build a Debian IPA image when DIB changes merge https://review.openstack.org/254405 | 11:06 |
| *** aysyd has joined #openstack-infra | 11:08 | |
| *** maishsk has joined #openstack-infra | 11:11 | |
| *** dims has quit IRC | 11:13 | |
| *** aysyd has quit IRC | 11:14 | |
| *** markvoelker has joined #openstack-infra | 11:15 | |
| *** aysyd has joined #openstack-infra | 11:15 | |
| *** markvoelker has quit IRC | 11:20 | |
| *** rfolco has quit IRC | 11:21 | |
| *** davideagnello has joined #openstack-infra | 11:23 | |
| *** placko_ has joined #openstack-infra | 11:25 | |
| *** davideagnello has quit IRC | 11:28 | |
| *** placko_ has quit IRC | 11:28 | |
| *** ramineni has left #openstack-infra | 11:30 | |
| *** ifarkas has joined #openstack-infra | 11:35 | |
| AJaeger | infra-root, tox 2.3.1 is released and should fix the tox problems. fungi, I let you self-approve the nodepool change https://review.openstack.org/#/c/256822/ ... | 11:36 |
| *** admin0 has quit IRC | 11:37 | |
| *** thorst has joined #openstack-infra | 11:38 | |
| *** Daisy has joined #openstack-infra | 11:39 | |
| *** chlong has joined #openstack-infra | 11:39 | |
| *** _nadya_ has joined #openstack-infra | 11:43 | |
| *** thorst has quit IRC | 11:44 | |
| *** dizquierdo has joined #openstack-infra | 11:45 | |
| *** masber has joined #openstack-infra | 11:46 | |
| *** derekh has joined #openstack-infra | 11:47 | |
| *** MarkAtwood has joined #openstack-infra | 11:48 | |
| *** vgridnev has quit IRC | 11:49 | |
| *** MarkAtwood has quit IRC | 11:53 | |
| *** pfallenop has quit IRC | 11:56 | |
| *** claudiub has joined #openstack-infra | 11:57 | |
| *** Daisy has quit IRC | 11:59 | |
| *** flepied1 has quit IRC | 11:59 | |
| *** flepied has joined #openstack-infra | 11:59 | |
| *** claudiub|2 has quit IRC | 11:59 | |
| *** maishsk has quit IRC | 12:01 | |
| *** zeih has joined #openstack-infra | 12:01 | |
| *** baoli has quit IRC | 12:03 | |
| *** piyush_ has joined #openstack-infra | 12:03 | |
| *** davideagnello has joined #openstack-infra | 12:03 | |
| *** admin0 has joined #openstack-infra | 12:07 | |
| *** davideagnello has quit IRC | 12:08 | |
| dmellado | hi guys, could you take a look at the grenade gate, I think I'm hitting a bug which would be fixed by https://review.openstack.org/#/c/257295/1/projects/60_nova/upgrade.sh | 12:09 |
| dmellado | on the grenade gate: log => http://logs.openstack.org/70/248170/24/check/gate-grenade-dsvm-neutron/7434897/logs/new/screen-n-api.txt.gz?level=TRACE | 12:09 |
| dmellado | could anyone confirm this, please? | 12:09 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/infra-specs: Expand on swift logs spec https://review.openstack.org/254718 | 12:11 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/infra-specs: Expand on swift logs spec https://review.openstack.org/254718 | 12:12 |
| *** pfallenop has joined #openstack-infra | 12:15 | |
| *** otter768 has joined #openstack-infra | 12:16 | |
| therve | dmellado, Yeah we talked about in #openstack-nova | 12:17 |
| therve | dmellado, https://review.openstack.org/#/c/257303/ is possibly the current fix, can you retry when it lands? | 12:18 |
| dmellado | therve: sure, thanks! | 12:18 |
| *** otter768 has quit IRC | 12:21 | |
| *** pfallenop has quit IRC | 12:21 | |
| *** pfallenop has joined #openstack-infra | 12:21 | |
| *** doug-fish has joined #openstack-infra | 12:23 | |
| nmagnezi | AJaeger, following to what I asked you about couple of hours ago: gate-grenade-dsvm-neutron fails for my patch despite the fact that all tests passed. So looks like it is infra related issue | 12:24 |
| nmagnezi | AJaeger, tests: http://logs.openstack.org/60/162260/27/check/gate-grenade-dsvm-neutron/1aab030/logs/testr_results.html.gz Log: http://logs.openstack.org/60/162260/27/check/gate-grenade-dsvm-neutron/1aab030/console.html | 12:25 |
| *** thorst has joined #openstack-infra | 12:27 | |
| openstackgerrit | Merged openstack/requirements: Updated from generate-constraints https://review.openstack.org/257155 | 12:28 |
| *** thorst_ has joined #openstack-infra | 12:28 | |
| *** masber has quit IRC | 12:29 | |
| *** ihrachys has joined #openstack-infra | 12:31 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack-infra/reviewstats: Add astara to projects https://review.openstack.org/257314 | 12:31 |
| *** eharney has joined #openstack-infra | 12:32 | |
| *** thorst has quit IRC | 12:32 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack-infra/reviewstats: Add chef related projects https://review.openstack.org/257316 | 12:32 |
| *** maishsk has joined #openstack-infra | 12:33 | |
| *** doug-fish has quit IRC | 12:34 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack-infra/reviewstats: Update cinder and related projects https://review.openstack.org/257319 | 12:34 |
| *** vgridnev has joined #openstack-infra | 12:34 | |
| therve | sdague, Got some random apt failure http://logs.openstack.org/03/257303/1/check/gate-grenade-dsvm-neutron/52fc9ed/logs/grenade.sh.txt.gz :/ | 12:35 |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack-infra/reviewstats: Update barbican and related projects https://review.openstack.org/257321 | 12:36 |
| *** ihrachys has quit IRC | 12:37 | |
| sdague | therve: yep, that's what this patch is supposed to help with - https://review.openstack.org/#/c/256469/ | 12:37 |
| *** ihrachys has joined #openstack-infra | 12:37 | |
| sdague | that bounced off the tox failure last, I just rechecked it | 12:38 |
| therve | God damn :) | 12:38 |
| *** dprince has joined #openstack-infra | 12:38 | |
| therve | The holiday spirit isn't with us | 12:38 |
| *** trown|outttypeww is now known as trown | 12:38 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack-infra/reviewstats: Add cloudkitty to projects https://review.openstack.org/257327 | 12:38 |
| *** doug-fish has joined #openstack-infra | 12:39 | |
| *** vgridnev has quit IRC | 12:39 | |
| therve | sdague, It already landed in stable/liberty though | 12:39 |
| *** vgridnev_ has joined #openstack-infra | 12:39 | |
| therve | Ah it may be some grenade weirdness | 12:40 |
| dmellado | therve: should you use help with that let me know ;) | 12:41 |
| therve | dmellado, I guess the grenade job for the stable/liberty branch uses stable/kilo as origin? | 12:41 |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack-infra/reviewstats: Add app-catalog to projects https://review.openstack.org/257330 | 12:42 |
| therve | So if we have a devstack bug it needs to be fixed all the way down | 12:42 |
| *** egonzalez has joined #openstack-infra | 12:42 | |
| nmagnezi | shall I file a bug? | 12:43 |
| dmellado | mmm I guess so | 12:43 |
| *** gordc has joined #openstack-infra | 12:44 | |
| nmagnezi | ack, since i never filed a bug to infra before, what do you guys expect me to attach? aside from the given logs and maybe related patch that I noticed the problem with | 12:44 |
| *** markvoelker has joined #openstack-infra | 12:45 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack-infra/reviewstats: Add chef related projects https://review.openstack.org/257316 | 12:45 |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack-infra/reviewstats: Update cinder and related projects https://review.openstack.org/257319 | 12:46 |
| sdague | therve: it's failing on the kilo side though | 12:47 |
| dmellado | nmagnezi: I guess the related patch + the logs showing the failure (but this is affecting several patches, so not really sure) | 12:47 |
| *** weshay_xchat has joined #openstack-infra | 12:48 | |
| *** hashar has joined #openstack-infra | 12:48 | |
| *** weshay_xchat is now known as weshay | 12:48 | |
| *** dims has joined #openstack-infra | 12:48 | |
| *** piyush_ has quit IRC | 12:49 | |
| *** markvoelker has quit IRC | 12:49 | |
| *** ihrachys has quit IRC | 12:50 | |
| *** vgridnev_ has quit IRC | 12:50 | |
| *** ihrachys has joined #openstack-infra | 12:51 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack-infra/reviewstats: Update nova and related projects https://review.openstack.org/257333 | 12:51 |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack-infra/reviewstats: Update nova and related projects https://review.openstack.org/257333 | 12:51 |
| *** salv-orl_ has joined #openstack-infra | 12:54 | |
| *** vgridnev_ has joined #openstack-infra | 12:54 | |
| *** aopincar has quit IRC | 12:55 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack-infra/reviewstats: Update ironic and related projects https://review.openstack.org/257336 | 12:55 |
| *** baoli has joined #openstack-infra | 12:55 | |
| *** salv-orlando has quit IRC | 12:56 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack-infra/reviewstats: Add chef related projects https://review.openstack.org/257316 | 12:58 |
| *** davideagnello has joined #openstack-infra | 12:58 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack-infra/reviewstats: Update cinder and related projects https://review.openstack.org/257319 | 12:59 |
| *** jaypipes has joined #openstack-infra | 13:00 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack-infra/reviewstats: Add cloudkitty to projects https://review.openstack.org/257327 | 13:00 |
| *** maishsk has quit IRC | 13:02 | |
| *** davideagnello has quit IRC | 13:03 | |
| AJaeger | nmagnezi: did you read the conversation above between dmellado and therve? | 13:03 |
| *** ihrachys has quit IRC | 13:04 | |
| *** ihrachys has joined #openstack-infra | 13:05 | |
| AJaeger | sdague: where should nmagnezi file the bug? | 13:05 |
| *** salv-orl_ has quit IRC | 13:05 | |
| *** vgridnev_ has quit IRC | 13:05 | |
| AJaeger | nmagnezi: I see that you chimed in, didn't read whole backscroll before answering | 13:05 |
| *** vgridnev has joined #openstack-infra | 13:06 | |
| sdague | AJaeger: ?? | 13:06 |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack-infra/reviewstats: Update infrastructure projects https://review.openstack.org/257337 | 13:08 |
| AJaeger | sdague: nmagnezi asked at 12:43 where to file a bug for this, devstack in launchpad? Or is this a nova bug? | 13:09 |
| *** bharathm has joined #openstack-infra | 13:11 | |
| sdague | devstack is fine, though the fix is already posted by therve | 13:11 |
| *** flepied has quit IRC | 13:11 | |
| AJaeger | thanks, sdague | 13:12 |
| nmagnezi | i'm on it :) | 13:12 |
| *** bharathm has quit IRC | 13:15 | |
| nmagnezi | AJaeger, https://bugs.launchpad.net/devstack/+bug/1525904 | 13:18 |
| openstack | Launchpad bug 1525904 in devstack "The gate-grenade-dsvm-neutron gate fails to run" [Undecided,New] | 13:18 |
| AJaeger | nmagnezi: better tell therve about it ;) | 13:19 |
| AJaeger | thanks | 13:19 |
| nmagnezi | :) | 13:19 |
| dmellado | nmagnezi: thanks ;) | 13:19 |
| dmellado | nmagnezi: wouldn't you have the nova-api logs around too? | 13:21 |
| dmellado | otherwise I'll just post mine, if you don't mind ;) | 13:21 |
| nmagnezi | dmellado, which log exactly? http://logs.openstack.org/60/162260/27/check/gate-grenade-dsvm-neutron/1aab030/logs/ | 13:22 |
| nmagnezi | dmellado, oh, this one? http://logs.openstack.org/60/162260/27/check/gate-grenade-dsvm-neutron/1aab030/logs/old/screen-n-api.txt.gz | 13:22 |
| dmellado | nmagnezi: kinda like http://logs.openstack.org/70/248170/24/check/gate-grenade-dsvm-neutron/7434897/logs/new/screen-n-api.txt.gz?level=TRACE | 13:22 |
| dmellado | yep, ;) | 13:22 |
| dmellado | I assume yours is failing in the exact same place | 13:23 |
| dmellado | yep | 13:23 |
| dmellado | exactly | 13:23 |
| dmellado | http://logs.openstack.org/60/162260/27/check/gate-grenade-dsvm-neutron/1aab030/logs/new/screen-n-api.txt.gz?level=TRACE | 13:23 |
| nmagnezi | dmellado, done :) | 13:24 |
| dmellado | thanks! | 13:24 |
| dmellado | I think it's more clear this way | 13:24 |
| *** davideagnello has joined #openstack-infra | 13:26 | |
| *** dkranz has quit IRC | 13:26 | |
| *** rhallisey has joined #openstack-infra | 13:26 | |
| *** mgrosz_ has joined #openstack-infra | 13:27 | |
| *** bcourt has joined #openstack-infra | 13:28 | |
| mgrosz_ | Hello, can someone please refer me to docs that describe how do I add vitrage tests to tempest? Do I need to open a new project (something like vitrage-tempest)? Do I add code to the main 'tempest' project? | 13:28 |
| *** jyuso1 has quit IRC | 13:29 | |
| *** davideagnello has quit IRC | 13:31 | |
| nmagnezi | AJaeger, shouldn't we just change project for https://bugs.launchpad.net/devstack/+bug/1525904 ? | 13:33 |
| openstack | Launchpad bug 1525904 in devstack "The gate-grenade-dsvm-neutron gate fails to run" [Undecided,New] | 13:33 |
| *** fhubik_brb is now known as fhubik | 13:33 | |
| *** kambiz has joined #openstack-infra | 13:34 | |
| *** MarkAtwood has joined #openstack-infra | 13:37 | |
| *** BobBall has quit IRC | 13:39 | |
| jordanP | mgrosz_, look at how manila is doing it | 13:39 |
| jordanP | they have a manila_tempest_tests directory in their source code repo | 13:40 |
| mgrosz_ | Do you know if they created it using some kind of a cookiecutter template? | 13:41 |
| *** jaypipes has quit IRC | 13:41 | |
| *** MarkAtwood has quit IRC | 13:41 | |
| mgrosz_ | Or do we just need to follow their directories stracture? | 13:42 |
| *** julim has joined #openstack-infra | 13:42 | |
| mordred | sdague, therve: anything you need from me on that grenade/neutron options removal patch? | 13:43 |
| *** BobBall has joined #openstack-infra | 13:43 | |
| mgrosz_ | jordanP: Do you know if they created it using some kind of a cookiecutter template? Or do we just need to follow their directories stracture? | 13:44 |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Only update nodepool images on Tuesdays" https://review.openstack.org/256822 | 13:44 |
| jordanP | mgrosz_, I am not aware of any template | 13:44 |
| *** Daisy has joined #openstack-infra | 13:44 | |
| jordanP | mgrosz_, now that I have googled it: http://docs.openstack.org/developer/tempest/plugin.html | 13:45 |
| jordanP | mgrosz_, if this doc is lacking, please submit a patch, that would help greatly | 13:45 |
| jordanP | this stuff is kinda new | 13:45 |
| *** fhubik is now known as fhubik_brb | 13:46 | |
| *** jaosorior has joined #openstack-infra | 13:47 | |
| *** fhubik_brb is now known as fhubik | 13:47 | |
| *** tlian has joined #openstack-infra | 13:48 | |
| *** Daisy has quit IRC | 13:50 | |
| *** ihrachys has quit IRC | 13:50 | |
| *** flepied has joined #openstack-infra | 13:51 | |
| *** flepied1 has joined #openstack-infra | 13:51 | |
| sdague | mordred: it's just grinding lots of patches because kilo / liberty failure rates | 13:53 |
| *** flepied has quit IRC | 13:55 | |
| *** jlanoux has quit IRC | 13:55 | |
| *** ihrachys has joined #openstack-infra | 13:55 | |
| *** jlanoux has joined #openstack-infra | 13:56 | |
| mordred | sdague: lovely | 13:56 |
| *** zeus has joined #openstack-infra | 13:57 | |
| *** gordc has quit IRC | 13:57 | |
| *** zeus is now known as Guest83257 | 13:57 | |
| sdague | yeh, instances like this, I do really wish we have a fast merge facility | 13:58 |
| *** Guest83257 is now known as zeus- | 13:58 | |
| *** zeus- is now known as zeus` | 13:58 | |
| therve | sdague, Isn't there a way to cancel builds at least? | 13:58 |
| sdague | because we as humans know the test failures for these are not relevant, and the current net result is neutron / heat / and a bunch of other projects are blocked until we grind these patches through | 13:58 |
| openstackgerrit | Paul Michali proposed openstack-infra/project-config: Make VPN constraints jobs voting https://review.openstack.org/255992 | 13:59 |
| sdague | therve: updating the patch | 13:59 |
| sdague | there is no cancel for a build | 13:59 |
| AJaeger | sdague: 257303 is failing right now ;( | 13:59 |
| sdague | AJaeger: yep, because of unrelated causes | 13:59 |
| AJaeger | sdague: you can ask mordred to push it at the top of the gate once it passed the check queue... | 13:59 |
| sdague | AJaeger: sure, but the fail rate is high enough that unless someone stays dedicated on grinding this, it will probably just bounce off | 14:00 |
| sdague | there are many ways we can spend lots of human time here | 14:00 |
| *** bcourt has quit IRC | 14:01 | |
| *** nikhil_k is now known as nikhil | 14:02 | |
| AJaeger | ;( | 14:03 |
| *** pblaho has joined #openstack-infra | 14:06 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Bug fix: create_stack() fails when waiting https://review.openstack.org/257367 | 14:09 |
| *** lykinsbd has joined #openstack-infra | 14:09 | |
| *** zeih has quit IRC | 14:09 | |
| *** dims has quit IRC | 14:11 | |
| *** gordc has joined #openstack-infra | 14:12 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Bug fix: create_stack() fails when waiting https://review.openstack.org/257367 | 14:13 |
| *** zhurong has joined #openstack-infra | 14:14 | |
| *** bcourt has joined #openstack-infra | 14:14 | |
| *** rlandy has joined #openstack-infra | 14:16 | |
| *** bookwar has joined #openstack-infra | 14:16 | |
| *** admin0 has quit IRC | 14:16 | |
| *** otter768 has joined #openstack-infra | 14:17 | |
| *** changbl has quit IRC | 14:17 | |
| *** ildikov has quit IRC | 14:18 | |
| mordred | mtreinish: comment on your shade patch - thanks for the patch btw! | 14:19 |
| *** jcoufal_ has joined #openstack-infra | 14:19 | |
| *** dims has joined #openstack-infra | 14:19 | |
| *** changbl has joined #openstack-infra | 14:19 | |
| *** ajmiller has joined #openstack-infra | 14:19 | |
| *** davideagnello has joined #openstack-infra | 14:20 | |
| *** jaypipes has joined #openstack-infra | 14:21 | |
| *** otter768 has quit IRC | 14:21 | |
| *** jcoufal has quit IRC | 14:22 | |
| *** alivigni has joined #openstack-infra | 14:22 | |
| *** peristeri has joined #openstack-infra | 14:23 | |
| *** hashar has quit IRC | 14:23 | |
| *** dkranz has joined #openstack-infra | 14:24 | |
| *** hashar has joined #openstack-infra | 14:24 | |
| *** annegentle has joined #openstack-infra | 14:24 | |
| *** davideagnello has quit IRC | 14:25 | |
| *** Piet has joined #openstack-infra | 14:26 | |
| *** yamamoto has quit IRC | 14:26 | |
| *** dims has quit IRC | 14:28 | |
| *** regXboi has joined #openstack-infra | 14:28 | |
| *** dims has joined #openstack-infra | 14:30 | |
| *** NobodyCa1 is now known as NobodyCam | 14:30 | |
| *** zeih has joined #openstack-infra | 14:31 | |
| *** ildikov has joined #openstack-infra | 14:32 | |
| *** lucasagomes is now known as lucas-hungry | 14:34 | |
| openstackgerrit | Paul Michali proposed openstack-infra/project-config: Make VPN constraints jobs voting https://review.openstack.org/255992 | 14:36 |
| *** trown is now known as trown|outttypeww | 14:40 | |
| *** samuelBartel has joined #openstack-infra | 14:40 | |
| *** zhurong has quit IRC | 14:40 | |
| *** zeih has quit IRC | 14:41 | |
| *** markvoelker has joined #openstack-infra | 14:43 | |
| openstackgerrit | Paul Michali proposed openstack-infra/project-config: Make FWaaS constraints jobs voting https://review.openstack.org/256702 | 14:44 |
| sdague | AJaeger: it looks like this ironic test http://tinyurl.com/p8t8osb is the one that's mostly killing what's going on here | 14:45 |
| dmellado | mgrosz_: I'm checking that out and also found cookiecutter to be outdated | 14:47 |
| dmellado | I'm taking congress patches as an example | 14:47 |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Adds an API migration and docs generation tool, fairy-slipper https://review.openstack.org/245352 | 14:48 |
| *** ildikov has quit IRC | 14:48 | |
| *** zeih has joined #openstack-infra | 14:48 | |
| *** zeih has quit IRC | 14:48 | |
| *** zeih has joined #openstack-infra | 14:48 | |
| phschwartz | fungi: clarkb: So things look a lot better in the local env you helped me with ty. nodepool is working but I still have one job logging NOT_REGISTERED | 14:50 |
| phschwartz | fungi: clarkb: build:dsvm-tempest-full:devstack-trusty 0 0 1 is there when I look at the status | 14:50 |
| AJaeger | phschwartz: do you have a review showing the NOT_REGISTERED? | 14:51 |
| phschwartz | AJaeger: no, it is running this one in a silent queue | 14:52 |
| phschwartz | AJaeger: I see 2015-11-25 19:11:20,055 DEBUG zuul.IndependentPipelineManager: Adding build <Build b1b956c2a6ea4d86a56f3fa7b5561f82 of dsvm-tempest-full on <Worker Unknown>> of job dsvm-tempest-full to item <QueueItem 0x7f1ff0058410 for <Change 0x7f1ff0058c10 249985,1> in silent> | 14:52 |
| phschwartz | - dsvm-tempest-full http://159.122.254.10/85/249985/1/silent/dsvm-tempest-full/b1b956c : NOT_REGISTERED | 14:52 |
| AJaeger | 2015-11-25 is old ;) | 14:52 |
| phschwartz | AJaeger: ah, I pasted an old one. log is riddled with the error including one from friday (last I tried it) | 14:53 |
| openstackgerrit | Paul Michali proposed openstack-infra/project-config: Make LBaaS constraints jobs voting https://review.openstack.org/256391 | 14:53 |
| AJaeger | phschwartz: ah, it's your own server, better continue discussing with others... | 14:53 |
| AJaeger | pc_m: are you around? | 14:53 |
| phschwartz | AJaeger: yeah, local setup | 14:53 |
| *** rbrndt has joined #openstack-infra | 14:54 | |
| fungi | phschwartz: compare the worker name in your gearman registration status with the worker name mentioned in your (recent!) zuul debug logs | 14:54 |
| *** tmorin has joined #openstack-infra | 14:54 | |
| *** tmorin has left #openstack-infra | 14:54 | |
| fungi | phschwartz: make sure zuul's really asking for dsvm-tempest-full:devstack-trusty and not dsvm-tempest-full:something-else | 14:55 |
| AJaeger | pc_m: glad to see the backport for LBaaS - can't you do this for the others as well? | 14:55 |
| *** salv-orlando has joined #openstack-infra | 14:55 | |
| phschwartz | fungi: ty as always that pointed it out. build:dsvm-tempest-full:bare-trusty it is looking for a job registered to bare-trusty when that job is looking for devstack-trusty. I have an issue in my zuul layout. | 14:55 |
| *** ajmiller has quit IRC | 14:56 | |
| *** xarses has quit IRC | 14:56 | |
| pc_m | AJaeger: Show all be up now. I also have jobs in that make the constraints jobs voting for liberty and master (and hopefully depend on the backports). | 14:56 |
| pc_m | Should all be up.. | 14:57 |
| jroll | sdague: jeeeez, what can I do to help there? (other than, that bug is my number one priority once we push this devstack plugin stuff through) | 14:57 |
| jroll | related: I'd love to get this landed this morning https://review.openstack.org/#/c/255942 | 14:57 |
| fungi | phschwartz: check that you don't have a zuul parameter function setting dsvm-tempest-full to use bare-trusty workers | 14:57 |
| pc_m | AJaeger: https://review.openstack.org/#/c/257366 https://review.openstack.org/#/c/257372 https://review.openstack.org/#/c/257412 | 14:57 |
| phschwartz | fungi: that is exactly what I have. correcting it now :) | 14:58 |
| pc_m | AJaeger: those are the backports. | 14:58 |
| anteaya | can I get someone with channel ops to update the channel topic in -meeting please? | 14:58 |
| dims | sdague : when you get a chance, please peek at https://review.openstack.org/#/c/256467/ (experimental grenade tests for oslo.messaging master) | 14:58 |
| anteaya | actually all meeting channels seem to need their topics addressed | 14:59 |
| sdague | dims: is there a reason those 2 patches aren't just sqashed together? | 14:59 |
| dims | sdague : i had it that way before and AJaeger preferred this | 15:00 |
| sdague | AJaeger: really? | 15:00 |
| sdague | it's just fixing an experimental queue thing | 15:00 |
| sdague | why revert it? | 15:00 |
| *** sridhar_ram has joined #openstack-infra | 15:00 | |
| anteaya | have to start my meeting now | 15:01 |
| anteaya | maybe after | 15:01 |
| *** MarkAtwood has joined #openstack-infra | 15:02 | |
| AJaeger | dims: you had it? Then I have screwed up in reviewing - or you did not mention it ;( | 15:04 |
| *** ig0r_ has quit IRC | 15:04 | |
| dims | AJaeger : oops, the same 2 lines were changed, sorry i should have mentioned it explicitly | 15:04 |
| dims | let me squash them | 15:05 |
| *** ildikov has joined #openstack-infra | 15:05 | |
| *** irenab has quit IRC | 15:05 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:05 | |
| AJaeger | dims: both works for me ;) | 15:06 |
| *** yonglihe has joined #openstack-infra | 15:07 | |
| *** amitgandhinz has joined #openstack-infra | 15:07 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Grenade jobs for testing libraries (esp. oslo.messaging) https://review.openstack.org/256467 | 15:08 |
| *** pradk has joined #openstack-infra | 15:08 | |
| openstackgerrit | Merged openstack-infra/shade: Stack API improvements https://review.openstack.org/256751 | 15:08 |
| dims | AJaeger : sdague : ready - https://review.openstack.org/#/c/256467/ | 15:08 |
| fungi | anteaya: looks like the meetbot must have restarted in the middle of some meetings last week or something. i'll try to get them fixed up when they don't have meetings going on (starting now with -4) | 15:10 |
| anteaya | fungi: thank you | 15:10 |
| openstackgerrit | Nicko Glayre proposed openstack-infra/jenkins-job-builder: Adding support for maven-metadata-plugin https://review.openstack.org/257430 | 15:10 |
| *** zz_dimtruck is now known as dimtruck | 15:13 | |
| *** annegentle has quit IRC | 15:13 | |
| *** bauzas has joined #openstack-infra | 15:15 | |
| *** yamamoto has joined #openstack-infra | 15:15 | |
| *** bexelbie_ has joined #openstack-infra | 15:16 | |
| anteaya | fungi: you can do -meeting now | 15:16 |
| *** zhenguo has joined #openstack-infra | 15:17 | |
| fungi | anteaya: thanks! done | 15:19 |
| *** mtanino has joined #openstack-infra | 15:19 | |
| jroll | fungi: don't forget #openstack-meeting-cp :) | 15:20 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Have inventory use os-client-config extra_config https://review.openstack.org/257436 | 15:20 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Make server variable expansion optional https://review.openstack.org/257437 | 15:20 |
| anteaya | fungi: thank you | 15:20 |
| *** ildikov has quit IRC | 15:20 | |
| anteaya | jroll: thanks, was trying to remember the name of "the other channel" | 15:20 |
| jroll | heh | 15:21 |
| anteaya | so many other things to remember | 15:21 |
| anteaya | the thing of it is that disassembling my mental model so I can construct a new one to encompass all the new things takes energy and time | 15:21 |
| anteaya | which I never seem to get | 15:22 |
| *** nmagnezi has quit IRC | 15:22 | |
| anteaya | so I seem to default to old mental model with the new things just tossed in the kitchen drawer | 15:22 |
| anteaya | with the elastic bands, bread clips and picture hangers | 15:22 |
| jroll | ++ that's my model | 15:23 |
| dims | AJaeger : sdague : thanks | 15:24 |
| bauzas | is it me or gerrit is super slow now ? | 15:24 |
| anteaya | okay, so all new channels shall henceforth be named kitchen-drawer-<number> and I'm good | 15:24 |
| mordred | Shrews: the two shade patches above and https://github.com/ansible/ansible/pull/13544 go with occ patch https://review.openstack.org/#/c/246587 | 15:24 |
| mordred | nibalizer, clarkb: ^^ this work above is what we need to not have uuids in our logs anymore | 15:25 |
| fungi | jroll: yeah, we can add a standard topic to the -cp channel too. right now it just has no topic | 15:25 |
| anteaya | bauzas: it _might_ be in garbage collection mode | 15:25 |
| therve | mordred, If you can bump https://review.openstack.org/#/c/257303/ at the top of the queue that'd be great | 15:26 |
| bauzas | anteaya: hah ack | 15:26 |
| anteaya | anyone have time to see what java melody is saying with regards to garbage collection? | 15:26 |
| mordred | fungi: ^^ can you help therve - I'm about to walk out the door | 15:26 |
| therve | I think it might be ok, looks there now | 15:27 |
| *** tsg has joined #openstack-infra | 15:27 | |
| mordred | Shrews, clarkb, nibalizer: I'm going to breakfast. when I get back I'll add info to the puppet apply etherpad on those- and also on the group config problem that we need to solve before we flip the puppet apply switch | 15:28 |
| sdague | gerrit could probably use a resetart | 15:28 |
| fungi | mordred: therve: in a meeting, but i'm looking | 15:28 |
| anteaya | mordred: enjoy food | 15:28 |
| *** jswarren has joined #openstack-infra | 15:29 | |
| *** migi has quit IRC | 15:32 | |
| *** kgiusti has joined #openstack-infra | 15:32 | |
| *** migi has joined #openstack-infra | 15:33 | |
| pabelanger | Woah, merge-check queue up to 520 on zuul | 15:33 |
| sdague | pabelanger: that just means a nova patch merged | 15:33 |
| *** _nadya_ has quit IRC | 15:34 | |
| anteaya | pabelanger: this numbers are of higher importance: Queue lengths: 29 events, 55 results. | 15:34 |
| *** _nadya_ has joined #openstack-infra | 15:34 | |
| fungi | pabelanger: yeah, it gets burned down very quickly as long as the merger workers aren't busy prepping refs for other jobs | 15:35 |
| anteaya | though when zuul is calculating a gate reset the number can spike at very high values | 15:35 |
| anteaya | so if queue lengths are large numbers for more than 20 minutes, that is worth knowing about | 15:35 |
| fungi | and merge-check doesn't use any job workers, it just no-ops after the merge commit is confirmed it can be created | 15:35 |
| *** _nadya_ has quit IRC | 15:35 | |
| *** lucas-hungry is now known as lucasagomes | 15:37 | |
| HeOS_ | Hello! Have you still planed maintenance of Gerrit on 12/16? | 15:37 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/shade: Fix server deletes when cinder isn't available https://review.openstack.org/256994 | 15:37 |
| *** fhubik is now known as fhubik_brb | 15:37 | |
| *** fhubik_brb is now known as fhubik | 15:37 | |
| *** superdan is now known as dansmith | 15:38 | |
| *** ildikov has joined #openstack-infra | 15:38 | |
| fungi | HeOS_: yes, we're operating under that expectation | 15:38 |
| *** sdake has joined #openstack-infra | 15:40 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/project-config: Add a test-accounts src-{mame} exp job https://review.openstack.org/257448 | 15:41 |
| *** davideagnello has joined #openstack-infra | 15:41 | |
| *** fhubik is now known as fhubik_brb | 15:42 | |
| *** xarses has joined #openstack-infra | 15:42 | |
| *** r-daneel has joined #openstack-infra | 15:43 | |
| *** fhubik_brb is now known as fhubik | 15:43 | |
| *** admin0 has joined #openstack-infra | 15:43 | |
| openstackgerrit | Kyle Mestery proposed openstack-infra/shade: Pedantic spelling corrections https://review.openstack.org/257452 | 15:45 |
| *** davideagnello has quit IRC | 15:47 | |
| dansmith | thank you for the gerrit restart.. so.much.faster. | 15:47 |
| *** _d34dh0r53_ is now known as d34dh0r53 | 15:47 | |
| betherly | could someone take a look at https://review.openstack.org/#/c/254845/ | 15:48 |
| *** Sukhdev has joined #openstack-infra | 15:48 | |
| betherly | https://review.openstack.org/#/c/254853/ | 15:48 |
| fungi | dansmith: gerrit was last restarted on december 3, so whatever caused your performance to speed up wasn't a restart | 15:49 |
| dansmith | fungi: it wasn't just restarted a minute ago? | 15:49 |
| fungi | nope | 15:50 |
| dansmith | it went 503 for a minute.. everyone thought it was a restart | 15:50 |
| *** fhubik is now known as fhubik_brb | 15:50 | |
| fungi | it was probably in particularly bad shape in that case. i see the load average was a lot higher but has been plummeting in the last few minutes | 15:51 |
| openstackgerrit | Kyle Mestery proposed openstack-infra/shade: Pedantic spelling correction https://review.openstack.org/257452 | 15:51 |
| dansmith | hmm, okay | 15:51 |
| fungi | i have a feeling zuul was beating it up. there was a pretty large spike in both check and gate pipelines which also coincided with a lot of merge-checks | 15:51 |
| *** fhubik_brb is now known as fhubik | 15:53 | |
| *** ildikov has quit IRC | 15:54 | |
| *** annegentle has joined #openstack-infra | 15:54 | |
| *** vgridnev has quit IRC | 15:54 | |
| *** ildikov has joined #openstack-infra | 15:54 | |
| openstackgerrit | Beth Elwell proposed openstack-infra/project-config: New project for Ironic UI plugin https://review.openstack.org/254845 | 15:55 |
| *** tonytan4ever has joined #openstack-infra | 15:56 | |
| *** fhubik has quit IRC | 15:56 | |
| *** zhenguo has quit IRC | 15:56 | |
| *** yamamoto has quit IRC | 15:56 | |
| *** 21WAAGP61 has quit IRC | 15:57 | |
| *** arxcruz has quit IRC | 15:57 | |
| *** andreaf_ has joined #openstack-infra | 15:57 | |
| HeOS_ | fungi: thanks. :) | 15:57 |
| *** yamamoto has joined #openstack-infra | 15:58 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/shade: Fix server deletes when cinder isn't available https://review.openstack.org/256994 | 15:58 |
| *** dtantsur is now known as dtantsur|afk | 15:59 | |
| *** sshnaidm has quit IRC | 16:03 | |
| fungi | okay, all the meeting channels have their default topics corrected again | 16:03 |
| *** jistr has quit IRC | 16:03 | |
| *** sridhar_ram has quit IRC | 16:04 | |
| *** amitgandhinz has quit IRC | 16:04 | |
| *** HeOS has joined #openstack-infra | 16:05 | |
| anteaya | fungi: thank you | 16:05 |
| *** mgrosz_ has quit IRC | 16:05 | |
| *** mrmartin has joined #openstack-infra | 16:05 | |
| anteaya | celebrating the small victories | 16:05 |
| *** ifarkas has quit IRC | 16:06 | |
| sdague | interesting, I'm getting a "waiting for available socket...." at the bottom of my gerrit window | 16:07 |
| jeblair | neat, never seen that. | 16:07 |
| *** krtaylor has quit IRC | 16:08 | |
| *** krtaylor has joined #openstack-infra | 16:08 | |
| *** davideagnello has joined #openstack-infra | 16:09 | |
| *** bdemers has quit IRC | 16:09 | |
| *** bdemers has joined #openstack-infra | 16:09 | |
| anteaya | sdague: you get all the fun messages | 16:09 |
| *** maurosr has quit IRC | 16:10 | |
| *** EricGonczer_ has joined #openstack-infra | 16:10 | |
| jeblair | i think our gerrit vhost config overrides /server-status.... | 16:10 |
| *** tonytan4ever has quit IRC | 16:11 | |
| *** ildikov has quit IRC | 16:11 | |
| *** dizquierdo has quit IRC | 16:12 | |
| pabelanger | wouldn't mind some help landing the following patch for diskimage-builder: https://review.openstack.org/#/c/256071/ Part of a patchset to fix fedora-minumal under centos7 | 16:12 |
| fungi | oh, i have a feeling the switch to mod_proxy means we get new error messages when gerrit is too loaded and it decides things are waiting too long to establish a new socket | 16:13 |
| *** davideagnello has quit IRC | 16:13 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-gerrit: Do not proxy /server-status https://review.openstack.org/257464 | 16:14 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet/syntax-3: adding psych gem explicitly https://review.openstack.org/257465 | 16:15 |
| jeblair | fungi, sdague, notmorgan: https://review.openstack.org/257464 | 16:15 |
| jeblair | completely untested | 16:15 |
| AJaeger | pc_m: Sorry, was in a meeting and will review later. Thanks! | 16:16 |
| notmorgan | jeblair: that doesn't look quite right. | 16:16 |
| notmorgan | jeblair: sec | 16:16 |
| notmorgan | jeblair: you want to prevent proxying server status right? | 16:16 |
| jeblair | notmorgan: doh; i missed a negation on the proxypassmatch didn't i :) | 16:17 |
| jeblair | notmorgan: yes | 16:17 |
| notmorgan | yeah | 16:17 |
| sdague | jeblair: oh, so you think we upped gerrit load because of some other checks | 16:17 |
| notmorgan | alos not sure the REWRITE_COND is needed | 16:17 |
| jeblair | sdague: no -- i just can't see what apache sees | 16:17 |
| sdague | ah, ok | 16:17 |
| *** ihrachys has quit IRC | 16:17 | |
| notmorgan | jeblair: so yeah at tleast you you need the ! negation | 16:17 |
| *** otter768 has joined #openstack-infra | 16:18 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-gerrit: Do not proxy /server-status https://review.openstack.org/257464 | 16:19 |
| jeblair | notmorgan: added ! | 16:19 |
| notmorgan | jeblair: is the rewrite cond to ensure you get server_status for the vhost? | 16:19 |
| *** admin0 has quit IRC | 16:19 | |
| *** rajinir_ has joined #openstack-infra | 16:20 | |
| openstackgerrit | Merged openstack-infra/project-config: Grenade jobs for testing libraries (esp. oslo.messaging) https://review.openstack.org/256467 | 16:21 |
| anteaya | jeblair: you asked last week about a word that means files not germane to a job, I have some thoughts: https://etherpad.openstack.org/p/Eu5meO0aLg | 16:21 |
| anteaya | jeblair: I'm leaning towards filigree myself at the moment | 16:21 |
| *** andreaf has quit IRC | 16:21 | |
| *** andreaf_ is now known as andreaf | 16:21 | |
| jeblair | notmorgan: it's (hopefully) to allow 'https://localhost/server-status' to work without hitting the 'redirect other hostnames to review.o.o' rule | 16:22 |
| *** jtomasek has quit IRC | 16:22 | |
| jeblair | (since localhost!=review.o.o) | 16:22 |
| notmorgan | hm | 16:22 |
| *** otter768 has quit IRC | 16:22 | |
| notmorgan | sure | 16:22 |
| fungi | jeblair: i believe the apache config is consistent between review and review-dev at this point, so should be pretty easy to confirm on the latter | 16:22 |
| notmorgan | oh hah missed the negation | 16:22 |
| jeblair | fungi: cool, so if notmorgan likes the current thing, i can try it out there by hand? | 16:23 |
| fungi | yeah, that makes the most sense | 16:23 |
| notmorgan | jeblair: LGTM without testing | 16:23 |
| *** matrohon has quit IRC | 16:24 | |
| fungi | current patch lgtm too, so assuming a byhand confirmation i'm cool approving it and then we can hopefully see what's going on with the current slowdown | 16:24 |
| *** tonytan4ever has joined #openstack-infra | 16:24 | |
| jeblair | hand-applied to dev | 16:24 |
| notmorgan | fungi: we're seeing slowdown on review? | 16:24 |
| jeblair | links https://localhost/server-status works | 16:25 |
| jeblair | the webui appears to work for me | 16:25 |
| jeblair | and https://review-dev.openstack.org/server-status is 403 forbidden as expected | 16:26 |
| notmorgan | jeblair: yeah seems to work for me too | 16:26 |
| *** eharney has quit IRC | 16:26 | |
| notmorgan | on review-dev | 16:26 |
| fungi | yeah, looking fine here. let's go for it | 16:26 |
| anteaya | notmorgan: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=25&rra_id=all | 16:27 |
| *** amitgandhinz has joined #openstack-infra | 16:27 | |
| anteaya | notmorgan: and sdague is getting some really interesting error messages about waiting for socket | 16:27 |
| phschwartz | fungi: I updated my layout and jenkins jobs to modify the job name so they are sent to the right type of node based on the job name (same as infra), now the jobs are getting into the zuul queue but never run on jenkins. I see the targets in when pulling the status and I have active nodes from nodepool as slaves in jenkins | 16:27 |
| phschwartz | fungi: Any thoughts | 16:28 |
| notmorgan | there might be tunables | 16:28 |
| sdague | did something substantially change about the proxy model recently? | 16:28 |
| fungi | phschwartz: check the jenkins server log. i believe the gearman plugin logs there | 16:28 |
| notmorgan | for proxypass or such | 16:29 |
| sdague | I'm seeing a 7s delay on a lot of api requests | 16:29 |
| sdague | http://dl.dropbox.com/u/6514884/screenshot_281.png | 16:29 |
| notmorgan | sdague: that might be some tunables | 16:29 |
| fungi | sdague: yes, we had to go from using mod_rewrite to full mod_proxy to deal with some issues with the openid consumer in gerrit 2.10 and later (in preparation for wednesday) | 16:29 |
| *** rossella_s has quit IRC | 16:30 | |
| *** rossella_s has joined #openstack-infra | 16:30 | |
| sdague | notmorgan: you poking at things with old screen or new screen? | 16:31 |
| notmorgan | sdague: never new screen | 16:31 |
| notmorgan | sdague: i find the new screen terrible/borderline unusable | 16:31 |
| fungi | notmorgan: always new screen after wednesday | 16:31 |
| notmorgan | fungi: yeah i'll adapt | 16:31 |
| phschwartz | fungi: after my last reconnect from a zuul restart when I reconfigured it I see an INFO log about sending initial grab job, nothing else. | 16:31 |
| fungi | but the new screen in 2.11 is less terrible than in 2.8 | 16:31 |
| notmorgan | but the 2.11 is better than the current new screen | 16:32 |
| notmorgan | yeah | 16:32 |
| sdague | I wonder if new screen is making more REST requests back | 16:32 |
| notmorgan | likely | 16:32 |
| sdague | so the estimation on how many workers needed in the proxy was undercounted | 16:32 |
| sdague | because it's the requests for the related changes (for instance) that are stalling | 16:32 |
| openstackgerrit | Merged openstack/python-jenkins: Fixed create_node() Error https://review.openstack.org/254644 | 16:32 |
| *** egonzalez has quit IRC | 16:33 | |
| sdague | so I'm basically at 9s minimum link follow now on gerrit | 16:34 |
| sdague | with it going up to 25s at time | 16:34 |
| notmorgan | sdague: yeah new screen is way slower | 16:34 |
| jeblair | garbage collection time is way up today | 16:36 |
| jeblair | (though it has had similar peaks each of the last 2 weeks as well) | 16:36 |
| anteaya | jeblair: thanks for looking | 16:36 |
| notmorgan | jeblair: was going to say the CPU graphs look similar-ish over the last couple weeks | 16:37 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config: Speed up release-critical post jobs https://review.openstack.org/257472 | 16:37 |
| openstackgerrit | Tomasz 'Zen' Napierala proposed openstack-infra/project-config: Create new project Solar https://review.openstack.org/255983 | 16:38 |
| *** ttx has quit IRC | 16:38 | |
| *** ttx has joined #openstack-infra | 16:39 | |
| *** andrearosa has joined #openstack-infra | 16:40 | |
| *** andrearosa has left #openstack-infra | 16:40 | |
| *** dims has quit IRC | 16:42 | |
| *** jaypipes has quit IRC | 16:44 | |
| *** bpokorny has joined #openstack-infra | 16:44 | |
| *** Swami_ has joined #openstack-infra | 16:46 | |
| pabelanger | fungi: do you see any issues with: https://review.openstack.org/#/c/256995/ enabling jenkins user to ssh to localhost | 16:47 |
| *** armax has joined #openstack-infra | 16:47 | |
| pabelanger | we do the same with beaker, but that is using root | 16:47 |
| *** bexelbie_ is now known as bexelbie | 16:48 | |
| *** pleia2 has joined #openstack-infra | 16:48 | |
| *** e0ne has quit IRC | 16:48 | |
| *** jlanoux has quit IRC | 16:49 | |
| *** peristeri has quit IRC | 16:49 | |
| *** davideagnello has joined #openstack-infra | 16:49 | |
| pleia2 | good morning | 16:50 |
| fungi | pabelanger: seems fine, but a bit excessive. we might want to consider tuning down the keysize if generating that ties up the job for too long. also why the chmod there? just in case ~jenkins/.ssh/authorized_keys didn't already exist? | 16:50 |
| pabelanger | fungi: Ya, keysize can decrease, something I default too. Ya, just being safe with chmod | 16:51 |
| fungi | pabelanger: we can be pretty well guaranteed that file already exists since that's how the jenkins master connects to the slave so it can install the slave agent which initiates the job in the first place | 16:51 |
| fungi | pabelanger: not to mention it seems odd that you're chmod'ing authorized_keys (which is already expected to exist) but not known_hosts (which probably won't yet) | 16:53 |
| *** gyee has joined #openstack-infra | 16:53 | |
| *** abregman has quit IRC | 16:53 | |
| fungi | anyway, i'd be willing to just trust the default umask in this situation | 16:54 |
| bkero | wouldn't it be safer to (unmask; touch)? | 16:54 |
| *** davideagnello has quit IRC | 16:54 | |
| fungi | bkero: of course it would | 16:54 |
| *** ashleighfarnham has joined #openstack-infra | 16:54 | |
| bkero | *umask | 16:54 |
| * bkero was taught that pattern at race condition school :) | 16:55 | |
| fungi | bkero: i'm mostly asking why even bother. these are throw-away machines which are running arbitrary code with sudo permissions to begin with | 16:55 |
| * bkero nod | 16:55 | |
| bkero | Security at all layers ;) | 16:55 |
| fungi | so worrying over the security of the jenkins user's ssh configuration on a disposable machine is a tad pedantic. if the argument is that it's attempting to set a good example in case that pattern gets copied elsewhere, i can understand it, but as you point out it's really not doing an ideal job of that either | 16:56 |
| fungi | still, i feel like simplicity wins out in this case | 16:57 |
| bkero | Since these machines are running potentially untrusted code as root, security seems like a moot point | 16:58 |
| bkero | So 'security' in this context would be to contain and restrict permission from | 16:58 |
| *** yamamoto has quit IRC | 16:59 | |
| *** hdd has joined #openstack-infra | 16:59 | |
| fungi | also the job-template name is a misleadingly general. i wouldn't have expected {pipeline}-{name}-dsvm-functional-{ostype}{job-suffix} to be specific to ansible functional jobs | 16:59 |
| fungi | though that's not the fault of this particular change | 16:59 |
| *** sputnik13 has quit IRC | 17:00 | |
| *** salv-orlando has quit IRC | 17:01 | |
| anteaya | morning pleia2 | 17:04 |
| *** sridhar_ram has joined #openstack-infra | 17:08 | |
| *** hashar is now known as hasharMeeting | 17:08 | |
| *** thiagop has joined #openstack-infra | 17:11 | |
| sdague | jeblair: so is there a gerrit restart coming? or is this slowness unrelated to the memory leakage and more related to the proxy | 17:13 |
| openstackgerrit | Merged openstack-infra/system-config: Add searchlight channel to statusbot list https://review.openstack.org/207519 | 17:13 |
| *** tzn has quit IRC | 17:14 | |
| *** ashleighfarnham has quit IRC | 17:16 | |
| jeblair | sdague: i do not know; my guess is it's the usual memory issues with the additional novelty of them presenting in a slightly different manner due to the proxy change. i also suspect that we could probably further tune the proxy to mitigate the actual errors (and get us back to simply "it's just slow"). | 17:16 |
| jeblair | sdague: but i haven't tried to verify that on prod yet | 17:16 |
| sdague | ok | 17:17 |
| jeblair | perhaps we should just restart gerrit and save proxy tuning for another day? | 17:17 |
| sdague | yeh, that would at least make doing reviews less painful | 17:17 |
| jeblair | fungi: thoughts ^? | 17:17 |
| sdague | especially given that the upgrade is probably going to change all these issues | 17:17 |
| jeblair | very true | 17:17 |
| *** jtomasek has joined #openstack-infra | 17:18 | |
| fungi | jeblair: fine by me | 17:19 |
| jeblair | will do | 17:19 |
| fungi | and agreed that there's little point in revisiting socket tuning until after the upgrade | 17:19 |
| fungi | we'll almost certainly have to do it again then anyway | 17:19 |
| jeblair | restarting gerrit now | 17:20 |
| *** tzn has joined #openstack-infra | 17:20 | |
| *** openstackstatus has quit IRC | 17:20 | |
| jeblair | it's back up | 17:20 |
| *** openstackstatus has joined #openstack-infra | 17:21 | |
| *** ChanServ sets mode: +v openstackstatus | 17:22 | |
| *** HeOS has quit IRC | 17:22 | |
| *** scheuran has quit IRC | 17:23 | |
| *** Piet has quit IRC | 17:24 | |
| *** samuelBartel has quit IRC | 17:25 | |
| *** tzn has quit IRC | 17:26 | |
| *** mrmartin has quit IRC | 17:27 | |
| openstackgerrit | Gal Sagie proposed openstack-infra/system-config: Add #openstack-dragonflow to channel logging https://review.openstack.org/249643 | 17:28 |
| *** yamamoto has joined #openstack-infra | 17:28 | |
| *** xyang1 has joined #openstack-infra | 17:30 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/project-config: Add a test-accounts src-{mame} exp job https://review.openstack.org/257448 | 17:30 |
| *** ashleighfarnham has joined #openstack-infra | 17:30 | |
| zaro | morning | 17:31 |
| *** harlowja_at_home has joined #openstack-infra | 17:31 | |
| openstackgerrit | Andreas Scheuring proposed openstack-infra/system-config: Correct Logstash Query Example https://review.openstack.org/257501 | 17:31 |
| anteaya | morning zaro | 17:31 |
| *** mrmartin has joined #openstack-infra | 17:32 | |
| *** jcoufal_ has quit IRC | 17:32 | |
| openstackgerrit | Andreas Scheuring proposed openstack-infra/system-config: Correct Logstash Query Example https://review.openstack.org/257501 | 17:33 |
| *** e0ne has joined #openstack-infra | 17:34 | |
| *** Piet has joined #openstack-infra | 17:35 | |
| *** doug-fish has quit IRC | 17:35 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Fix toggle ci button for CS2 on Gerrit 2.11 https://review.openstack.org/235079 | 17:35 |
| *** unicell1 has quit IRC | 17:35 | |
| *** doug-fish has joined #openstack-infra | 17:35 | |
| openstackgerrit | Andreas Scheuring proposed openstack-infra/system-config: Correct Logstash Query Example https://review.openstack.org/257501 | 17:36 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Add a subscriber_id parameter to various GET API endpoints https://review.openstack.org/257505 | 17:36 |
| *** sridhar_ram has quit IRC | 17:36 | |
| *** sridhar_ram has joined #openstack-infra | 17:37 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Update plugin version for Gerrit 2.11 on review.o.o https://review.openstack.org/245457 | 17:37 |
| fungi | pabelanger: left comments on your 256995 change. it has at least one error, but I think you can also simplify it greatly given the lack of security concerns for the environment in which it's being performed | 17:38 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Add a subscriber_id parameter to various GET API endpoints https://review.openstack.org/257505 | 17:40 |
| *** tzn has joined #openstack-infra | 17:40 | |
| *** doug-fish has quit IRC | 17:40 | |
| *** tonytan4ever has quit IRC | 17:40 | |
| *** lykinsbd has quit IRC | 17:41 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Add a page to view all of your subscriptions https://review.openstack.org/257509 | 17:41 |
| *** apoorvad has joined #openstack-infra | 17:42 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Enable SSH access for ansible-role-jobs https://review.openstack.org/256995 | 17:42 |
| clarkb | morning | 17:43 |
| *** jordanP has quit IRC | 17:43 | |
| pabelanger | fungi: Yup, just did update based on your suggestions | 17:43 |
| clarkb | jeblair: not sure if you caught it but on the ml yesterday I noted that ovh dvestack-trusty hosts have almost 800MB less total memory than rax | 17:44 |
| *** davideagnello has joined #openstack-infra | 17:44 | |
| clarkb | and ~500MB less than hpcloud, seems to be why magnum tests fail there running out of memory (they boot large 2GB VMs for some reason) | 17:45 |
| *** yamamoto has quit IRC | 17:46 | |
| * clarkb finds a link | 17:47 | |
| fungi | yeah, that strikes me as odd. maybe ovh will be able to shed some light on it | 17:47 |
| *** ociuhandu has quit IRC | 17:47 | |
| clarkb | http://lists.openstack.org/pipermail/openstack-dev/2015-December/082143.html | 17:47 |
| fungi | we're not booting a different kernel than in hpcloud, so in theory reserved kernel memory should be roughly equivalent. not sure what else would be subtracted from that, so seems likely they've defined the memory limit smaller instead | 17:48 |
| *** davideagnello has quit IRC | 17:49 | |
| fungi | i mean, in hpcloud we're booting something like 30gb flavors and then confining them with kernel parameters to begin with | 17:49 |
| sdague | hmmm any idea why 257303,1 is still runing at top of gate 2.5 hrs later? | 17:49 |
| sdague | did something go wonky with some of it's test nodes and it get restarted? | 17:49 |
| clarkb | sdague: or a change aead of it failed | 17:49 |
| sdague | clarkb: no, it was promoted | 17:50 |
| fungi | i promoted it so there shouldn't have been any subsequent reset | 17:50 |
| openstackgerrit | Andrew Woodward proposed openstack-infra/project-config: Create a seperate puppet job-template and jobs for puppet-ceph https://review.openstack.org/256088 | 17:50 |
| sdague | fungi: yep | 17:50 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Bump Gerrit version on review-dev.o.o https://review.openstack.org/256838 | 17:50 |
| *** ashleighfarnham has quit IRC | 17:50 | |
| fungi | i see a few changes in the check pipeline that might have seen similar impact | 17:51 |
| fungi | we might have nodes winking out in one of our providers | 17:51 |
| openstackgerrit | Niall Bunting proposed openstack/requirements: Bump glanceclient to use the newest version https://review.openstack.org/257512 | 17:51 |
| sdague | yeh, about 1/2 of openstack is queued up behind that fix at the moment, will be good to have it finally land | 17:52 |
| *** harlowja_at_home has quit IRC | 17:53 | |
| *** ashleighfarnham has joined #openstack-infra | 17:53 | |
| *** hasharMeeting has quit IRC | 17:54 | |
| *** hashar has joined #openstack-infra | 17:55 | |
| *** pradk has quit IRC | 17:57 | |
| openstackgerrit | Merged openstack-infra/puppet-gerrit: Do not proxy /server-status https://review.openstack.org/257464 | 17:58 |
| openstackgerrit | Merged openstack-infra/project-config: Speed up release-critical post jobs https://review.openstack.org/257472 | 17:58 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Add a gate-bindep-fallback-ubuntu-trusty job https://review.openstack.org/256122 | 17:59 |
| clarkb | oh I know | 17:59 |
| *** ZZelle_ has joined #openstack-infra | 18:00 | |
| clarkb | the timer isn't reset when its promoted if it was already in the gate is it? | 18:00 |
| * clarkb looks in irc logs | 18:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Make install-distro-packages.sh configurable https://review.openstack.org/256632 | 18:00 |
| clarkb | hrm that isn't super helpful | 18:01 |
| *** salv-orlando has joined #openstack-infra | 18:02 | |
| *** hashar has quit IRC | 18:02 | |
| *** yamahata has joined #openstack-infra | 18:03 | |
| *** cdent has quit IRC | 18:03 | |
| *** Swanson has joined #openstack-infra | 18:03 | |
| *** salv-orlando has quit IRC | 18:03 | |
| *** salv-orlando has joined #openstack-infra | 18:03 | |
| *** jerryz has joined #openstack-infra | 18:03 | |
| *** Piet has quit IRC | 18:03 | |
| fungi | clarkb: yeah i promoted it a few minutes after HeOS_ asked in here, though i probably forgot to say anything since i was still coffeeing | 18:04 |
| *** derekh has quit IRC | 18:04 | |
| *** Piet has joined #openstack-infra | 18:04 | |
| *** sabeen1 has joined #openstack-infra | 18:04 | |
| *** zeih has quit IRC | 18:04 | |
| openstackgerrit | Merged openstack-infra/project-config: Enable SSH access for ansible-role-jobs https://review.openstack.org/256995 | 18:05 |
| *** unicell has joined #openstack-infra | 18:05 | |
| jeblair | clarkb: can you email the ovh folks? i'm not sure they'll see that message without a pointer | 18:06 |
| *** sdake has quit IRC | 18:06 | |
| *** rhallisey has quit IRC | 18:06 | |
| *** sputnik13 has joined #openstack-infra | 18:06 | |
| *** tzn has quit IRC | 18:09 | |
| clarkb | 2015-12-14 17:41:37,346 DEBUG zuul.Scheduler: Adding complete event for build: <Build 4b3ea070f86c42c3b4cd40d4bc1401dd of gate-grenade-dsvm on <Worker Unknown>> result: None | 18:09 |
| clarkb | I think thats the first set of results back for that grenade job | 18:09 |
| clarkb | took about an hour to run | 18:09 |
| clarkb | jeblair: same folks on the neutron port "leak" thread? | 18:10 |
| *** AJaeger has quit IRC | 18:11 | |
| *** AJaeger has joined #openstack-infra | 18:11 | |
| clarkb | sdague: fungi and log confirms http://logs.openstack.org/03/257303/1/gate/gate-grenade-dsvm/4b3ea07/console.html | 18:12 |
| *** sdake has joined #openstack-infra | 18:12 | |
| clarkb | http://logs.openstack.org/03/257303/1/gate/gate-grenade-dsvm/522b9f7/console.html its persistent too | 18:12 |
| *** markus_z has quit IRC | 18:12 | |
| *** rhallisey has joined #openstack-infra | 18:12 | |
| fungi | clarkb: oh yeah, looks like something's tanking the worker for the grenade jobs | 18:13 |
| *** rhallisey_ has joined #openstack-infra | 18:13 | |
| nibalizer | crinkle: ping | 18:13 |
| *** rhallisey_ has quit IRC | 18:13 | |
| *** rhallisey_ has joined #openstack-infra | 18:13 | |
| clarkb | fungi: ya though the first time ti ran it worked on ovh so it may be less specific to grenade | 18:13 |
| *** ashleighfarnham has quit IRC | 18:13 | |
| *** pradk has joined #openstack-infra | 18:13 | |
| fungi | clarkb: all on ovh? maybe they're running out of memory? ;) | 18:13 |
| clarkb | and either ovh/devstack related | 18:13 |
| fungi | oh | 18:13 |
| nibalizer | there is some fun DNS stuff in infra-cloud | 18:13 |
| jeblair | clarkb: yeah | 18:13 |
| fungi | nibalizer: is that actual fun or sarcastic fun? | 18:14 |
| nibalizer | well... I think its interesting | 18:14 |
| fungi | don't get me wrong, i likes me some good ol' fashioned sarcastic fun too | 18:14 |
| nibalizer | mordred: how precisely did you fix `facter fqdn` on the infracloud controller | 18:14 |
| *** HeOS has joined #openstack-infra | 18:14 | |
| nibalizer | because I suspect you changed /etc/hosts and that has broken some things | 18:14 |
| *** rhallisey_ has quit IRC | 18:15 | |
| clarkb | nibalizer: did nova stop working? | 18:15 |
| nibalizer | clarkb: no nova worked fine | 18:15 |
| clarkb | nova <3's its hostname | 18:15 |
| crinkle | nibalizer: hi | 18:15 |
| nibalizer | but it tried to lookup 'controller01.blahblah' on every call to glance... which wasn't what it thought | 18:15 |
| clarkb | nibalizer: nova did that? | 18:16 |
| nibalizer | so glacne was listening on 15.numbers, and in the past when nova looked up 'controlle01.blah' it would get that 15 adress | 18:16 |
| nibalizer | but now it is getting the 127.0.1.1 adress | 18:16 |
| clarkb | ah | 18:16 |
| clarkb | in any case be careful changing hostnames under nova | 18:16 |
| nibalizer | so for a while there nova image-list would not work but glance image-list would | 18:16 |
| *** mgrosz_ has joined #openstack-infra | 18:16 | |
| nibalizer | which is pretty amusing | 18:16 |
| *** peristeri has joined #openstack-infra | 18:16 | |
| nibalizer | bind_host = controller00.hpuswest.ic.openstack.org | 18:16 |
| nibalizer | is what is in our glance config | 18:17 |
| nibalizer | so when i restarted glance, it began listening on 127.0.1.1 insteag, which means that now glance image-list doesn't work abut nova image-list does | 18:17 |
| nibalizer | because one goes through nova | 18:17 |
| nibalizer | crinkle: did you get all that? I can try to explain better if its confusing | 18:18 |
| crinkle | i think /etc/hosts needs 15.184.52.4 instead of 127.0.1.1 | 18:18 |
| *** otter768 has joined #openstack-infra | 18:19 | |
| nibalizer | I think thats how you had it before yea | 18:19 |
| crinkle | will that break `hostname -f`? | 18:19 |
| nibalizer | if we change it back everything goes back to working I think | 18:19 |
| *** trown|outttypeww is now known as trown | 18:20 | |
| crinkle | nibalizer: want to try it? | 18:20 |
| nibalizer | I'm not sure thats the right approach for a final state, but I like the idea of just putting it back to working as an experiment | 18:21 |
| AJaeger | fungi, pabelanger (reading backscroll) : Should we rename {pipeline}-{name}-dsvm-functional-{ostype}{job-suffix} to {pipeline}-{name}-ansible-dsvm-functional-{ostype}{job-suffix} ? Will have in the job then twice ansible... | 18:22 |
| nibalizer | we shouldn't have to hardcode a dns entry into /etc/hosts, that should be in the global dns | 18:22 |
| nibalizer | crinkle: i want to try it, do you want to put the changes in place or me? | 18:22 |
| *** vilobhmm11 has joined #openstack-infra | 18:22 | |
| crinkle | nibalizer: go for it | 18:22 |
| AJaeger | dims, could you review https://review.openstack.org/#/c/256929/, please? | 18:22 |
| fungi | AJaeger: i'm not convinced it's an issue, just seems like it has a possibility to collide with other job names | 18:23 |
| *** Sukhdev has quit IRC | 18:23 | |
| *** otter768 has quit IRC | 18:23 | |
| pabelanger | AJaeger: fungi: Ya, we could make it safer I suspect | 18:23 |
| AJaeger | pabelanger: could you review https://review.openstack.org/257031, please? | 18:23 |
| nibalizer | crinkle: rgr | 18:23 |
| fungi | (we have a lot of dsvm-functional jobs which are not for ansible projects, with presumably more on the way) | 18:23 |
| AJaeger | and grepping I found {pipeline}-heat-dsvm-functional-{engine}-{datastore}{job-suffix} {pipeline}-ceilometer-dsvm-functional-{backend}{job-suffix}'... | 18:24 |
| nibalizer | crinkle: okay i made the change | 18:24 |
| nibalizer | so restart glance, but not nova ya? | 18:24 |
| * nibalizer restarts glance | 18:25 | |
| AJaeger | fungi: so, the ansible ones are the only ones with {name} in it, the rest look like jobs | 18:25 |
| *** ashleighfarnham has joined #openstack-infra | 18:25 | |
| pabelanger | AJaeger: Sure | 18:25 |
| nibalizer | so now we're back to 15.184.52.4:9292 (from netstat) and we had 127.0.1.1:9292 before | 18:26 |
| AJaeger | wrong, the rest are templates as well... | 18:26 |
| pabelanger | AJaeger: dsvm-ansible-functional WFM since we do tox -eansible-functional | 18:26 |
| crinkle | nibalizer: all the others are listening on 0.0.0.0, why is glance being different? | 18:26 |
| nibalizer | i think its beause of that bind_address line I pasted | 18:26 |
| clarkb | fungi: I wonder, rax is xen is it possible that kernel overhead is not counted against hte VM and is deduped in fun ways in the hypervisor? | 18:26 |
| nibalizer | I was going to ask you if we should just have glance listening on 0.0.0.0 | 18:26 |
| nibalizer | crinkle: 'bind_host = controller00.hpuswest.ic.openstack.org | 18:27 |
| nibalizer | from glance-api.conf | 18:27 |
| *** mudassirlatif has joined #openstack-infra | 18:27 | |
| AJaeger | pabelanger: let me see how large that change is ;) | 18:27 |
| *** thiagop has quit IRC | 18:27 | |
| nibalizer | so I think it does a lookup on that then binds to the addr | 18:27 |
| fungi | clarkb: possible, and possible that we end up using space outside the boot configured cap for the kernel on hpcloud | 18:27 |
| sdague | is it possible neutron + ovh is not happy at all? | 18:27 |
| fungi | since there is additional raw memory available | 18:27 |
| sdague | do we know neutron jobs work on ovh ? | 18:27 |
| clarkb | sdague: right | 18:28 |
| clarkb | sdague: it was tested as working yes | 18:28 |
| clarkb | nova net was the only thing t have trouble on ovh | 18:28 |
| sdague | hmmmm | 18:28 |
| sdague | http://logs.openstack.org/03/257303/1/gate/gate-grenade-dsvm/4b3ea07/console.html | 18:28 |
| crinkle | nibalizer: yeah it should probably be changed to 0.0.0.0, nova and neutron both have that | 18:28 |
| nibalizer | ++ | 18:28 |
| pabelanger | AJaeger: okay, so you are saying bashate should be merged to pep8? | 18:29 |
| pabelanger | for ansible | 18:29 |
| sdague | if I had to guess on that, I'd say that there is an ovs collision of some sort | 18:29 |
| AJaeger | pabelanger: that was done already | 18:29 |
| sdague | oh, wait, that's not neutron | 18:29 |
| AJaeger | pabelanger: there's not a single ansible bashate job anymore, so the filter is not needed | 18:29 |
| crinkle | nibalizer: i'm wondering if i did that on purpose because of ssl | 18:30 |
| fungi | sdague: it looks like when the grenade-forward job reran in rax it succeeded, and the regular grenade job is running on bluebox now so taking way longer than expected | 18:30 |
| pabelanger | AJaeger: let me check with ansible-openstack, I don't think bashate is actually tested any more. Since the projects I looked at still have tox -ebashate | 18:30 |
| fungi | it's just at the upgrading phase when jenkins thinks it's had enough time to complete | 18:30 |
| sdague | fungi: yeh, the standard grenade job must have failed more than once I think | 18:30 |
| AJaeger | pabelanger: it's not in project-config anymore if I looked correctly | 18:31 |
| * anteaya bets AJaeger looked correctly | 18:31 | |
| pabelanger | AJaeger: Ya, same | 18:31 |
| crinkle | nibalizer: going to try changing the bind_host manually to see what it does | 18:31 |
| odyssey4me | pabelanger AJaeger as per your request we removed the bashate jobs, but still need to figure out how to merge them into the pep8 job | 18:31 |
| *** vgridnev has joined #openstack-infra | 18:31 | |
| *** lykinsbd has joined #openstack-infra | 18:31 | |
| pabelanger | Ah | 18:31 |
| fungi | sdague: well, this particular run has been going 45 minutes and just gotten into the upgrades phase | 18:31 |
| pabelanger | AJaeger: so, is the plan to merge bashate for all projects into pep8? | 18:32 |
| crinkle | nibalizer: glance image-list and nova image-list both work when bind_host is 0.0.0.0 | 18:32 |
| AJaeger | pabelanger: the goal is to have a single lint target and that's pep8 | 18:32 |
| nibalizer | did you change back /etc/hosts? | 18:32 |
| sdague | fungi: ah, yeh, I guess ovh is slower | 18:32 |
| AJaeger | unless somebody does a global rename ;) | 18:32 |
| crinkle | nibalizer: i did just now | 18:32 |
| fungi | sdague: bluebox in this case | 18:32 |
| sdague | also, what's up with the pypi wheel mirror? | 18:32 |
| nibalizer | ok so thats the real test | 18:32 |
| pabelanger | AJaeger: fair enough | 18:32 |
| nibalizer | if that works then I think we make that change | 18:33 |
| sdague | because that would save us about 20 minutes on these jobs | 18:33 |
| crinkle | nibalizer: still seems to work | 18:33 |
| pabelanger | q: is hound.o.o production ready? DNS not working here | 18:33 |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/system-config: Add docs for renaming an IRC channel https://review.openstack.org/244860 | 18:33 |
| AJaeger | pabelanger: do you know why some ansible jobs use ansible-role-functional-jobs (using centos7) tempalte and others use check/gate function jobs from trusty? | 18:33 |
| nibalizer | pabelanger: codesearch.o.o is the thing | 18:33 |
| nibalizer | hound was always a dev box | 18:33 |
| odyssey4me | pabelanger AJaeger I'd be happy to go along with a global convergence of lint jobs into a job named something like 'lint', but the tox config to combine the checks are a mystery well beyond me :p | 18:33 |
| nibalizer | there is an ipv6 addr for hound that escaped purging, I'll look into that | 18:34 |
| jroll | sorry to be annoying, but can someone get this project-config change landed for me please? trying to make our gate not terrible and this is in a weird middle place https://review.openstack.org/#/c/255942/ | 18:34 |
| AJaeger | odyssey4me: give me a repo name that I can make a prototype patch for... | 18:34 |
| pabelanger | AJaeger: Mostly for the stuff I have, I haven't made trusty stable yet. Was hoping to fix that up this week | 18:34 |
| *** sabeen3 has joined #openstack-infra | 18:34 | |
| pabelanger | AJaeger: then move it into ansible-functional-jobs | 18:34 |
| pabelanger | AJaeger: however, the openstack-ansible team, only support trusty ATM | 18:35 |
| pabelanger | nibalizer: thanks | 18:35 |
| clarkb | fungi: sdague if grenade is bypassing the config to run nova net on the bridge interface and is instead running against eth0 I would expect those failures | 18:35 |
| clarkb | the problem in ovh with nova net is that ovh uses a /32 address on eth0 with the default gateway living (necessarily) outside of that range | 18:36 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: Fix glance-api bind host https://review.openstack.org/257525 | 18:36 |
| clarkb | nova net has decided this is invalid when it sets up networking and kills eth0 | 18:36 |
| crinkle | nibalizer: ^ | 18:36 |
| fungi | clarkb: yeah, jobs should not assume anything about host interface configuration | 18:36 |
| *** sabeen1 has quit IRC | 18:36 | |
| fungi | as we have basically no control over it | 18:36 |
| clarkb | the "fix" for this was to always setup an internal bridge networking like we do on multinode and run nova net against that | 18:36 |
| sdague | jroll: +A | 18:36 |
| jroll | thanks sdague | 18:37 |
| clarkb | fungi: right nova net should handle this better but no one seems to want to deal with nova net anymore | 18:37 |
| AJaeger | odyssey4me, pabelanger: If I rename the functional jobs, I can templatize them directly if that's beneficial. Could we loook together at https://etherpad.openstack.org/p/ansible-jobs ? | 18:37 |
| clarkb | so we work around it it in the test itself | 18:37 |
| *** dizquierdo has joined #openstack-infra | 18:38 | |
| sdague | clarkb: so does that mean every time the grenade job lands on ovh we've been doing this reset? | 18:39 |
| *** ig0r_ has joined #openstack-infra | 18:39 | |
| fungi | greghaynes: mordred: on the wheel mirror front, i've heard a couple times that you ended up having to rethink part of the design... i still don't see any comments in https://review.openstack.org/165240 indicating why we shouldn't move forward with that change. is it safe to approve? | 18:40 |
| clarkb | sdague: maybe? I don't know if this is a new issue due to some cange in grenade | 18:40 |
| nibalizer | pabelanger: okay should be really gone now | 18:40 |
| clarkb | I was 99% sure we fixed it when we got ovh workin | 18:40 |
| sdague | there have been very few changes to grenade recently | 18:41 |
| openstackgerrit | Merged openstack-infra/project-config: Switch all ironic jobs to use devstack plugin https://review.openstack.org/255942 | 18:41 |
| nibalizer | crinkle: lgtm | 18:42 |
| *** david-lyle_ has joined #openstack-infra | 18:42 | |
| *** bharathm has joined #openstack-infra | 18:42 | |
| fungi | clarkb: sdague: 18:13 <clarkb> fungi: ya though the first time ti ran it worked on ovh so it may be less specific to grenade | 18:42 |
| fungi | that implies it was working there at some point? | 18:42 |
| greghaynes | fungi: wip'd it | 18:43 |
| greghaynes | with some info | 18:43 |
| clarkb | oh right we eve nhave logs of it working | 18:43 |
| * clarkb goes to refind them | 18:43 | |
| greghaynes | krotscheck: with the wheel mirror deal in unified mirrors - have your patches totally superceded https://review.openstack.org/#/c/165240/ ? | 18:43 |
| clarkb | oh it was aborted | 18:43 |
| clarkb | http://logs.openstack.org/03/257303/1/gate/gate-grenade-dsvm/b75106e/console.html | 18:44 |
| fungi | clarkb: maybe it's concistently failing in one ovh region and consistently working in the other ovh region? | 18:44 |
| clarkb | fungi: I think it may have aborted prior to the network blip so my assertion may have been bad | 18:44 |
| fungi | greghaynes: thanks, people keep asking how soon we'll have the wheel mirrors and that's the change i'd been referencing previously | 18:44 |
| *** david-lyle has quit IRC | 18:45 | |
| nibalizer | any core want to review https://review.openstack.org/#/c/257525/ since its live on the systems anyways | 18:45 |
| *** rhedlind has joined #openstack-infra | 18:45 | |
| greghaynes | fungi: Yea, I had to unload it on krotscheck for now so I am not entirely clear on the current state | 18:45 |
| *** ybathia has joined #openstack-infra | 18:46 | |
| *** baoli has quit IRC | 18:47 | |
| *** gordc has quit IRC | 18:48 | |
| sdague | yay, the grenade job successed! | 18:49 |
| anteaya | yay | 18:49 |
| EmilienM | hi Infra, I have an outstanding request today, latest bundler release is broken and it breaks all our unit tests jobs - I'll have a patch coming in project-config I might need an eye on it, please | 18:50 |
| fungi | mergy-mergy! | 18:50 |
| *** bexelbie has quit IRC | 18:51 | |
| fungi | clarkb: sdague: yeah, quite a few of the overdue changes in the check pipeline look to have more recently started grenade jobs, suggesting that we have some number of them tanking | 18:52 |
| rhedlind | One of my CI jobs has been failing consistently since Dec 12, 2015 7:22:07 AM during devstack setup. I would appreciate if someone could take a moment and look at http://paste.openstack.org/show/481862/ to see if you have any pointers where to look or if there is another channel I should ask the question. | 18:52 |
| fungi | so the theory is a good one | 18:52 |
| krotscheck | greghaynes, fungi: Can I answer this on monday? I'm on a volcano drinking umbrella drinks. | 18:52 |
| greghaynes | krotscheck: haha, ofc | 18:52 |
| *** davideagnello has joined #openstack-infra | 18:52 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Rename and templatize ansible functional jobs https://review.openstack.org/257530 | 18:53 |
| fungi | krotscheck: get back in^H^Hto the volcano | 18:53 |
| AJaeger | odyssey4me, pabelanger: Here's the change, I added and used both centos7 and trusty templates | 18:53 |
| AJaeger | odyssey4me, pabelanger: 257530 aboe ^ | 18:53 |
| AJaeger | s/aboe/above/ | 18:53 |
| fungi | rhedlind: https://bitbucket.org/hpk42/tox/issues/294/tox-23-regression | 18:54 |
| sdague | rhedlind: you have tox 2.3 installed, you need to install 2.3.1 or 2.2 | 18:54 |
| clarkb | jeblair: mail sent to ovh with you cc'd | 18:54 |
| *** davideagnello has quit IRC | 18:54 | |
| *** davideagnello has joined #openstack-infra | 18:55 | |
| openstackgerrit | Jim Rollenhagen proposed openstack-infra/project-config: Default to ipxe for ironic jobs https://review.openstack.org/257531 | 18:55 |
| AJaeger | pleia2: thanks for reviewing my pre-christmas cleaning! | 18:55 |
| *** doug-fish has joined #openstack-infra | 18:55 | |
| openstackgerrit | Eric Brown proposed openstack-infra/project-config: Add bandit-docs job to gate of Bandit project https://review.openstack.org/257533 | 18:55 |
| pleia2 | AJaeger: thanks for doing pre-christmas cleaning! | 18:55 |
| rhedlind | fungi, sdague: Ah, I will create a new image and try again. Awesome help! Thanks! | 18:56 |
| asselin_ | rhedlind, which version of tox? there was a recently release that broke evrything | 18:56 |
| AJaeger | pleia2: you'll find some more of these small changes of mine ;) | 18:57 |
| rhedlind | asselin_: nodepool decided to create a new image and I guess it pulled in tox 2.3. For some reason it did not keep the older working image, so I have nothing to revert back to. | 18:57 |
| asselin_ | rhedlind, it builds an image every day....so the old one was probably deleted already | 18:58 |
| clarkb | sdague: fungi https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/devstack-vm-gate.sh#n166 is the function grenade branch calls to setup networking for nova net | 18:58 |
| clarkb | sdague: fungi I am guessingthat the additions to the localrc there are ignored by the first pass of grenade? still digging | 18:58 |
| anteaya | rhedlind: I believe there was a conversation in friday's log about nodepool images and settings to allow it to keep the latest image and the one before that | 18:59 |
| rhedlind | asselin_: from what I understand it will keep the old image around until the new one has a successful run? | 18:59 |
| fungi | rhedlind: nodepool knows nothing about your jobs or their success/failure rates | 18:59 |
| rhedlind | anteaya: ok | 18:59 |
| asselin_ | rhedlind, successfully built | 18:59 |
| flip214 | question about the Nova CI: I've got a change (https://review.openstack.org/#/c/256292/) marked with a "Depends" on an os-brick change, but the CI still just fetches os-brick=0.6.0 and doesn't care about the dependency. | 18:59 |
| flip214 | so the CI fails. does that mean that os-brick needs to do a new release? | 19:00 |
| fungi | flip214: is the job installing os-brick from a requirements list, or is it a job which is designed to install os-brick from source? | 19:00 |
| asselin_ | flip214, I'm not sure depends-on works with libraries | 19:00 |
| *** changbl has quit IRC | 19:00 | |
| sdague | clarkb: localrc for grenade isn't going to be used in the devstacks | 19:00 |
| rhedlind | fungi, asselin_: I guess I confused succesfully built and successful run. | 19:00 |
| *** doug-fish has quit IRC | 19:00 | |
| clarkb | sdague: I think I see that it needs to do similar to what setup multinode networking does | 19:01 |
| flip214 | fungi: If I read the logfiles correctly, the CI run just fetches the library ... as asselin_ said. | 19:01 |
| clarkb | sdague: will have a patch in a few | 19:01 |
| rhedlind | I will try a new image and see if that solves the issue. Thanks all for the help! | 19:01 |
| *** pvaneck has joined #openstack-infra | 19:01 | |
| *** doug-fish has joined #openstack-infra | 19:01 | |
| *** changbl has joined #openstack-infra | 19:01 | |
| fungi | flip214: depends-on doesn't magically turn jobs into integration tests if they weren't already designed to be integration tests. oslo has some jobs which explicitly cross-test some requirements from source, though i think they use devstack-gate to track what should come from source and what from released packages | 19:02 |
| *** sambetts is now known as sambetts-afk | 19:02 | |
| *** bpokorny_ has joined #openstack-infra | 19:02 | |
| flip214 | fungi: thanks. so I'll ask Hemna about a new os-brick release including my changes. | 19:02 |
| flip214 | thank you, good night. | 19:02 |
| sdague | clarkb: ok, cool | 19:02 |
| *** doug-fis_ has joined #openstack-infra | 19:03 | |
| sdague | yeh, if this manages to get down to the nova.conf, grenade won't touch it later, it's just never getting passed that far | 19:03 |
| *** gordc has joined #openstack-infra | 19:03 | |
| AJaeger | odyssey4me: please check https://review.openstack.org/257536 | 19:04 |
| *** doug-fis_ has quit IRC | 19:04 | |
| sdague | ok, walking away from the computer for a while, will check back in later | 19:04 |
| *** doug-fis_ has joined #openstack-infra | 19:04 | |
| asselin_ | rhedlind, just saw on the mailing list: tox 2.3.1 was released that supposedly fixes the issue | 19:04 |
| *** gordc has quit IRC | 19:04 | |
| *** doug-fish has quit IRC | 19:05 | |
| *** bpokorny has quit IRC | 19:05 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: pin bundler to 1.10.6 https://review.openstack.org/257465 | 19:06 |
| *** sparkyco_ has joined #openstack-infra | 19:06 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Add a gate-bindep-fallback-ubuntu-trusty job https://review.openstack.org/256122 | 19:06 |
| EmilienM | crinkle: please look https://review.openstack.org/257465 | 19:06 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Fix grenade on OVH with nova net https://review.openstack.org/257537 | 19:08 |
| clarkb | sdague: ^ something likethat | 19:08 |
| *** tonytan4ever has joined #openstack-infra | 19:09 | |
| *** gordc has joined #openstack-infra | 19:10 | |
| anteaya | clarkb: I believe that commit message has a typo | 19:11 |
| AJaeger | EmilienM: could you check some cleanup I did, please? https://review.openstack.org/#/c/257030/ and https://review.openstack.org/257029 | 19:11 |
| clarkb | anteaya: ya, will update once I hVe some job results showing this fixes the problem | 19:11 |
| EmilienM | AJaeger: sure, looking now | 19:11 |
| anteaya | clarkb: yup | 19:12 |
| anteaya | clarkb: I don't know enough to comment on the point of the patch, trying to follow along and learn | 19:13 |
| AJaeger | thanks EmilienM ! | 19:13 |
| clarkb | zaro: I should go ahead and approve https://review.openstack.org/#/c/256838/2 now so that we can test that war prior to upgrade ya? | 19:14 |
| EmilienM | AJaeger: +1, nice catches... | 19:14 |
| *** alex_xu has quit IRC | 19:15 | |
| AJaeger | EmilienM: I checked that each regex matches a job... | 19:16 |
| zaro | clarkb: yeah, that makes sense. although puppet is turned off on review-dev right now. | 19:16 |
| *** alex_xu has joined #openstack-infra | 19:16 | |
| EmilienM | AJaeger: *useful* | 19:16 |
| zaro | clarkb: i can do manual run once it's apprv though | 19:16 |
| clarkb | zaro: sounds good | 19:16 |
| fungi | zaro: clarkb lgtm too | 19:16 |
| fungi | confirmed that's our latest build | 19:16 |
| EmilienM | fungi, AJaeger: I have an outstanding request for today, our CI is broken: https://review.openstack.org/#/c/257465/ | 19:17 |
| *** EricGonczer_ has quit IRC | 19:17 | |
| AJaeger | EmilienM: If there would be a good way to get the list of all configured jobs out of zuul, we could automate it.... | 19:17 |
| *** yamahata has quit IRC | 19:17 | |
| AJaeger | EmilienM: looking... | 19:17 |
| EmilienM | AJaeger: how do you do in the meantime? | 19:17 |
| zaro | clarkb: this one too if you wouldn't mind https://review.openstack.org/#/c/257073/ | 19:17 |
| *** yamahata has joined #openstack-infra | 19:17 | |
| *** bpokorny_ has quit IRC | 19:18 | |
| clarkb | zaro: there are two with the same timestamp on tarballs.o.o how do we know which is the correct one? | 19:18 |
| *** bpokorny has joined #openstack-infra | 19:18 | |
| fungi | clarkb: highest patch number | 19:19 |
| AJaeger | EmilienM: run tox -e zuul, grep/sed all job names out of it - and then parse zuul/layout.yaml regexes on those... | 19:19 |
| AJaeger | and then manual inspection why a regex did not work and fix it | 19:19 |
| fungi | clarkb: more patches means more better | 19:19 |
| zaro | it's the 3fefa35, built from here: https://gerrit.googlesource.com/plugins/javamelody/+/stable-2.11 | 19:19 |
| *** tzn has joined #openstack-infra | 19:19 | |
| nibalizer | crinkle: mtreinish jeblair for infra-cloud I think we're back to mysql max connections | 19:19 |
| AJaeger | EmilienM: nothing I like to do too often ;) | 19:19 |
| EmilienM | AJaeger: :( | 19:19 |
| nibalizer | show processlist seems to have 152 things in it(though I am no mysql expert) | 19:19 |
| clarkb | fungi: for the plugins we don't get patc hbumbers | 19:19 |
| nibalizer | so I think we need https://review.openstack.org/#/c/256897/ | 19:20 |
| *** EricGonczer_ has joined #openstack-infra | 19:20 | |
| fungi | clarkb: oh, you were talking about the javamelody change, i thought you were asking about the gerrit war build | 19:20 |
| AJaeger | EmilienM: the scripting was fun, just the manual inspection at the end not | 19:20 |
| clarkb | fungi: ya plugin jar | 19:20 |
| clarkb | zaro: gotcha so we check the commit sha of the plugin we want | 19:20 |
| fungi | zaro: i thought i already reviewed (approved?) a plugin update similar to this... was there another? | 19:20 |
| EmilienM | nibalizer: please also check https://review.openstack.org/#/c/257465/ to make sure that's fine for you | 19:21 |
| zaro | yeah, we did but newer build after all of the openstack/2.8.4 changes landed | 19:21 |
| clarkb | nibalizer: safe to just approve that because its all manual puppeting right? | 19:21 |
| fungi | zaro: okay, thanks | 19:21 |
| EmilienM | fungi: maybe our CI could run zuul and if for a rule there is no match, break the script? | 19:21 |
| clarkb | zaro: you mean 2.11.4 changes right? | 19:22 |
| EmilienM | not fungi, sorry, AJaeger | 19:22 |
| anteaya | EmilienM: is 257465 ready to merge? you had a comment you were waiting on 257421 which failed jenkins | 19:22 |
| jeblair | nibalizer: what's the controller hostname again? | 19:22 |
| zaro | clarkb: oops, you are correct | 19:22 |
| clarkb | nibalizer: well its approved so get ready :) | 19:22 |
| jeblair | nibalizer: controller00.hpuswest.ic.openstack.org | 19:22 |
| EmilienM | anteaya: the -1 is not valid anymore but I hoped crinkle or nibalizer could look before we merge it | 19:22 |
| anteaya | EmilienM: okay tell me when you are happy | 19:22 |
| EmilienM | anteaya: thanks a lot! | 19:22 |
| jeblair | clarkb, nibalizer: yes: 152 rows in set (0.00 sec) | 19:22 |
| anteaya | np | 19:22 |
| *** salv-orlando has quit IRC | 19:22 | |
| *** fawadkhaliq has quit IRC | 19:23 | |
| jeblair | where did the default of 150 come from? | 19:23 |
| clarkb | jeblair: I want to say thats ubuntu default? | 19:24 |
| jeblair | k | 19:24 |
| crinkle | jeblair: https://github.com/puppetlabs/puppetlabs-mysql/blob/51950bf2d11b071cff21bf86379e245fd08f2070/manifests/params.pp#L403 | 19:24 |
| crinkle | that may or may not be based on something in mysql's reality | 19:24 |
| jeblair | crinkle: do you know why that was chosen? | 19:24 |
| crinkle | jeblair: no idea | 19:24 |
| AJaeger | EmilienM: I haven't found a nice way to script it completely yet... | 19:24 |
| clarkb | maybe its mysql default, devstack had the same issue | 19:24 |
| clarkb | and devstack doesn't puppet | 19:24 |
| nibalizer | clarkb: correct its manual right now | 19:25 |
| crinkle | looks like it's the default in mysql https://dev.mysql.com/doc/refman/5.5/en/too-many-connections.html | 19:25 |
| jeblair | crinkle, clarkb, nibalizer, SpamapS, mordred: i think that a much higher default in puppet would make sense too | 19:25 |
| EmilienM | anteaya: go! | 19:26 |
| jeblair | crinkle, clarkb, nibalizer, SpamapS, mordred: because we know that openstack keeps lots of connections open under normal circumstances | 19:26 |
| *** tsg has quit IRC | 19:26 | |
| *** woodster_ has joined #openstack-infra | 19:26 | |
| clarkb | jeblair: I agree | 19:27 |
| clarkb | (jsut like devstack just does it for you) | 19:27 |
| nibalizer | note thats not our mysql module its the upstream one | 19:27 |
| nibalizer | but we could PR it | 19:27 |
| SpamapS | indeed, there is very little reason to keep max_connections lower than 1000 in any real situation. | 19:27 |
| clarkb | nibalizer: I think jeblair means in the openstack module deployment of mysql | 19:27 |
| jeblair | nibalizer: oh, yeah, what clarkb said | 19:28 |
| * anteaya goes | 19:28 | |
| jeblair | nibalizer: sorry, i wasn't reading closely enough to notice that was just plain old mysql not "openstack-puppet configuring mysql" | 19:28 |
| SpamapS | one thread can eat most of yur RAM anyway with a not-so-difficult-to-construct set of tables and queries. | 19:28 |
| *** sparkyco_ has quit IRC | 19:28 | |
| nibalizer | jeblair: as it stands there is one puppet module per service for openstack | 19:29 |
| jeblair | nibalizer: but considering what SpamapS says, heh, maybe it should just be the puppet mysql default :) | 19:29 |
| crinkle | i don't think the puppetlabs mysql module should change its max_connections default, it's using the default for mysql itself | 19:29 |
| nibalizer | there isn't really a single composition module that groups them and uses them | 19:29 |
| nibalizer | our infracloud module is that | 19:29 |
| nibalizer | i think anyways | 19:29 |
| crinkle | right | 19:29 |
| *** dizquierdo has quit IRC | 19:30 | |
| jeblair | nibalizer: okay, then setting it in our infracloud module sounds like the thing for now. [then when everyone uses our infracloud module they will have a sane configuration] | 19:30 |
| openstackgerrit | David Lyle proposed openstack-infra/project-config: Adding trove-dashboard repo and jobs https://review.openstack.org/256605 | 19:30 |
| nibalizer | heh | 19:30 |
| *** hashar has joined #openstack-infra | 19:31 | |
| nibalizer | jeblair: hunner bmjen and others would be happy to chat with us about changing the upstream default im sure, maybe you can follow up with them? | 19:31 |
| *** bharathm has quit IRC | 19:31 | |
| *** Piet has quit IRC | 19:31 | |
| nibalizer | EmilienM: can you offer an explanation for why infras puppet syntax jobs seem to be passing? | 19:31 |
| EmilienM | nibalizer: they use puppet3 | 19:32 |
| EmilienM | puppet4 sorry | 19:32 |
| SpamapS | It's not 100% cut and dry that 1000 will always work. | 19:32 |
| *** dprince has quit IRC | 19:32 | |
| EmilienM | puppet3 jobs are broken | 19:32 |
| SpamapS | If you have a small server and a very small app, 1000 is going to break you. | 19:33 |
| SpamapS | But I can't think of any real situation where that matters. | 19:33 |
| openstackgerrit | Alon Marx proposed openstack-infra/system-config: Allow to disable cron jobs https://review.openstack.org/250374 | 19:33 |
| SpamapS | So, yeah, for OpenStack... 1000 seems right. For general mysql.... it might not be possible. | 19:34 |
| SpamapS | I'd tend to trust mysql itself for the more general case. | 19:34 |
| SpamapS | even though I personally have never worked on an app that small. :) | 19:35 |
| jeblair | wow, zuul is close to going idle. | 19:35 |
| openstackgerrit | Merged openstack-infra/shade: Pedantic spelling correction https://review.openstack.org/257452 | 19:36 |
| *** Piet has joined #openstack-infra | 19:38 | |
| crinkle | nibalizer: they use different jobs, infra's use 'puppet parser validate' instead of 'bundle exec rake syntax' | 19:40 |
| nibalizer | crinkle: ok | 19:41 |
| nibalizer | seems like we should probably centralize that, but hey at least we aren't broken | 19:41 |
| crinkle | nibalizer: i vaguely remember there being a reason for the different jobs, but i don't recall it now | 19:42 |
| *** sridhar_ram has quit IRC | 19:42 | |
| openstackgerrit | Yan Ning proposed openstack-infra/project-config: Propose use seperate job builder for monasca experimental job https://review.openstack.org/257546 | 19:42 |
| *** tonytan4ever has quit IRC | 19:43 | |
| fungi | jeblair: it seems wrongly idle... we had a gate reset nearly 30 minutes ago and the gate pipeline has only just started getting workers assigned behind it. i think that was nodepool going idle while zuul worked through events for half an hour | 19:44 |
| fungi | it's just now cleared out the couple of changes that were at the top of the gate | 19:45 |
| fungi | so i think it's back into normal operation again | 19:45 |
| openstackgerrit | JJ Asghar proposed openstack-infra/project-config: osops-tools-generic: add bashate job https://review.openstack.org/229043 | 19:45 |
| jeblair | fungi: aha | 19:46 |
| jeblair | perhaps a zuul reload then | 19:46 |
| fungi | jeblair: possibly. a job change for ironic merged about an hour ago | 19:47 |
| flwang | AJaeger: ping re distil | 19:47 |
| AJaeger | hi flwang ! | 19:47 |
| fungi | though i haven't looked at zuul's log to correlate, it would match behavior we've seen around config reloads in the past | 19:47 |
| flwang | AJaeger: do you have a few mins to discuss the repo? | 19:48 |
| AJaeger | flwang: I wonder why you setup distil in OpenStack infra at all, there's never been any activity around it. | 19:48 |
| AJaeger | flwang: yes | 19:48 |
| flwang | AJaeger: it's a long story | 19:48 |
| flwang | we was trying to collaborate with cloudkitty guys about the rating project | 19:49 |
| flwang | and distil is used by us internally | 19:49 |
| *** ajmiller has joined #openstack-infra | 19:50 | |
| openstackgerrit | Paul Michali proposed openstack-infra/project-config: Make VPN constraints jobs voting https://review.openstack.org/255992 | 19:50 |
| flwang | but after some long discussion with cloudkitty guys, we realized their current functions are still far away for our production env | 19:50 |
| flwang | so we would like to contribute our code back to community | 19:50 |
| AJaeger | flwang: so, will there be collaboratoin going forward? Then let's at least fix the repository so that it's useable... | 19:50 |
| clarkb | I am going to hav eto pop out and run errands soon, the fix for ovh grenade is up and should be tested soon, sent ovh email about VM memory, and hopefully got the outstanding gerrit change smerged | 19:50 |
| flwang | AJaeger: yep, i think so | 19:50 |
| clarkb | anything else I should be looking at before I AFK? | 19:51 |
| *** lucasagomes is now known as lucas-dinner | 19:51 | |
| flwang | and Distil do have some great capabilities that cloudkitty missing | 19:51 |
| *** MarkAtwood has quit IRC | 19:51 | |
| *** e0ne has quit IRC | 19:51 | |
| flwang | like the intergration with OpenERP, etc | 19:51 |
| flwang | yep, i will fix the repo for sure | 19:51 |
| flwang | i'm really sorry for the 'sorry state' | 19:52 |
| openstackgerrit | Paul Michali proposed openstack-infra/project-config: Make LBaaS constraints jobs voting https://review.openstack.org/256391 | 19:52 |
| flwang | AJaeger: btw | 19:52 |
| *** e0ne has joined #openstack-infra | 19:52 | |
| flwang | what's current policy if there are more than one projects focus on same domain? | 19:52 |
| fungi | clarkb: too bad it didn't end up with ovh for its grenade job on the first shot. presumably we want to recheck it a bit? | 19:53 |
| flwang | like distil and cloudkitty, both of them targeting for rating service | 19:53 |
| fungi | clarkb: nothing else to my knowledge | 19:53 |
| AJaeger | flwang: then humour me with approving https://review.openstack.org/257549, please | 19:53 |
| clarkb | fungi: ya we should recheck until we get it to run there, but making sure we still work on not ovh is important too | 19:53 |
| fungi | clarkb: agreed | 19:53 |
| AJaeger | flwang: with the big tent model: Basically everybody is welcome | 19:53 |
| fungi | i'll pump it with a recheck or two when i see it come back with results | 19:53 |
| *** mgrosz_ has quit IRC | 19:54 | |
| flwang | AJaeger: cool, that's awesome | 19:54 |
| flwang | AJaeger: thanks for the heads up | 19:54 |
| *** dprince has joined #openstack-infra | 19:54 | |
| *** maishsk has joined #openstack-infra | 19:54 | |
| openstackgerrit | Jim Rollenhagen proposed openstack-infra/project-config: Fix typo in ironic plugin https://review.openstack.org/257551 | 19:54 |
| fungi | c | 19:54 |
| flwang | i will approve the patch and upload some new patches | 19:54 |
| AJaeger | flwang: I'll remove the tests from distil in our CI infrastructure since distil has no tox.ini file and therefore will never pass | 19:54 |
| fungi | hah, this is not my mutt window! | 19:54 |
| flwang | AJaeger: ok, no problem. i will fix the tox issue and add the tests back when it's ready | 19:55 |
| jroll | can someone fast track this please, I managed to nuke ironic's gate because I'm apparently terrible at typing https://review.openstack.org/#/c/257551/ | 19:55 |
| jroll | sorry :( | 19:55 |
| fungi | nibalizer: thanks for sending the maintenance reminder! | 19:55 |
| jroll | fungi: ^^ if you have a sec | 19:56 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove distil jobs https://review.openstack.org/257552 | 19:57 |
| *** annegentle has quit IRC | 19:57 | |
| openstackgerrit | Paul Michali proposed openstack-infra/project-config: Make LBaaS constraints jobs voting https://review.openstack.org/256391 | 19:57 |
| fungi | jroll: i had somehow never noticed that syntax. i guess +=$ is a bash variable concatenation operator? i see it's used further up so presumably it works | 19:57 |
| AJaeger | flwang: here's the change ^ (257552) that you then can revert once your repo is setup... | 19:57 |
| *** electrofelix has quit IRC | 19:57 | |
| jeblair | fungi has just verbalized the same mental process i went through | 19:58 |
| fungi | heh | 19:58 |
| *** annegentle has joined #openstack-infra | 19:58 | |
| *** hongbin has joined #openstack-infra | 19:58 | |
| sdague | so what's up with the nodepool graphs where it looks like everything is starving then popping back up? | 19:58 |
| jroll | fungi: this was the first time I'd seen it too :) | 19:58 |
| fungi | followed by digging in the terribly formatted bash(1) manpage | 19:58 |
| fungi | and resorting to manual testing instead | 19:58 |
| openstackgerrit | Merged openstack/python-jenkins: Deprecated tox -downloadcache option removed https://review.openstack.org/256711 | 19:59 |
| *** annegentle has quit IRC | 19:59 | |
| *** tsg has joined #openstack-infra | 19:59 | |
| openstackgerrit | Merged openstack-infra/project-config: Add translation jobs for neutron-vpnaas https://review.openstack.org/257043 | 19:59 |
| *** annegentle has joined #openstack-infra | 19:59 | |
| openstackgerrit | Paul Michali proposed openstack-infra/project-config: Make FWaaS constraints jobs voting https://review.openstack.org/256702 | 19:59 |
| *** dizquierdo has joined #openstack-infra | 20:00 | |
| fungi | oh! is it just a += operator and then $'\n' getting prepended to the rest of the addition? | 20:00 |
| jroll | fungi: yeah, and TIL the $ makes it translate the \n as expected | 20:00 |
| fungi | jroll: yep, confirmed localls | 20:00 |
| fungi | locally | 20:00 |
| fungi | that's what it's doing | 20:01 |
| *** ybathia has quit IRC | 20:01 | |
| fungi | anteaya has already beaten me to approving it anyway | 20:01 |
| jroll | yep, ty anteaya and jeblair | 20:01 |
| *** ildikov has joined #openstack-infra | 20:02 | |
| fungi | sdague: in the past it's been event pileups during a zuul config reload | 20:02 |
| *** eil397 has joined #openstack-infra | 20:02 | |
| AJaeger | yeah, that $'\n' syntax is unusual but get's cargo-copied around lately... | 20:02 |
| openstackgerrit | Merged openstack-infra/project-config: Add translation jobs for neutron-fwaas https://review.openstack.org/257045 | 20:03 |
| openstackgerrit | Merged openstack-infra/system-config: Bump Gerrit version on review-dev.o.o https://review.openstack.org/256838 | 20:03 |
| fungi | sdague: so zuul ends up not requesting new jobs run for a while, and nodepool starts trending toward its idle state as a result | 20:03 |
| sdague | fungi: ok, seems like it's happened twice in the last 24 hrs? | 20:03 |
| fungi | sdague: yeah, the graph at least shows two somewhat lengthy events along those lines | 20:04 |
| *** tonytan4ever has joined #openstack-infra | 20:04 | |
| openstackgerrit | Merged openstack-infra/system-config: Update javamelody plugin on review-dev.o.o https://review.openstack.org/257073 | 20:05 |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Bump mysql connections to 1024 https://review.openstack.org/256897 | 20:05 |
| fungi | seems like the job volume is unusually high for us given it's mid-december. previous years it's been mostly dead around here by now | 20:06 |
| clarkb | nibalizer: zaro ^ that should get things moving for your respective tasks | 20:06 |
| AJaeger | fungi, some of that might come from the grenade and tox failures the last days - now everybody rechecks... | 20:06 |
| fungi | might be | 20:07 |
| *** dtardivel has quit IRC | 20:07 | |
| nibalizer | clarkb: thanks | 20:08 |
| nibalizer | crinkle: im gonna run puppet on controller00 | 20:09 |
| * AJaeger waves good night | 20:09 | |
| *** AJaeger has quit IRC | 20:09 | |
| crinkle | nibalizer: okay good luck | 20:09 |
| *** david-lyle_ has quit IRC | 20:10 | |
| nibalizer | since https://review.openstack.org/#/c/257525/ isn't landed it's gonna break our glance right? | 20:11 |
| *** eil397 has quit IRC | 20:11 | |
| *** yonglihe has quit IRC | 20:11 | |
| *** placko_ has joined #openstack-infra | 20:11 | |
| *** eil397 has joined #openstack-infra | 20:12 | |
| hongbin | clarkb: Hi, do you have time? Want to follow up with you regarding http://lists.openstack.org/pipermail/openstack-dev/2015-December/082143.html | 20:12 |
| clarkb | hongbin a little time left before I need to go | 20:12 |
| *** david-lyle has joined #openstack-infra | 20:12 | |
| hongbin | clarkb: OK. Just a general question. How to debug this issue further. Do you have any suggestion? | 20:13 |
| openstackgerrit | Merged openstack-infra/gerritbot: Fix reconnection method https://review.openstack.org/257174 | 20:14 |
| clarkb | hongbin what portion do you want to debug further? | 20:14 |
| hongbin | clarkb: I want to eliminate this failure from magnum gate | 20:14 |
| clarkb | hongbin simplest thing would be to stop using 2gb ram flavor | 20:15 |
| hongbin | clarkb: I tried 1GB flavor before, it failed with the same error | 20:15 |
| *** salv-orlando has joined #openstack-infra | 20:15 | |
| *** asselin__ has joined #openstack-infra | 20:15 | |
| *** Sukhdev has joined #openstack-infra | 20:16 | |
| clarkb | hongbin how many VMs are you booting at one time? | 20:16 |
| *** Piet has quit IRC | 20:16 | |
| hongbin | clarkb: 2 | 20:16 |
| *** _nadya_ has joined #openstack-infra | 20:16 | |
| clarkb | ok thats why then | 20:16 |
| clarkb | we have less than 2GB | 20:16 |
| clarkb | do you need 2 at a time? | 20:17 |
| hongbin | clarkb: Yes, it is a must | 20:17 |
| openstackgerrit | Merged openstack-infra/project-config: Drop quantum from ENABLED_SERVICES https://review.openstack.org/250913 | 20:17 |
| clarkb | why? | 20:17 |
| *** asselin has quit IRC | 20:17 | |
| hongbin | we setup a kubernetes cluster, which requires one vm as master, another as worker | 20:17 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: Enable network.service with systemd https://review.openstack.org/257230 | 20:18 |
| openstackgerrit | Merged openstack-infra/project-config: Remove icehouse/juno handling from networking-ofagent https://review.openstack.org/256924 | 20:18 |
| *** Piet has joined #openstack-infra | 20:18 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove icehouse/juno from networking-bgvpn https://review.openstack.org/256926 | 20:19 |
| clarkb | so there are basically 4 options 1. reduce memory used by openstack 2. reduce memory used by tests 3. figure out why we have less than 8gb and increase to 8gb 4. go bigger than 8gb | 20:19 |
| clarkb | 4 is likely not possible | 20:19 |
| clarkb | I have emailed ovh to work on 3 | 20:19 |
| clarkb | that means your options are 1 and 2 | 20:19 |
| *** otter768 has joined #openstack-infra | 20:19 | |
| hongbin | Good points | 20:20 |
| clarkb | hongbin can kubetnetes not all in one? | 20:20 |
| clarkb | I would start with that on 1GB vm | 20:20 |
| hongbin | clarkb: Kubernetes can be all in one, but that is not what we are trying to test | 20:20 |
| openstackgerrit | Merged openstack-infra/project-config: Remove icehouse/juno branch filter from magnum https://review.openstack.org/256928 | 20:21 |
| hongbin | clarkb: I could try option #1 and #2, it would be handy if I can get a test VM from ovh | 20:21 |
| openstackgerrit | Merged openstack-infra/project-config: Further juno cleanup: ampq1, zmq plugins https://review.openstack.org/256929 | 20:21 |
| vilobhmm11 | clarkb : why can;t the scheduler pick computes with more available resources than picking the wrong ones ? | 20:21 |
| vilobhmm11 | hongbin : ^^ | 20:21 |
| clarkb | vilobhmm11 because we area multicloud application | 20:22 |
| clarkb | the tests must run across the clouds | 20:22 |
| *** bharathm has joined #openstack-infra | 20:22 | |
| clarkb | hongbin what are you trying to test? | 20:22 |
| rajinir_ | hi I'm using the new repo and readme to setup the thirdparty CI. Would anyone be able to help me debug the nodepool-> jenkins slaves launch error ? | 20:22 |
| clarkb | that magnum can make a kubernetes cluster? that is doable with allinone imo | 20:23 |
| hongbin | clarkb: I will try to reproduce the error if I can | 20:23 |
| *** baoli has joined #openstack-infra | 20:23 | |
| *** dimtruck is now known as zz_dimtruck | 20:23 | |
| clarkb | hongbin to reproduce just restrict memory to the value I gave you | 20:23 |
| clarkb | you dont need an ovh vm for that | 20:23 |
| hongbin | clarkb: OK. Make sense | 20:24 |
| *** baoli has quit IRC | 20:24 | |
| hongbin | I can try that | 20:24 |
| *** otter768 has quit IRC | 20:24 | |
| *** baoli has joined #openstack-infra | 20:24 | |
| hongbin | clarkb: Thanks for the suggestions | 20:26 |
| clarkb | no problem but ya 1and 2 are good places to start, I am working 3 too | 20:27 |
| *** zz_dimtruck is now known as dimtruck | 20:28 | |
| *** aysyd has quit IRC | 20:28 | |
| hongbin | ack. Please sync up with us if any process on #3. Greatly appreciate that :) | 20:29 |
| *** dims has joined #openstack-infra | 20:29 | |
| clarkb | will do | 20:31 |
| *** dkranz has quit IRC | 20:32 | |
| *** dkranz has joined #openstack-infra | 20:33 | |
| anteaya | rajinir_: please ask your question and if someone can help they will | 20:33 |
| anteaya | rajinir_: if you have build logs or stacktraces please put them in a paste service, such as paste.openstack.org and bring the url of the paste back to channel | 20:34 |
| rajinir_ | Nodepool is launching the node in CI Cloud, but times out before it can ssh. Are there any special config from CI-Node => Devstack ? | 20:35 |
| rajinir_ | anteaya : thanks | 20:35 |
| openstackgerrit | Merged openstack-infra/project-config: puppet: pin bundler to 1.10.6 https://review.openstack.org/257465 | 20:38 |
| *** rhallisey_ has joined #openstack-infra | 20:39 | |
| *** rhallisey_ has quit IRC | 20:40 | |
| *** _nadya_ has quit IRC | 20:40 | |
| *** rhallisey_ has joined #openstack-infra | 20:40 | |
| *** rhallisey_ has quit IRC | 20:41 | |
| sdague | docs publishing issue | 20:41 |
| sdague | http://docs.openstack.org/developer/nova/unit_tests.html | 20:41 |
| sdague | is a thing, that is findable in google | 20:42 |
| sdague | but it's not in our source tree | 20:42 |
| sdague | so it's not updatable | 20:42 |
| *** mrmartin has quit IRC | 20:42 | |
| fungi | sdague: probably it got deleted/renamed in the source tree at some point | 20:42 |
| sdague | fungi: probably did | 20:42 |
| mordred | ya. we don't rsync --delete docs | 20:42 |
| sdague | we don't purge everything? | 20:42 |
| mordred | because of tags | 20:42 |
| *** kushal has quit IRC | 20:42 | |
| sdague | hmmm... so basically it's all orphans all the time? | 20:43 |
| fungi | sdague: the ftp publisher that uploads documentation to the rackspace cloudsites service hosting docs.o.o only knows new state, doesn't know old state and try to clean up | 20:43 |
| mordred | yup | 20:43 |
| *** hongbin has left #openstack-infra | 20:43 | |
| *** apoorvad has quit IRC | 20:43 | |
| sdague | ok, so that makes me kind of a sad panda | 20:43 |
| mordred | yah | 20:43 |
| sdague | so people will just have the wrong information forever | 20:44 |
| mordred | we can probably manually delete that file | 20:44 |
| mordred | but the general problem is not one that we've solved | 20:44 |
| fungi | yeah, manual deletes are possible, the docs team usually handles it | 20:44 |
| fungi | i think | 20:44 |
| sdague | right, but it's kind of weird to have this automatic publishing pipeline | 20:44 |
| mordred | sdague: also, you could add a unit_tests.rst that's blank | 20:44 |
| mordred | sdague: totally. it's not ideal | 20:44 |
| openstackgerrit | Tomasz 'Zen' Napierala proposed openstack-infra/project-config: Create new project Solar https://review.openstack.org/255983 | 20:44 |
| mordred | it just hasn't made it to solution stage yet | 20:45 |
| mordred | we have plans to stop publishing to cloud sites | 20:45 |
| sdague | fungi: not any more, the docs team has spread out the responsibility | 20:45 |
| mordred | which will give us better flexibility | 20:45 |
| sdague | so now it's a dozen plus teams | 20:45 |
| *** Sukhdev has quit IRC | 20:45 | |
| mordred | but that's on the list for after logs-to-swift | 20:45 |
| *** baoli has quit IRC | 20:45 | |
| fungi | part of the challenge is that parts of the docs site aren't (or at least weren't for a very long time) sandboxed to specific subtrees per publication, so if you wiped out everything when uploading one publication you'd blow away files still used for another publication | 20:45 |
| *** lykinsbd has quit IRC | 20:45 | |
| *** baoli has joined #openstack-infra | 20:46 | |
| mordred | honestly, easiest way to correct this psecific thing is a blank unit_test.rst file to blank it from google | 20:46 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: Enable network.service with systemd https://review.openstack.org/257230 | 20:46 |
| openstackgerrit | Ian Wienand proposed openstack-infra/glean: Fixup some logging issues https://review.openstack.org/257119 | 20:46 |
| *** baoli has quit IRC | 20:46 | |
| *** baoli has joined #openstack-infra | 20:47 | |
| fungi | keep in mind that some of the shortcomings of the current implementation were designing the automation to work around the existing state of the documentation site rather than starting fresh from the ground up | 20:47 |
| jeblair | actually the general problem _is_ one we have solved | 20:47 |
| fungi | yeah, if we can find people to work on http://specs.openstack.org/openstack-infra/infra-specs/specs/doc-publishing.html then this should no longer be an issue | 20:48 |
| jeblair | right, that ^ | 20:48 |
| jeblair | that spec is _primarily_ aimed at addressing this problem | 20:48 |
| *** dizquierdo has quit IRC | 20:49 | |
| *** EricGonc_ has joined #openstack-infra | 20:52 | |
| *** rhallisey has quit IRC | 20:52 | |
| openstackgerrit | Terry Howe proposed openstack-infra/jenkins-job-builder: Change documentation to install from pypi https://review.openstack.org/254990 | 20:53 |
| mordred | jeblair: right. I meant "implemented" not "designed solution for" by "solved" | 20:53 |
| * mordred used a bad word | 20:54 | |
| *** EricGonczer_ has quit IRC | 20:55 | |
| *** gordc has quit IRC | 20:58 | |
| openstackgerrit | Yan Ning proposed openstack-infra/project-config: Propose use seperate job builder for monasca experimental job https://review.openstack.org/257546 | 20:58 |
| *** sdake has quit IRC | 20:59 | |
| *** lykinsbd has joined #openstack-infra | 21:01 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix typo in ironic plugin https://review.openstack.org/257551 | 21:01 |
| *** david-lyle has quit IRC | 21:01 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Add a gate-bindep-fallback-ubuntu-trusty job https://review.openstack.org/256122 | 21:02 |
| anteaya | jroll: ^^ | 21:02 |
| jroll | anteaya: nice, ty for the ping :D | 21:02 |
| anteaya | welcome, hope it fixes you | 21:03 |
| *** Piet has quit IRC | 21:03 | |
| *** harlowja has quit IRC | 21:04 | |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: add legacy drivers job for keystone https://review.openstack.org/257566 | 21:05 |
| *** apoorvad has joined #openstack-infra | 21:06 | |
| *** harlowja has joined #openstack-infra | 21:07 | |
| *** david-lyle has joined #openstack-infra | 21:07 | |
| anteaya | stevemar: have you ever run a job that invokes this tox env in our gate? http://git.openstack.org/cgit/openstack/keystone/tree/tox.ini#n73 | 21:07 |
| stevemar | anteaya: nope! i should make it non-voting eh | 21:08 |
| anteaya | I would suggest that | 21:08 |
| *** baoli has quit IRC | 21:08 | |
| anteaya | just to ensure you have the tox syntax doing as you expect | 21:08 |
| *** baoli has joined #openstack-infra | 21:08 | |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: add legacy drivers job for keystone https://review.openstack.org/257566 | 21:09 |
| stevemar | anteaya: done | 21:09 |
| *** aeng has joined #openstack-infra | 21:09 | |
| * anteaya looks again | 21:09 | |
| stevemar | anteaya: i think it's as expected... | 21:09 |
| *** baoli has quit IRC | 21:09 | |
| *** baoli has joined #openstack-infra | 21:10 | |
| stevemar | anteaya: assuming it works the same way as the bandit job, i'm good with that | 21:10 |
| *** harlowja_ has joined #openstack-infra | 21:11 | |
| anteaya | stevemar: almost there, comments in patch | 21:11 |
| stevemar | danke | 21:12 |
| *** david-lyle has quit IRC | 21:12 | |
| anteaya | thanks for listening | 21:12 |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: add legacy drivers job for keystone https://review.openstack.org/257566 | 21:13 |
| stevemar | thanks anteaya | 21:13 |
| stevemar | why wouldn't i listen? | 21:13 |
| stevemar | you're the expert | 21:13 |
| anteaya | ha ha ha | 21:13 |
| anteaya | you ask the best questions | 21:13 |
| stevemar | i'm the chump just copying things around | 21:13 |
| *** dkranz has quit IRC | 21:13 | |
| *** Swami_ has quit IRC | 21:13 | |
| *** matrohon has joined #openstack-infra | 21:13 | |
| stevemar | :) | 21:13 |
| *** kgiusti has left #openstack-infra | 21:13 | |
| *** harlowja has quit IRC | 21:13 | |
| *** Swami_ has joined #openstack-infra | 21:13 | |
| anteaya | read the comment I left on the commit message too please | 21:14 |
| anteaya | you have a nice hand with your copy/paste | 21:14 |
| *** sridhar_ram has joined #openstack-infra | 21:16 | |
| stevemar | whoops | 21:17 |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: add legacy drivers job for keystone https://review.openstack.org/257566 | 21:17 |
| *** jamielennox|away is now known as jamielennox | 21:17 | |
| *** annegentle has quit IRC | 21:18 | |
| anteaya | stevemar: I am now happy | 21:18 |
| *** e0ne has quit IRC | 21:20 | |
| *** gordc has joined #openstack-infra | 21:20 | |
| *** annegentle has joined #openstack-infra | 21:22 | |
| *** baoli has quit IRC | 21:22 | |
| stevemar | yahoo! | 21:22 |
| *** dkranz has joined #openstack-infra | 21:23 | |
| *** e0ne has joined #openstack-infra | 21:23 | |
| anteaya | and your username is the right colour of purple and everything | 21:24 |
| openstackgerrit | Eric Brown proposed openstack-infra/project-config: Add bandit-docs job to gate of Bandit project https://review.openstack.org/257533 | 21:25 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Add a gate-bindep-fallback-ubuntu-trusty job https://review.openstack.org/256122 | 21:25 |
| *** mudassirlatif has quit IRC | 21:25 | |
| *** alex_xu has quit IRC | 21:25 | |
| *** Piet has joined #openstack-infra | 21:26 | |
| sdague | jroll: hmmm gate-tempest-dsvm-ironic-agent_ssh at 75% fail rate and climbing | 21:26 |
| sdague | http://tinyurl.com/p8t8osb | 21:26 |
| *** jffische has joined #openstack-infra | 21:27 | |
| jroll | sdague: a project-config typo nuked the gate for a couple hours | 21:27 |
| *** alex_xu has joined #openstack-infra | 21:27 | |
| sdague | jroll: it's still an issue, I just got a fail back 10 minutes ago | 21:27 |
| jroll | sdague: it's fixed now https://review.openstack.org/#/c/257551/ | 21:27 |
| jroll | ugh | 21:27 |
| jroll | sorry, been eating lunch, will look | 21:27 |
| *** dprince has quit IRC | 21:28 | |
| sdague | although, I guess that could have had a good hour delay on it | 21:28 |
| jroll | yeah | 21:28 |
| jroll | it merged about half an hour ago | 21:28 |
| sdague | ok | 21:28 |
| sdague | jroll: although on kilo you remain with a huge failure rate on the old code | 21:29 |
| sdague | https://review.openstack.org/#/c/256469/ I've been trying to get that one in to actually decrease kilo failures for a couple of days | 21:30 |
| jroll | sdague: oh? | 21:30 |
| jroll | sigh, same old timeout crap | 21:30 |
| jroll | I'm working on that right now | 21:31 |
| sdague | clarkb: http://logs.openstack.org/46/247246/8/gate/gate-grenade-dsvm-multinode/86c3de1//logs/ - no subnode appears to have been allocated | 21:32 |
| *** sridhar_ram has quit IRC | 21:33 | |
| *** rhallisey has joined #openstack-infra | 21:34 | |
| regXboi | sdague: how's your request queue looking? I've got a devstack question for you :) | 21:34 |
| *** mudassirlatif has joined #openstack-infra | 21:35 | |
| *** sdake has joined #openstack-infra | 21:35 | |
| *** mudassirlatif has quit IRC | 21:36 | |
| *** Piet has quit IRC | 21:37 | |
| jeblair | sdague: it was allocated | 21:39 |
| jeblair | sdague: http://logs.openstack.org/46/247246/8/gate/gate-grenade-dsvm-multinode/86c3de1//console.html#_2015-12-14_21_04_04_221 | 21:39 |
| *** e0ne has quit IRC | 21:40 | |
| *** Daisy has joined #openstack-infra | 21:41 | |
| *** Piet has joined #openstack-infra | 21:42 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Upgrade the concat module https://review.openstack.org/257572 | 21:44 |
| *** Daisy has quit IRC | 21:45 | |
| *** alex_xu has quit IRC | 21:45 | |
| *** sridhar_ram has joined #openstack-infra | 21:46 | |
| *** Piet has quit IRC | 21:47 | |
| *** mudassirlatif has joined #openstack-infra | 21:47 | |
| *** Sukhdev has joined #openstack-infra | 21:48 | |
| *** alex_xu has joined #openstack-infra | 21:48 | |
| jhesketh | Morning | 21:48 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Add a gate-bindep-fallback-ubuntu-trusty job https://review.openstack.org/256122 | 21:50 |
| openstackgerrit | Steve Martinelli proposed openstack-infra/project-config: add legacy drivers job for keystone https://review.openstack.org/257566 | 21:50 |
| sdague | jeblair: hmmm it didn't get git trees? | 21:51 |
| *** baoli has joined #openstack-infra | 21:51 | |
| jeblair | sdague: it looks it it could not connect | 21:51 |
| stevemar | anteaya: new patch, was missing a bracket, doh! | 21:51 |
| *** chlong has quit IRC | 21:51 | |
| jeblair | this is over the private interface, so we don't have independent testing of connectivity (nodepool only uses public interfaces) | 21:51 |
| anteaya | stevemar: doh | 21:52 |
| anteaya | morning jhesketh | 21:52 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/system-config: Add more logstash timestamp formats https://review.openstack.org/251518 | 21:53 |
| *** pradk has quit IRC | 21:54 | |
| *** jtomasek has quit IRC | 21:55 | |
| zaro | clarkb, fungi: /etc/puppet/hiera.yaml not found on review-dev? http://paste.openstack.org/show/481885/ | 21:56 |
| *** sdake has quit IRC | 21:56 | |
| clarkb | zaro: because its been disabled on the ansible side I think | 21:57 |
| clarkb | sdague: for some reason that isn't loading for me | 21:57 |
| zaro | looks like it got heira.yaml got moved to /etc/puppet/hieradata/production/common.yaml | 21:57 |
| fungi | clarkb: zaro: yeah, has it ever successfully run there since we split up the hiera files? | 21:57 |
| zaro | clarkb: what do i need to run to update that ? | 21:57 |
| clarkb | zaro: I don't know | 21:58 |
| clarkb | fungi: mordred is it just scp'ing the files over? | 21:58 |
| mordred | reading scrollback | 21:58 |
| mordred | hiera.yaml is in recent puppet | 21:58 |
| mordred | so because puppet is disabled on review-dev | 21:58 |
| mordred | that means that it probably has never had a hiera.yaml installed | 21:59 |
| *** ybathia has joined #openstack-infra | 21:59 | |
| mordred | zaro: no - hiera.yaml and /etc/puppet/hieradata/production/common.yaml are different things | 21:59 |
| mordred | hiera.yaml is what configures hiera lookup paths | 21:59 |
| *** tongli has joined #openstack-infra | 21:59 | |
| mordred | we need one of those on all nodes now, and the file is managed by puppet | 21:59 |
| mordred | /etc/puppet/hieradata/production/common.yaml is the file with the hiera values | 21:59 |
| *** unicell1 has joined #openstack-infra | 21:59 | |
| *** annegentle has quit IRC | 21:59 | |
| clarkb | is logs.openstack.org unworky for anyone else? | 21:59 |
| fungi | clarkb: ovh ftw! https://jenkins04.openstack.org/job/gate-grenade-dsvm/9861/console | 21:59 |
| fungi | clarkb: i was just looking at stuff on logs.o.o so doesn't seem to be broken to me anyway | 22:00 |
| zaro | mordred: i tried to do a manual puppet apply run | 22:00 |
| zaro | mordred: what do i need to do to get puppet all happy? | 22:00 |
| clarkb | fungi: heh it loads now so I dunno might've been a local issue | 22:00 |
| openstackgerrit | Spencer Krum proposed openstack-infra/nodepool: Use shade to detect ip address https://review.openstack.org/257576 | 22:00 |
| mordred | zaro: if you run puppet agent --test, it sohuld do the right things | 22:01 |
| mordred | zaro: what were you trying to do that did not work | 22:01 |
| mordred | ? | 22:01 |
| *** unicell has quit IRC | 22:02 | |
| clarkb | sdague: it had a subnode http://logs.openstack.org/46/247246/8/gate/gate-grenade-dsvm-multinode/86c3de1/console.html#_2015-12-14_21_03_54_009 | 22:02 |
| *** sdake has joined #openstack-infra | 22:02 | |
| zaro | mordred: i basically ran run_puppet.sh | 22:03 |
| clarkb | sdague: lack of logs seems to be because it broke along the way | 22:03 |
| zaro | mordred: to apply all the new changes that were recently merged for review-dev | 22:03 |
| mordred | zaro: ah - so, don't do that yet | 22:03 |
| mordred | zaro: we're not actually live with puppet apply yet | 22:03 |
| mordred | zaro: you should still run puppet agent --test to run puppet locally | 22:03 |
| mordred | zaro: and you'll need to follow the instructions on skipping the lock file | 22:04 |
| mordred | because it's disabled | 22:04 |
| clarkb | sdague: ya I don't think greande ran so no longs to collect on the subnode | 22:04 |
| zaro | mordred: wha? i thought you told me to run that maybe a week ago? | 22:04 |
| mordred | zaro: it's possible I was lying to you | 22:05 |
| zaro | mordred: runnig puppet agent --test says i should run puppet agent --enable | 22:05 |
| zaro | mordred: also it didn't bring in /etc/puppet/hiera.yaml | 22:06 |
| zaro | Notice: Skipping run of Puppet configuration client; administratively disabled (Reason: 'reason not specified'); | 22:06 |
| zaro | Use 'puppet agent --enable' to re-enable. | 22:06 |
| clarkb | nibalizer: instead of raming shade in there I think the simplest thing is going to be adding another case in the current check code for IPs | 22:07 |
| *** placko_ has quit IRC | 22:07 | |
| zaro | just ran puppet agent --enable so i guess i just wait until puppet runs? | 22:07 |
| clarkb | nibalizer: that said I would expect this to be very similar to rax since its just direct public IP | 22:07 |
| clarkb | nibalizer: so maybe we can just extend that a bit to work | 22:07 |
| nibalizer | ok ill try that | 22:08 |
| *** julim has quit IRC | 22:08 | |
| nibalizer | i dont really think that solution is at all good but its the first thing I found that worked and didn't seem to duplicate code | 22:08 |
| *** tsg has quit IRC | 22:08 | |
| mordred | zaro: now that you've run that, you can run puppet agent --test and then puppet agent --disable again | 22:08 |
| zaro | ok, puppet is doing it's thing now. | 22:10 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/project-config: Publish per-branch astara-appliance images https://review.openstack.org/256076 | 22:10 |
| clarkb | nibalizer: for addr in server.addresses.get('public', []) is rax and ovh is similar | 22:11 |
| zaro | not sure if i'm remembering incorrectly, but i think run_puppet.sh worked for me last time which was about 1-2 weeks ago. | 22:11 |
| *** tongli has quit IRC | 22:11 | |
| clarkb | nibalizer: if you do a nova show on the host what is it called on omfra cloud? | 22:11 |
| jeblair | clarkb, nibalizer: it may be a matter of just naming our network 'public' | 22:12 |
| clarkb | or Ext-Net | 22:12 |
| jeblair | clarkb: yeah that | 22:12 |
| nibalizer | provider-net network | 15.184.54.18 ? | 22:12 |
| nibalizer | is that what you were looking for? | 22:12 |
| clarkb | ya let me compare to rax really quick | 22:12 |
| crinkle | you can fix that here http://git.openstack.org/cgit/openstack-infra/puppet-infracloud/tree/manifests/controller.pp#n251 | 22:13 |
| *** matrohon has quit IRC | 22:13 | |
| jeblair | crinkle: that's so cool | 22:13 |
| clarkb | | public network | 2001:4800:7819:104:be76:4eff:fe01:f511, 104.239.140.83 | 22:13 |
| nibalizer | it loonks like we just run 'server.get('public_v4')' ? maybe I'm looking at the wrong place in the code | 22:13 |
| zaro | mordred: thanks, review-dev has been update. | 22:13 |
| clarkb | so y ajust s/provider-net/public/ | 22:14 |
| zaro | clarkb, fungi ^ | 22:14 |
| mordred | zaro: woot! | 22:14 |
| mordred | nibalizer: aroo? | 22:14 |
| anteaya | astronaut applications are open: https://www.usajobs.gov/GetJob/ViewDetails/423817000 | 22:14 |
| *** trown is now known as trown|outttypeww | 22:14 | |
| *** vilobhmm11 has quit IRC | 22:14 | |
| clarkb | nibalizer: crinkle or Ext-Net (I kinda prefer public) | 22:14 |
| jeblair | clarkb, nibalizer: yeah, 'public' or 'Ext-Net' will work equally well as long as there is no network named 'private'. | 22:14 |
| anteaya | note: Frequent travel may be required | 22:15 |
| *** vilobhmm11 has joined #openstack-infra | 22:15 | |
| jeblair | if there is a network called 'private', 'public' would be better. | 22:15 |
| mordred | uhm - for infra cloud ... how about if we set router:external to True too? | 22:15 |
| mordred | yes, I know that is not exactly what that flag is supposed to be used for | 22:15 |
| mordred | BUT - we also don't care about per-user public networks | 22:16 |
| *** vilobhmm11 has quit IRC | 22:16 | |
| clarkb | mordred: sure, but it also won't fix nodepool | 22:16 |
| clarkb | so its a nop ut one worht making priobably | 22:16 |
| cbader | I am having an issue with a multinode_setup.sh not being copied to a multinode vm I get file not found. anyone seen this? | 22:16 |
| mordred | oh. right. because we haven't finished shade-nodepool | 22:16 |
| mordred | so nevermind | 22:16 |
| mordred | sigh | 22:16 |
| *** vilobhmm11 has joined #openstack-infra | 22:16 | |
| * mordred realy needs to finish that patch | 22:16 | |
| openstackgerrit | Merged openstack-infra/infra-specs: mark centralize-release-tagging spec as implemented https://review.openstack.org/251990 | 22:16 |
| clarkb | cbader: do oyu have links to logs? | 22:16 |
| *** vilobhmm11 has quit IRC | 22:16 | |
| nibalizer | i'd also like to land https://review.openstack.org/#/c/257525/ if possible | 22:16 |
| clarkb | cbader: from nodepool | 22:17 |
| *** vilobhmm11 has joined #openstack-infra | 22:17 | |
| nibalizer | since its live on the systems and I have to keep manuallying it whenever I run puppet | 22:17 |
| openstackgerrit | Merged openstack-infra/infra-specs: add "complete the reviewable release automation" https://review.openstack.org/245907 | 22:17 |
| clarkb | nibalizer: approved | 22:17 |
| *** vilobhmm11 has quit IRC | 22:17 | |
| *** vilobhmm11 has joined #openstack-infra | 22:17 | |
| clarkb | cbader: but if I had to guess you don't have that script in your nodepool scripts so after nodepool copies the scripts over and tries to execute that one it fails | 22:18 |
| *** vilobhmm11 has quit IRC | 22:18 | |
| *** vilobhmm11 has joined #openstack-infra | 22:18 | |
| cbader | clarkb, yes nodepool keeps failing with log saying "stdout: bash: ./multinode_setup.sh: No such file or directory" then deletes building and start all over again. | 22:19 |
| clarkb | cbader: yup make sure you have that script in your nodepool scripts dir | 22:20 |
| clarkb | they get copied over and run so if that script doesn't exist then it can'trun | 22:20 |
| nibalizer | clarkb: ty | 22:20 |
| cbader | clarkb, I was able to ssh in the vm for a second and look in the /opt/nodepool-scripts directory and can't see the file | 22:20 |
| clarkb | cbader: right os make sure its present on your nodepool host | 22:21 |
| cbader | clarkb, where are they copied from the nodepool server of is it part of the image that is build on the provider? | 22:21 |
| *** otter768 has joined #openstack-infra | 22:21 | |
| clarkb | cbader: wherever your nodepool config says they are | 22:21 |
| clarkb | for us its /etc/nodepool/scripts iirc | 22:21 |
| *** thorst_ has quit IRC | 22:22 | |
| *** alivigni has quit IRC | 22:22 | |
| cbader | clarkb, I have it in that directory also | 22:22 |
| *** tsg has joined #openstack-infra | 22:22 | |
| *** thorst_ has joined #openstack-infra | 22:22 | |
| zaro | yikes, we just lost the toggleci button on review-dev. | 22:22 |
| clarkb | zaro: do you need to restart/touch the file | 22:23 |
| clarkb | oh wait I didn't approve it on friday because I didn't want to do that | 22:23 |
| clarkb | zaro: so it likely just overwrote with the old thing | 22:23 |
| clarkb | cbader: so the script exists where your nodepoold runs? | 22:24 |
| clarkb | cbader: is it readable by nodepoold? is it also +x | 22:24 |
| zaro | clarkb: ahh, let me check that | 22:24 |
| cbader | clarkb, yes | 22:24 |
| *** peristeri has quit IRC | 22:24 | |
| *** tzn has quit IRC | 22:25 | |
| clarkb | cbader: I am not sure then would have to see the logs to track it further | 22:25 |
| *** otter768 has quit IRC | 22:25 | |
| cbader | clarkb, here is pastbin of log error nodepool.log http://paste.openstack.org/show/481892/ | 22:26 |
| cbader | clarkb, bottom of log. don't get it | 22:26 |
| *** gildub has joined #openstack-infra | 22:26 | |
| *** eharney has joined #openstack-infra | 22:27 | |
| clarkb | cbader: do you have the debug logs? | 22:27 |
| *** tzn has joined #openstack-infra | 22:28 | |
| cbader | clarkb, yes I will get it. | 22:28 |
| *** tzn has quit IRC | 22:28 | |
| *** tzn has joined #openstack-infra | 22:28 | |
| clarkb | cbader: ok reading the source I see another way it may fail, it does the copy at image build time so if your image was built prior to having those files the image wouldn't include it | 22:29 |
| clarkb | in that case rebuild your imgae | 22:29 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-infracloud: Rename network to 'public' https://review.openstack.org/257590 | 22:30 |
| clarkb | ok logstash-worker20 survived the weekend with 3 log workers | 22:30 |
| nibalizer | crinkle: jeblair clarkb ^^ there is a patch | 22:30 |
| *** thorst_ has quit IRC | 22:31 | |
| nibalizer | I'm going to be really impressed if puppet can just apply that and dtrt | 22:31 |
| clarkb | nibalizer: +2 | 22:31 |
| zaro | clarkb: yep, that was it. toggle-ci is working again. | 22:31 |
| cbader | clarkb, Ok I understand. We had an issue with not being able to build images on this repomasterci machine so we took one from another ci system and it doesn't have multinode enabled so we didn't get the file we needed. thank you | 22:31 |
| *** vilobhmm11 has quit IRC | 22:31 | |
| crinkle | nibalizer: it should create a new network but leave the old one hanging around | 22:31 |
| *** hashar has quit IRC | 22:31 | |
| *** vilobhmm11 has joined #openstack-infra | 22:31 | |
| crinkle | nibalizer: oh you have ensure absent, so it should just work i think | 22:32 |
| *** dkranz has quit IRC | 22:32 | |
| openstackgerrit | Merged openstack-infra/puppet-infracloud: Fix glance-api bind host https://review.openstack.org/257525 | 22:32 |
| crinkle | nibalizer: the neutron_subnet might get confused :/ | 22:34 |
| *** lykinsbd has quit IRC | 22:36 | |
| nibalizer | crinkle: yea... | 22:37 |
| nibalizer | its already complaining that the provider can't reassign | 22:38 |
| nibalizer | like before we even added this change | 22:38 |
| nibalizer | idunno we can apply this and see where it goes and fix it manually if we have to right? | 22:38 |
| *** vgridnev has quit IRC | 22:38 | |
| clarkb | nibalizer: without knowing how those puppet resources work I am willing to bet yes | 22:38 |
| *** flepied1 has quit IRC | 22:38 | |
| *** david-lyle has joined #openstack-infra | 22:39 | |
| crinkle | nibalizer: you'll have to also delete the neutron_subnet and re-add it with the new network name | 22:39 |
| nibalizer | ok | 22:39 |
| crinkle | it does complain about the allocation_pools but that's a bug in the provider https://review.openstack.org/#/c/243948/ | 22:39 |
| *** tonytan4ever has quit IRC | 22:40 | |
| anteaya | clarkb: logstash20 is running logstash 2.0? | 22:40 |
| clarkb | anteaya: yes has been for a few days to as a test node | 22:41 |
| anteaya | clarkb: any grokparse failures? | 22:41 |
| clarkb | anteaya: it is looking good so I think after gerrit upgrade madness I may go ahead and do the rest | 22:41 |
| clarkb | anteaya: not other than the ones we already know about like libvirt | 22:41 |
| clarkb | seems to be working fine | 22:41 |
| *** vilobhmm11 has quit IRC | 22:41 | |
| anteaya | clarkb: wonderful, didn't mention it since yeah gerrit upgrade, but good | 22:41 |
| anteaya | wonderful | 22:41 |
| *** vilobhmm11 has joined #openstack-infra | 22:41 | |
| anteaya | sorry do I know about a libvirt grokparse failure? | 22:42 |
| clarkb | anteaya: it is preexisting | 22:42 |
| anteaya | oh | 22:42 |
| clarkb | fungi: https://jenkins04.openstack.org/job/gate-grenade-dsvm/9861/console looks like it ran to completion | 22:43 |
| *** tzn has quit IRC | 22:45 | |
| *** amitgandhinz has quit IRC | 22:45 | |
| fungi | clarkb: yep, unfortunately it's got gate-tempest-dsvm-ironic-agent_ssh still failing on it (when did we add that as a voting job for devstack-gate?) | 22:46 |
| clarkb | fungi: which should be unrelated because ironic doesn't support nova net | 22:47 |
| clarkb | and this change only touches nova net | 22:47 |
| *** weshay has quit IRC | 22:47 | |
| jroll | sdague: is there a good way to correlate job failures with the provider they're running on? | 22:49 |
| jroll | also was ovh just recently enabled? | 22:49 |
| clarkb | jroll: its bee nseveral weeks now | 22:49 |
| jroll | ok | 22:49 |
| *** Sukhdev has quit IRC | 22:50 | |
| *** MarkAtwood has joined #openstack-infra | 22:50 | |
| jroll | maybe a red herring then, many recent ironic fails seem to be from ovh | 22:50 |
| clarkb | jroll: how ar ethey failing? | 22:51 |
| jroll | clarkb: I think you know the bug number by heart | 22:51 |
| jroll | the timeout thing | 22:51 |
| clarkb | oh that thing | 22:51 |
| clarkb | it was happening well before we used ovh | 22:51 |
| jroll | yeah, trying to dig into it hard today | 22:51 |
| clarkb | like years before | 22:51 |
| jroll | right, seems much more common now | 22:51 |
| fungi | clarkb: yeah, just pointing out that we can't easily merge this fix until either gate-tempest-dsvm-ironic-agent_ssh gets back to passing or we stop gating devstack-gate changes on it (i'm unclear on the benefit it's providing there anyway) | 22:51 |
| clarkb | jroll: gotcha | 22:51 |
| jroll | but that may just be a feeling, graphs look steady for the last couple months | 22:51 |
| jroll | fungi: I'm not sure why we need dsg gating on that job, honestly | 22:52 |
| *** mgrosz_ has joined #openstack-infra | 22:52 | |
| openstackgerrit | David Lyle proposed openstack-infra/project-config: Adding trove-dashboard repo and jobs https://review.openstack.org/256605 | 22:53 |
| *** tsg has quit IRC | 22:53 | |
| JayF | jroll: fungi: It was added because we got broken by dsg 3 times in 3 months | 22:53 |
| *** Piet has joined #openstack-infra | 22:53 | |
| JayF | jroll: fungi: At this point around 9+ months ago | 22:53 |
| JayF | iirc | 22:53 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Don't pass duplicate projects to tripleo.sh --delorean-build https://review.openstack.org/249819 | 22:54 |
| jroll | JayF: by devstack-gate? that seems like it would be hard to do | 22:54 |
| *** mgrosz_ has quit IRC | 22:54 | |
| fungi | JayF: interesting, do you recall what special devstack-gate is doing for that job that it doesn't do for other devstack-based jobs? | 22:54 |
| JayF | jroll: it was all the swift user credential stuff | 22:54 |
| jroll | that was devstack | 22:54 |
| JayF | jroll: where we depended on certain swift configs | 22:54 |
| JayF | hmm. okay if you're sure | 22:54 |
| anteaya | fungi: my git blame points to this patch as the last file that touched that ironic job in for devstack-gate, however the patch doesn't touch it so I am confused: https://review.openstack.org/#/c/236719/1 | 22:54 |
| JayF | my memory is not awesome | 22:54 |
| jroll | pretty sure | 22:54 |
| fungi | yeah, i don't think devstack-gate has a hand in configuring that aspect of the jobs | 22:55 |
| jroll | anteaya: yeah, need to go back before that, I'm looking now | 22:55 |
| fungi | anteaya: i poked back through git history a bit, but between layout file refactoring and a rename of that job itself it's not clear | 22:55 |
| anteaya | agreed | 22:55 |
| anteaya | oh if that patch reordered the lines of the file that would be sufficient for git blame to attribute a change to that patch | 22:59 |
| *** Sukhdev has joined #openstack-infra | 23:00 | |
| *** jamielennox is now known as jamielennox|away | 23:01 | |
| *** Piet has quit IRC | 23:01 | |
| *** vilobhmm11 has quit IRC | 23:01 | |
| *** jamielennox|away is now known as jamielennox | 23:01 | |
| *** vilobhmm11 has joined #openstack-infra | 23:01 | |
| *** mnencia has quit IRC | 23:01 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Stop using puppetlabs-mysql for paste https://review.openstack.org/257597 | 23:01 |
| *** Piet has joined #openstack-infra | 23:02 | |
| *** sridhar_ram has quit IRC | 23:03 | |
| fungi | i also question the benefit of running the swift-functional job there. plenty of projects have functional jobs based on devstack-gate now and they're all slightly different so likely to be broken in different ways regardless | 23:04 |
| *** chlong has joined #openstack-infra | 23:05 | |
| fungi | oh, we have neutron's running on it too, both non-voting | 23:06 |
| anteaya | crinkle: do you care that paste is still running on precise? | 23:06 |
| anteaya | crinkle: not saying you should if you don't, I'm just curious if you do | 23:06 |
| *** hurgleburgler has joined #openstack-infra | 23:07 | |
| fungi | aha, i bet what happened is it was added everywhere nonvoting, then got switched to voting when it should have remained nv on devstack-gate changes | 23:08 |
| crinkle | anteaya: I don't think my patch should see any difference whether paste is precise or trusty | 23:09 |
| anteaya | crinkle: agreed | 23:09 |
| *** gordc has quit IRC | 23:09 | |
| anteaya | crinkle: I just thought of it is all, while looking at your patch | 23:09 |
| jroll | fungi: reading the git history, it looks like this turned them on: https://review.openstack.org/#/c/134620/ | 23:10 |
| *** ashleighfarnham has quit IRC | 23:10 | |
| jroll | fungi: I'm not sure how correct the commit message is, it seems wrong at a glance but /shrug | 23:10 |
| *** maishsk has quit IRC | 23:10 | |
| anteaya | jroll: looks like that is the change | 23:11 |
| jroll | I'm going to put up a change to make it non-voting on devstack-gate, I wonder if it should be nv on devstack as well but I'll hold off on that for now | 23:12 |
| jroll | (that wondering is with the current state of the gate, if it was all green I would say keep it) | 23:12 |
| jeblair | is there a reason to run it on d-g at all? | 23:12 |
| jroll | jeblair: I tabbed back to irc to say "well, I'm just going to drop it" :P | 23:13 |
| fungi | jroll: agreed, i think there may have been a zuul layout problem adam_g was alluding to in that commit message, though it's been a while. something about how the branch limit was being added after the rule which made all -nv jobs voting:false and so inadvertently had been running that job with voting true | 23:13 |
| jeblair | jroll: ++ :) | 23:13 |
| openstackgerrit | Jim Rollenhagen proposed openstack-infra/project-config: Drop ironic jobs from devstack-gate check queue https://review.openstack.org/257602 | 23:14 |
| jroll | ^ | 23:14 |
| *** maishsk has joined #openstack-infra | 23:14 | |
| jroll | fungi: could be yeah | 23:14 |
| *** regXboi has quit IRC | 23:14 | |
| openstackgerrit | Jim Rollenhagen proposed openstack-infra/project-config: Make ironic non-voting on devstack changes https://review.openstack.org/257604 | 23:15 |
| jroll | similarly, ^ | 23:15 |
| jroll | sdague: that should help you out on kilo too | 23:16 |
| fungi | and now that the not-juno branch match has been removed from teh zuul layout for those jobs, the -nv variant should be properly nonvoting now | 23:17 |
| fungi | anyway, once 257602 lands we can recheck 257537 and then approve it to stop grenade crashing in ovh | 23:18 |
| *** doug-fis_ has quit IRC | 23:18 | |
| fungi | thanks jroll/JayF! | 23:18 |
| *** doug-fish has joined #openstack-infra | 23:18 | |
| jroll | np fungi | 23:19 |
| openstackgerrit | Merged openstack-dev/cookiecutter: Don't include file openstack-common.conf https://review.openstack.org/255256 | 23:20 |
| *** doug-fis_ has joined #openstack-infra | 23:20 | |
| *** shardy has quit IRC | 23:21 | |
| *** doug-fis_ has quit IRC | 23:21 | |
| *** doug-fis_ has joined #openstack-infra | 23:22 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:22 | |
| *** doug-fish has quit IRC | 23:23 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Add a gate-bindep-fallback-ubuntu-trusty job https://review.openstack.org/256122 | 23:24 |
| zaro | clarkb: it doesn't look like the gerrit plugin installation is working, https://git.openstack.org/cgit/openstack-infra/puppet-gerrit/tree/manifests/plugin.pp#n70 | 23:24 |
| *** xyang1 has quit IRC | 23:24 | |
| openstackgerrit | Merged openstack-infra/project-config: Drop ironic jobs from devstack-gate check queue https://review.openstack.org/257602 | 23:24 |
| zaro | when i run the puppet i don't see any errors in the log | 23:24 |
| zaro | clarkb: i'm not seeing why it would not work. everything up to that point seems to be ok. it's just not doing the last step. | 23:25 |
| clarkb | does it think its already done? | 23:25 |
| *** mgrosz_ has joined #openstack-infra | 23:26 | |
| *** mudassirlatif has quit IRC | 23:26 | |
| *** reed_ has joined #openstack-infra | 23:27 | |
| *** mgrosz_ has quit IRC | 23:27 | |
| zaro | you mean puppet? how can i tell? | 23:27 |
| *** rlandy has quit IRC | 23:28 | |
| *** thorst has joined #openstack-infra | 23:28 | |
| openstackgerrit | Merged openstack-infra/project-config: Make ironic non-voting on devstack changes https://review.openstack.org/257604 | 23:29 |
| hurgleburgler | Does anyone know how to get a group added to 'Core in: ' page list on stackalyics? | 23:30 |
| *** mudassirlatif has joined #openstack-infra | 23:30 | |
| clarkb | zaro: usually execs have an unless or only if | 23:30 |
| *** mudassirlatif has quit IRC | 23:31 | |
| clarkb | zaro: I would check if that exists and see the condition | 23:31 |
| clarkb | nibalizer: how has network renaming gone? | 23:32 |
| anteaya | hurgleburgler: this is the repo you would have to offer a patch to: http://git.openstack.org/cgit/openstack/stackalytics/tree/ | 23:33 |
| anteaya | hurgleburgler: I'm guessing you are asking about UX? | 23:33 |
| openstackgerrit | Spencer Krum proposed openstack-infra/puppet-infracloud: Rename network to 'public' https://review.openstack.org/257590 | 23:33 |
| anteaya | hurgleburgler: mostly stackalytics scrapes gerrit but UX core are not reflected in gerrit | 23:33 |
| hurgleburgler | We have a repo now, do we just need to start contributing to it? | 23:33 |
| anteaya | hurgleburgler: I think you would have to ask a stackalytics code reviewer how to add that | 23:33 |
| anteaya | hurgleburgler: well you would have to be core on that repo | 23:34 |
| anteaya | do you have a core group on that repo that reflects the UX core reviewers? | 23:34 |
| hurgleburgler | yup | 23:34 |
| fungi | argh! zaro: any clue what i'm missing with https://review.openstack.org/256122 ? i've tried every way i can think of to override the node parameter in this job definition but it keeps taking the node parameter set in the project itself instead | 23:34 |
| nibalizer | clarkb: i got puppet-lint'd | 23:34 |
| anteaya | how old? I'm not sure how often stackalytics updates | 23:34 |
| hurgleburgler | https://launchpad.net/~openstack-ux-drivers | 23:35 |
| anteaya | hurgleburgler: that isn't a gerrit group | 23:35 |
| anteaya | hurgleburgler: stackalytics scrapes gerrit | 23:35 |
| *** cbader_ has joined #openstack-infra | 23:35 | |
| anteaya | not launchpad | 23:35 |
| *** thorst has quit IRC | 23:35 | |
| hurgleburgler | Ah, gotcha | 23:36 |
| anteaya | you don't have to create a gerrit repo to get your core reviewers recognized by stackalytics | 23:36 |
| pleia2 | as far as I can tell, "core in" in stackalytics is actually based on whether you've voted +2 on something, not groups | 23:36 |
| *** jcoufal has joined #openstack-infra | 23:36 | |
| anteaya | in fact that would be a poor reason to create a repo | 23:36 |
| fungi | well, stackalytics _does_ scrape launchpad, but specifically for bug statistics | 23:36 |
| pleia2 | since it doesn't show everything I'm core in, just what I vote on (which is not everything) | 23:36 |
| anteaya | pleia2: ah thanks for that, I didn't know | 23:37 |
| anteaya | fungi: oh okay thank you, I didn't know that | 23:37 |
| fungi | and yeah, it seems to be inferring core reviewers based on where it sees +2 votes | 23:37 |
| pleia2 | anteaya: I haven't looked at the code, just a data point :) | 23:37 |
| anteaya | pleia2: yup | 23:37 |
| hurgleburgler | Its not why we created a repo, we plan on housing excepted designs in a location that can be versioned | 23:37 |
| fungi | pabelanger also may know | 23:37 |
| anteaya | hurgleburgler: so I think you need to have a chat with some stackalytics maintainers to get UX recognized | 23:37 |
| *** cbader_ has quit IRC | 23:37 | |
| anteaya | hurgleburgler: that is a good reason to create a repo | 23:38 |
| *** ZZelle_ has quit IRC | 23:38 | |
| anteaya | my understanding of the situation from past conversations with piet is that UX presents a unique problem space in that most core reviewer activity is not done on gerrit | 23:38 |
| hurgleburgler | anteaya thanks for all the info | 23:39 |
| anteaya | so any tool that evaluates gerrit activity is not going to acknowledge the existance UX core reviewers | 23:39 |
| anteaya | hurgleburgler: welcome pleia2 and fungi helped too | 23:39 |
| hurgleburgler | Yeah, we've gone back and forth about the right tooling for it | 23:39 |
| anteaya | hurgleburgler: hope you find what you need | 23:39 |
| hurgleburgler | pleia2 fungi thanks too of course :) | 23:39 |
| anteaya | hurgleburgler: if you need a bigger discussion on the problem space do consider creating an agenda item on the infra team meeting agenda | 23:39 |
| anteaya | so we can get more of the team's ideas | 23:40 |
| fungi | and yes the ux team faces similar challenges to the translation team in that most other project-teams in openstack are primarily focused around (or at least do the majority of their work in) git repositories | 23:40 |
| anteaya | also it can't hurt for the broader education of non-UX folks to understand the UX workflow better | 23:40 |
| fungi | anteaya: yep, though there's only so much to understand since it's currently invite-only access to a proprietary platform | 23:41 |
| hurgleburgler | Agreed. A lot of times the discussions completely surround the designer, whereas the developer has to eventually take the designs and work with it | 23:41 |
| fungi | we've had multiple conversations to try to come up with open tools that fit their problem space, and there are some promising possibilities but they're not going to be in place especially soon | 23:42 |
| anteaya | fungi: yes, piet and I have had long talks about using open source tooling, when he came by last week it was to ask how to get agreement for infra to host lime survey (open source) for them | 23:42 |
| hurgleburgler | anteaya i'm working on the spec for that now too | 23:43 |
| fungi | oh, is that back on again? at the summit i thought he said he'd figured out it wasn't going to be needed? | 23:43 |
| hurgleburgler | will have that up soon | 23:43 |
| anteaya | hurgleburgler: wonderful | 23:43 |
| anteaya | fungi: oh? | 23:43 |
| * fungi will keep an eye out for that spec | 23:43 | |
| anteaya | fungi: I talked to him on the Monday and we talked lime survey | 23:43 |
| fungi | i guess there's been another shift since the summit in that case | 23:43 |
| *** doug-fis_ has quit IRC | 23:43 | |
| * anteaya shrugs | 23:43 | |
| anteaya | I guess we'll have to see the spec | 23:44 |
| anteaya | in any case my sense is they are trying to use open source things | 23:44 |
| fungi | yep, but for the design wireframe markup stuff that's apparently a hard niche. hopefully some of the image markup stuff in phabricator will fit the bill | 23:45 |
| *** rajinir_ has quit IRC | 23:45 | |
| *** mudassirlatif has joined #openstack-infra | 23:45 | |
| anteaya | yes | 23:45 |
| *** Daisy has joined #openstack-infra | 23:45 | |
| anteaya | hopefully | 23:45 |
| *** mudassirlatif has quit IRC | 23:46 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Add a gate-bindep-fallback-ubuntu-trusty job https://review.openstack.org/256122 | 23:48 |
| clarkb | fungi: so we confirmed that the d-g fix for ovh grenade works on ovh? | 23:48 |
| fungi | clarkb: yes, it passed grenade on a worker in ovh | 23:49 |
| fungi | on my first recheck | 23:49 |
| fungi | luck of the draw | 23:49 |
| *** annegentle has joined #openstack-infra | 23:49 | |
| clarkb | cool so now I guess its just a decision of whether or not thats how the code should look I am going to psuh a fix for the commit messafe | 23:49 |
| fungi | clarkb: that was the job console log i had linked you to in jenkins | 23:49 |
| *** claudiub has quit IRC | 23:49 | |
| zaro | clarkb: yeah, i check that. condition seems ok to me.. unless => "test -f ${plugin_dir}/${base_plugin}" | 23:50 |
| fungi | clarkb: might want to hold off pushing until you're sure the removal of the ironic ssh job from devstack-gate's job list has been applied | 23:50 |
| clarkb | zaro: so does the file already exist? | 23:50 |
| zaro | clarkb: file does exists | 23:50 |
| clarkb | zaro: then it won't run | 23:50 |
| clarkb | fungi: good idea will wait | 23:50 |
| fungi | clarkb: merged at 23:24 utc and zuul says last configured at 23:36, so probably already safe? | 23:51 |
| zaro | clarkb: should runs if false, correct? ohh 0 is true? | 23:51 |
| *** Daisy has quit IRC | 23:51 | |
| clarkb | fungi: should be thanks | 23:52 |
| *** Daisy has joined #openstack-infra | 23:52 | |
| clarkb | zaro: 0 is true | 23:52 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Fix grenade on OVH with nova net https://review.openstack.org/257537 | 23:52 |
| zaro | ok. i think that plugin.pp needs to be redone anyways because it looks like gerrit will spaz out if you replace a plugin from under it while it's running | 23:52 |
| zaro | yeah, got it. | 23:52 |
| fungi | heh, yeah maybe just as well that it wasn't "working" yet ;_ | 23:52 |
| zaro | the plugin update needs to run while gerrit is stopped. | 23:53 |
| *** mudassirlatif has joined #openstack-infra | 23:53 | |
| zaro | fungi: will take a look at that jjb issue | 23:54 |
| *** annegentle has quit IRC | 23:54 | |
| *** Daisy has quit IRC | 23:54 | |
| *** Daisy has joined #openstack-infra | 23:55 | |
| *** ashleighfarnham has joined #openstack-infra | 23:55 | |
| *** Daisy_ has joined #openstack-infra | 23:55 | |
| *** markvoelker has quit IRC | 23:55 | |
| fungi | zaro: i'm down to just trying random things (list expansion in a job group, non-list individual parameter settings, quoted names with parameters versus non-quoted, and so on) | 23:56 |
| *** xarses has quit IRC | 23:56 | |
| fungi | problem is i see several working examples elsewhere in our configuration, but for this addition i can't seem to make it work even copying exactly what they're doing (i think) | 23:57 |
| *** jaosorior has quit IRC | 23:57 | |
| fungi | zaro: anyway, it's no rush, the gerrit upgrade prep takes precedence of course | 23:58 |
| *** sdake_ has joined #openstack-infra | 23:58 | |
| *** sdake has quit IRC | 23:58 | |
| Piet | fungi: hurgleburgler: anteaya: we eventually decided it would be easier if the OpenStack UX team had it's own survey tool and data. Of course, the infra team and foundation would have access to the data as well | 23:59 |
| *** sridhar_ram has joined #openstack-infra | 23:59 | |
| *** Daisy has quit IRC | 23:59 | |
| zaro | fungi: examples from the docs? jjb/tests folder? | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
