| *** sdake has quit IRC | 00:00 | |
| *** otter768 has quit IRC | 00:01 | |
| *** salv-orlando has joined #openstack-infra | 00:06 | |
| jeblair | the carbon cache daemon seems to have disappeared from cacti.o.o | 00:08 |
|---|---|---|
| jeblair | like, it's not on the filesystem | 00:08 |
| jeblair | we CD it, so that could be an upstream change | 00:08 |
| jeblair | oh, it's because th fs is full | 00:10 |
| jeblair | also s/cacti/graphite/ ^ | 00:11 |
| jeblair | (cacti is fine, graphite is what i meant) | 00:11 |
| *** August1914 has joined #openstack-infra | 00:12 | |
| jeblair | cacti says the fs completely filled up very quickly yesterday... it's the root fs that's the problem, not the fs holding the metrics data | 00:12 |
| jeblair | http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=446&rra_id=1&view_type=&graph_start=1446855161&graph_end=1446941561 | 00:12 |
| *** salv-orlando has quit IRC | 00:13 | |
| *** cburgess_ has quit IRC | 00:14 | |
| *** lyz has quit IRC | 00:14 | |
| *** lyz has joined #openstack-infra | 00:15 | |
| *** grantbow has quit IRC | 00:15 | |
| jeblair | (so probably something filled up the fs and then the next deploy ran out of space explaining the missing files) | 00:15 |
| *** cburgess has joined #openstack-infra | 00:16 | |
| *** grantbow has joined #openstack-infra | 00:17 | |
| jeblair | i'm running du to try to find the culprit | 00:21 |
| *** fedexo has quit IRC | 00:22 | |
| *** zumbi has quit IRC | 00:23 | |
| jeblair | i think i'm about to get kicked out of the googleplex... so if any other infra-root can take over, that'd be great. | 00:25 |
| jeblair | my du has not found the cause yet -- it's taking a long time. | 00:26 |
| jeblair | i'm running it in screen as root, so feel free to rejoin that session and hopefully it will be done soon | 00:26 |
| *** otter768 has joined #openstack-infra | 00:31 | |
| *** otter768 has quit IRC | 00:32 | |
| *** otter768 has joined #openstack-infra | 00:32 | |
| *** wenlock has joined #openstack-infra | 00:34 | |
| *** EricGonczer_ has quit IRC | 00:35 | |
| *** wenlock has quit IRC | 00:41 | |
| *** shashank_hegde has quit IRC | 00:47 | |
| *** shashank_hegde has joined #openstack-infra | 00:56 | |
| *** _denisra has quit IRC | 01:06 | |
| *** EricGonczer_ has joined #openstack-infra | 01:08 | |
| *** sputnik13 has joined #openstack-infra | 01:21 | |
| *** jhesketh has quit IRC | 01:24 | |
| *** otter768 has quit IRC | 01:24 | |
| *** jhesketh has joined #openstack-infra | 01:28 | |
| *** sputnik13 has quit IRC | 01:48 | |
| *** shane-wang has joined #openstack-infra | 01:50 | |
| *** asselin_ has joined #openstack-infra | 01:54 | |
| *** fawadkhaliq has joined #openstack-infra | 02:00 | |
| *** nibalizer has quit IRC | 02:01 | |
| asselin_ | clarkb, I'm back and just tried the one on github and it fails as expected when applied. Will propose that update. | 02:05 |
| shane-wang | test | 02:06 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-jenkins: SECURITY-218: Update to latest cli-shutdown.groovy https://review.openstack.org/242787 | 02:14 |
| asselin_ | clarkb, jeblair, fungi ok this works for me ^^ http://15.125.105.66:8080/cli/ | 02:17 |
| *** EricGonczer_ has quit IRC | 02:17 | |
| *** ildikov has joined #openstack-infra | 02:20 | |
| *** salv-orlando has joined #openstack-infra | 02:23 | |
| *** shane-wang has quit IRC | 02:27 | |
| *** fawadkhaliq has quit IRC | 02:28 | |
| *** shashank_hegde has quit IRC | 02:29 | |
| *** wenlock has joined #openstack-infra | 02:35 | |
| *** salv-orlando has quit IRC | 02:36 | |
| *** tjones has joined #openstack-infra | 02:40 | |
| *** Ng has quit IRC | 02:41 | |
| *** j^2 has quit IRC | 02:41 | |
| *** zeljkof has quit IRC | 02:41 | |
| *** xgerman has quit IRC | 02:41 | |
| *** rackertom has quit IRC | 02:41 | |
| *** wenlock has quit IRC | 02:46 | |
| *** dims has quit IRC | 02:51 | |
| *** dims has joined #openstack-infra | 02:51 | |
| *** wenlock has joined #openstack-infra | 03:01 | |
| *** amitgandhinz has joined #openstack-infra | 03:07 | |
| *** rguillebert has quit IRC | 03:08 | |
| *** amitgandhinz has quit IRC | 03:11 | |
| *** baoli has quit IRC | 03:18 | |
| *** tjones has quit IRC | 03:18 | |
| *** sdake has joined #openstack-infra | 03:20 | |
| *** sdake_ has joined #openstack-infra | 03:24 | |
| *** sdake has quit IRC | 03:24 | |
| *** otter768 has joined #openstack-infra | 03:25 | |
| *** otter768 has quit IRC | 03:31 | |
| *** ayoung has quit IRC | 03:33 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/237894 | 03:35 |
| *** sdake_ is now known as sdake | 03:39 | |
| *** sdake_ has joined #openstack-infra | 03:49 | |
| *** sridhar_ram has joined #openstack-infra | 03:50 | |
| *** sdake has quit IRC | 03:52 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Deal with underscores in blacklist during generate-constraints https://review.openstack.org/242798 | 04:00 |
| *** baoli has joined #openstack-infra | 04:04 | |
| *** salv-orlando has joined #openstack-infra | 04:09 | |
| *** dims has quit IRC | 04:14 | |
| *** salv-orlando has quit IRC | 04:17 | |
| *** btully has joined #openstack-infra | 04:23 | |
| *** achanda has joined #openstack-infra | 04:26 | |
| *** btully has quit IRC | 04:28 | |
| *** weihan has joined #openstack-infra | 04:42 | |
| *** yamamoto has joined #openstack-infra | 04:43 | |
| *** weihan_ has joined #openstack-infra | 04:52 | |
| *** weihan_ has quit IRC | 04:55 | |
| *** weihan has quit IRC | 04:55 | |
| *** Liuqing has joined #openstack-infra | 04:57 | |
| *** weihan has joined #openstack-infra | 04:57 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-openstackci: Prepopulate log server for single-node-ci https://review.openstack.org/242800 | 05:05 |
| *** weihan has quit IRC | 05:07 | |
| *** weihan has joined #openstack-infra | 05:11 | |
| *** bpokorny has joined #openstack-infra | 05:12 | |
| *** yamamoto has quit IRC | 05:19 | |
| *** yamamoto has joined #openstack-infra | 05:19 | |
| *** weihan has quit IRC | 05:20 | |
| *** weihan has joined #openstack-infra | 05:22 | |
| *** weihan has quit IRC | 05:25 | |
| *** otter768 has joined #openstack-infra | 05:26 | |
| *** salv-orlando has joined #openstack-infra | 05:29 | |
| *** otter768 has quit IRC | 05:31 | |
| *** weihan has joined #openstack-infra | 05:31 | |
| *** weihan has quit IRC | 05:32 | |
| *** salv-orlando has quit IRC | 05:33 | |
| *** e0ne has joined #openstack-infra | 05:33 | |
| *** weihan has joined #openstack-infra | 05:33 | |
| *** devl has joined #openstack-infra | 05:33 | |
| *** sileht has quit IRC | 05:35 | |
| *** weihan has quit IRC | 05:36 | |
| *** weihan has joined #openstack-infra | 05:36 | |
| *** weihan has quit IRC | 05:37 | |
| *** otter768 has joined #openstack-infra | 05:38 | |
| *** weihan has joined #openstack-infra | 05:39 | |
| *** e0ne has quit IRC | 05:39 | |
| *** e0ne has joined #openstack-infra | 05:40 | |
| *** yamamoto has quit IRC | 05:40 | |
| *** yamamoto has joined #openstack-infra | 05:41 | |
| *** e0ne has quit IRC | 05:44 | |
| *** e0ne has joined #openstack-infra | 05:48 | |
| *** otter768 has quit IRC | 05:48 | |
| *** e0ne has quit IRC | 05:49 | |
| *** maishsk has quit IRC | 05:51 | |
| *** zz_dimtruck is now known as dimtruck | 05:51 | |
| *** dimtruck is now known as zz_dimtruck | 05:52 | |
| *** sridhar_ram has quit IRC | 05:53 | |
| *** krotscheck has left #openstack-infra | 05:58 | |
| *** abregman has joined #openstack-infra | 06:02 | |
| *** yfried has quit IRC | 06:08 | |
| *** weihan has quit IRC | 06:09 | |
| *** btully has joined #openstack-infra | 06:12 | |
| *** unicell has joined #openstack-infra | 06:15 | |
| *** unicell1 has quit IRC | 06:15 | |
| *** bpokorny has quit IRC | 06:17 | |
| *** nadya has joined #openstack-infra | 06:17 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/237894 | 06:29 |
| *** ihrachys has joined #openstack-infra | 06:38 | |
| *** garyk has joined #openstack-infra | 06:40 | |
| *** salv-orlando has joined #openstack-infra | 06:41 | |
| *** salv-orlando has quit IRC | 06:44 | |
| *** gampel has joined #openstack-infra | 06:46 | |
| *** gampel has left #openstack-infra | 06:47 | |
| *** fawadkhaliq has joined #openstack-infra | 07:03 | |
| *** sileht has joined #openstack-infra | 07:09 | |
| *** yfried has joined #openstack-infra | 07:09 | |
| *** sdake_ has quit IRC | 07:10 | |
| *** yfried_ has joined #openstack-infra | 07:13 | |
| *** links has joined #openstack-infra | 07:13 | |
| *** yfried has quit IRC | 07:14 | |
| *** yfried_ is now known as yfried|afk | 07:15 | |
| *** yfried|afk is now known as yfried_ | 07:16 | |
| *** baoli has quit IRC | 07:36 | |
| *** yolanda has quit IRC | 07:39 | |
| *** maishsk has joined #openstack-infra | 07:44 | |
| *** yfried_ is now known as yfried|afk | 07:47 | |
| *** yamamoto has quit IRC | 07:48 | |
| *** yamamoto has joined #openstack-infra | 07:48 | |
| *** gchamoul has quit IRC | 07:48 | |
| *** otter768 has joined #openstack-infra | 07:49 | |
| *** yfried|afk is now known as yfried_ | 07:50 | |
| *** salv-orl_ has joined #openstack-infra | 07:52 | |
| *** salv-orl_ has quit IRC | 07:53 | |
| *** otter768 has quit IRC | 07:54 | |
| *** ihrachys has quit IRC | 07:58 | |
| *** yamamoto has quit IRC | 08:01 | |
| *** yamamoto has joined #openstack-infra | 08:02 | |
| *** fawadkhaliq has quit IRC | 08:04 | |
| *** fawadkhaliq has joined #openstack-infra | 08:05 | |
| *** fawadkhaliq has quit IRC | 08:08 | |
| *** aopincar has joined #openstack-infra | 08:17 | |
| openstackgerrit | Merged openstack-infra/project-config: Add Evoque client project https://review.openstack.org/242309 | 08:18 |
| *** hdd has joined #openstack-infra | 08:18 | |
| *** yamamoto has quit IRC | 08:21 | |
| *** yamamoto has joined #openstack-infra | 08:23 | |
| *** yamamoto has quit IRC | 08:23 | |
| *** baoli has joined #openstack-infra | 08:23 | |
| *** baoli has quit IRC | 08:28 | |
| *** hdd has quit IRC | 08:29 | |
| *** kozhukalov_ has joined #openstack-infra | 08:31 | |
| *** openstackgerrit has quit IRC | 08:31 | |
| *** openstackgerrit has joined #openstack-infra | 08:31 | |
| *** yfried__ has joined #openstack-infra | 08:34 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Add an extension capability for adding targets https://review.openstack.org/242363 | 08:35 |
| *** yfried_ has quit IRC | 08:36 | |
| *** yfried has joined #openstack-infra | 08:36 | |
| *** ZZelle has quit IRC | 08:38 | |
| *** ZZelle has joined #openstack-infra | 08:39 | |
| *** yfried__ has quit IRC | 08:39 | |
| *** achanda has quit IRC | 08:40 | |
| *** yamamoto has joined #openstack-infra | 08:41 | |
| *** hdd has joined #openstack-infra | 08:45 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/subunit2sql: Add an extension capability for adding targets https://review.openstack.org/242363 | 08:46 |
| *** sshnaidm has joined #openstack-infra | 08:47 | |
| openstackgerrit | Alexey Weyl proposed openstack-infra/irc-meetings: Adding vitrage team meeting https://review.openstack.org/242816 | 08:47 |
| openstackgerrit | Maty Grosz proposed openstack-infra/project-config: Initialize vitrage-dashboard new project https://review.openstack.org/242817 | 08:50 |
| *** hdd has quit IRC | 08:57 | |
| *** mgrosz has joined #openstack-infra | 08:59 | |
| *** yamamoto has quit IRC | 08:59 | |
| *** btully has quit IRC | 08:59 | |
| mgrosz | Hello, can someone please review https://review.openstack.org/#/c/242817/? Added a new project vitrage-dashboard. Thanks | 09:00 |
| *** rossella_s has quit IRC | 09:07 | |
| *** rossella_s has joined #openstack-infra | 09:07 | |
| *** mgrosz has quit IRC | 09:10 | |
| *** yfried has quit IRC | 09:13 | |
| *** yfried has joined #openstack-infra | 09:13 | |
| *** Liuqing has quit IRC | 09:17 | |
| *** Liuqing has joined #openstack-infra | 09:18 | |
| *** ttx has quit IRC | 09:19 | |
| *** Liuqing has quit IRC | 09:27 | |
| *** Liuqing has joined #openstack-infra | 09:28 | |
| *** FallenPegasus has joined #openstack-infra | 09:30 | |
| *** btully has joined #openstack-infra | 09:37 | |
| *** btully has quit IRC | 09:41 | |
| *** otter768 has joined #openstack-infra | 09:50 | |
| *** kozhukalov_ has quit IRC | 09:51 | |
| *** sdake has joined #openstack-infra | 09:53 | |
| *** sdake has joined #openstack-infra | 09:54 | |
| *** otter768 has quit IRC | 09:55 | |
| *** kozhukalov_ has joined #openstack-infra | 09:58 | |
| *** nadya has quit IRC | 09:58 | |
| *** yamamoto has joined #openstack-infra | 10:00 | |
| *** yamamoto has quit IRC | 10:05 | |
| *** sdake has quit IRC | 10:07 | |
| *** sdake has joined #openstack-infra | 10:07 | |
| *** salv-orlando has joined #openstack-infra | 10:11 | |
| *** kaisers has quit IRC | 10:13 | |
| *** kaisers has joined #openstack-infra | 10:15 | |
| *** yamamoto has joined #openstack-infra | 10:18 | |
| *** salv-orlando has quit IRC | 10:19 | |
| *** yamamoto has quit IRC | 10:20 | |
| *** yamamoto has joined #openstack-infra | 10:20 | |
| *** yamamoto has quit IRC | 10:22 | |
| *** kozhukalov_ has quit IRC | 10:24 | |
| *** rguillebert has joined #openstack-infra | 10:24 | |
| *** mgrosz has joined #openstack-infra | 10:26 | |
| *** HeOS has joined #openstack-infra | 10:27 | |
| *** yamamoto has joined #openstack-infra | 10:28 | |
| *** asselin_ has quit IRC | 10:30 | |
| *** mgrosz has quit IRC | 10:33 | |
| *** mgrosz has joined #openstack-infra | 10:37 | |
| *** mgrosz has quit IRC | 10:38 | |
| *** yamamoto has quit IRC | 10:48 | |
| *** nadya has joined #openstack-infra | 10:50 | |
| *** chlong has joined #openstack-infra | 10:52 | |
| *** mgrosz has joined #openstack-infra | 10:59 | |
| *** mgrosz has quit IRC | 11:01 | |
| *** yfried is now known as yfried|afk | 11:01 | |
| *** sputnik13 has joined #openstack-infra | 11:02 | |
| *** leakypipes has quit IRC | 11:02 | |
| *** yamamoto has joined #openstack-infra | 11:12 | |
| *** FallenPegasus has quit IRC | 11:13 | |
| *** yfried|afk is now known as yfried | 11:13 | |
| *** wenlock has quit IRC | 11:13 | |
| *** August1914 has quit IRC | 11:14 | |
| *** yamamoto has quit IRC | 11:16 | |
| *** pbourke has quit IRC | 11:16 | |
| *** pbourke has joined #openstack-infra | 11:16 | |
| *** cody-somerville has quit IRC | 11:17 | |
| *** nadya has quit IRC | 11:21 | |
| *** zhiwei has joined #openstack-infra | 11:26 | |
| *** zhiwei has left #openstack-infra | 11:27 | |
| *** cody-somerville has joined #openstack-infra | 11:33 | |
| *** mgrosz has joined #openstack-infra | 11:37 | |
| *** Maky has joined #openstack-infra | 11:39 | |
| *** qba73 has joined #openstack-infra | 11:40 | |
| *** sdake has quit IRC | 11:41 | |
| openstackgerrit | Maty Grosz proposed openstack-infra/project-config: Initialize vitrage-dashboard new project https://review.openstack.org/242817 | 11:43 |
| *** doronshai has joined #openstack-infra | 11:46 | |
| doronshai | what is the way in openstack to push code to main trunk but in a very early stage so jenkins build should not be triggered? | 11:47 |
| mgrosz | General question... Does OpenStack CI process uses any monitor code cleanliness tool, code coverage tool or any monitor memory consumption/leaks tools? | 11:47 |
| *** maishsk has quit IRC | 11:48 | |
| *** qba73 has quit IRC | 11:49 | |
| *** maishsk has joined #openstack-infra | 11:50 | |
| *** maishsk has quit IRC | 11:51 | |
| *** maishsk has joined #openstack-infra | 11:51 | |
| *** maishsk has quit IRC | 11:51 | |
| *** otter768 has joined #openstack-infra | 11:51 | |
| *** otter768 has quit IRC | 11:56 | |
| *** yamamoto has joined #openstack-infra | 12:14 | |
| *** sputnik13 has quit IRC | 12:16 | |
| *** nadya has joined #openstack-infra | 12:21 | |
| *** nadya has quit IRC | 12:26 | |
| *** olaph has joined #openstack-infra | 12:39 | |
| *** dims has joined #openstack-infra | 12:41 | |
| *** j^2 has joined #openstack-infra | 12:41 | |
| *** devl_ has joined #openstack-infra | 12:44 | |
| *** maishsk has joined #openstack-infra | 12:44 | |
| *** Liuqing has quit IRC | 12:44 | |
| *** devl has quit IRC | 12:45 | |
| *** yolanda has joined #openstack-infra | 12:48 | |
| *** yolanda has quit IRC | 12:48 | |
| *** yolanda has joined #openstack-infra | 12:49 | |
| *** yolanda_ has joined #openstack-infra | 12:49 | |
| *** yolanda has quit IRC | 12:51 | |
| *** yolanda has joined #openstack-infra | 12:54 | |
| *** maishsk has quit IRC | 12:56 | |
| *** yolanda has quit IRC | 12:56 | |
| *** Ng has joined #openstack-infra | 12:58 | |
| *** maishsk has joined #openstack-infra | 12:58 | |
| *** yolanda has joined #openstack-infra | 13:00 | |
| *** yolanda has quit IRC | 13:01 | |
| *** yolanda has joined #openstack-infra | 13:06 | |
| *** yolanda has quit IRC | 13:07 | |
| *** xgerman has joined #openstack-infra | 13:08 | |
| *** mgrosz has quit IRC | 13:10 | |
| *** yolanda has joined #openstack-infra | 13:11 | |
| mordred | doronshai: the only time code can be pushed to master without ci running is in the initial code import | 13:12 |
| *** yolanda has quit IRC | 13:12 | |
| *** devl has joined #openstack-infra | 13:16 | |
| *** devl_ has quit IRC | 13:17 | |
| *** yolanda has joined #openstack-infra | 13:17 | |
| *** mgrosz has joined #openstack-infra | 13:17 | |
| *** yolanda has quit IRC | 13:17 | |
| *** rackertom has joined #openstack-infra | 13:18 | |
| *** yolanda has joined #openstack-infra | 13:22 | |
| *** nadya has joined #openstack-infra | 13:23 | |
| *** yolanda has quit IRC | 13:23 | |
| *** nadya has quit IRC | 13:27 | |
| *** yolanda has joined #openstack-infra | 13:28 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Let os-client-config handle session creation https://review.openstack.org/240734 | 13:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Remove designate support https://review.openstack.org/242095 | 13:28 |
| *** yolanda has quit IRC | 13:30 | |
| *** yolanda has joined #openstack-infra | 13:33 | |
| *** zeljkof has joined #openstack-infra | 13:34 | |
| *** yolanda has quit IRC | 13:34 | |
| mordred | infra-root: I'm looking at jeblair's screen session and the tmp dir problem | 13:37 |
| mordred | there is this: /tmp/pip-HDH182-uninstall in /tmp | 13:37 |
| fungi | jeblair: clarkb: mordred: looks like the graphite full filesystem is in /tmp/pip-HDH182-uninstall/var/lib/graphite/storage/whisper/ | 13:38 |
| mordred | before I remove it, I'm going to look at the ansible/puppet logs | 13:38 |
| fungi | hah, i was just looking into it too | 13:38 |
| *** yolanda has joined #openstack-infra | 13:39 | |
| mordred | fungi: :) | 13:39 |
| fungi | and yeah, let's not remove that | 13:39 |
| fungi | i think that's our data | 13:39 |
| mordred | yah | 13:39 |
| mordred | but the question is - why is pip trying to remove /var/lib | 13:39 |
| mordred | ? | 13:39 |
| mordred | fungi: is your connection to puppetmaster _very_ slow? | 13:39 |
| *** mgrosz has quit IRC | 13:40 | |
| *** yolanda has quit IRC | 13:40 | |
| fungi | i'm not connecting through puppetmaster | 13:40 |
| fungi | but i'll check | 13:40 |
| mordred | I was going there to look at puppet logs | 13:40 |
| *** yfried is now known as yfried|afk | 13:40 | |
| mordred | seems better now | 13:41 |
| fungi | seems fine for m | 13:41 |
| fungi | e | 13:41 |
| *** achanda has joined #openstack-infra | 13:41 | |
| *** zhurong has joined #openstack-infra | 13:43 | |
| *** achanda has quit IRC | 13:43 | |
| fungi | mordred: oh! | 13:43 |
| fungi | that's not our real data | 13:43 |
| mordred | neat | 13:43 |
| fungi | we've got a cinder volume for /var/lib/graphite/storage | 13:44 |
| fungi | i forgot we added that a while back | 13:44 |
| fungi | so that's pip insanely attempting to backup /var/lib/graphite/storage when updating i guess | 13:44 |
| *** EricGonczer_ has joined #openstack-infra | 13:44 | |
| mordred | oh wow | 13:44 |
| *** yolanda has joined #openstack-infra | 13:45 | |
| fungi | and there's simply not enough room in /tmp (e.g. our rootfs) to contain it, or even like half of it | 13:45 |
| mordred | yeah | 13:45 |
| mordred | also | 13:45 |
| mordred | maybe pip shouldn't do that | 13:45 |
| mordred | dstufft: ^^ | 13:45 |
| fungi | i would agree with you, except on the maybe part | 13:45 |
| mordred | :) | 13:45 |
| fungi | something tells me there's a configuration issue with the graphite package | 13:45 |
| mordred | yea - I just started looking at it | 13:46 |
| fungi | it probably does something to make sure the /var/lib/graphite directory tree is populated at installation | 13:46 |
| fungi | which causes pip to think it's responsible for everything under it | 13:46 |
| *** yfried|afk is now known as yfried | 13:47 | |
| mordred | I think we should take this: : --install-option="--install-data=/var/lib/graphite" | 13:48 |
| *** yolanda has quit IRC | 13:48 | |
| mordred | out of our puppet | 13:48 |
| mordred | and maybe set that path to another location so that can be where graphite installs default files or something | 13:48 |
| fungi | instead we should have puppet do something to populate that on initial installation, i guess? | 13:49 |
| mordred | yah | 13:49 |
| fungi | i wonder what triggered it to attempt to reinstall graphite though | 13:49 |
| fungi | oh, right, we install from source not pypi | 13:50 |
| fungi | i didn't think to check their vcs | 13:50 |
| * mordred has puppet patch coming | 13:50 | |
| fungi | anyway, short term, we should be able to wipe out /tmp/pip-*, reinstall the binaries for whatever failed to install properly, and reboot the server i guess | 13:51 |
| mordred | yes. but let's stop puppet on the server while we get that sorted out | 13:51 |
| fungi | doing that now | 13:52 |
| fungi | it's disabled now | 13:52 |
| *** yolanda has joined #openstack-infra | 13:52 | |
| *** otter768 has joined #openstack-infra | 13:52 | |
| *** salv-orlando has joined #openstack-infra | 13:52 | |
| fungi | any objection to me deleting that tempdir? it'll take a few minutes to complete | 13:52 |
| mordred | nope. go for it | 13:53 |
| mordred | btw - looks like we have a sha pin on grpahite that is for an upstream issue that is marked fixed upstream | 13:53 |
| fungi | that makes it doubly odd that we reinstalled then | 13:53 |
| fungi | unless that pin was added in the past few days | 13:53 |
| fungi | i need to step away for a bit, but once you no longer see /tmp/pip-HDH182-uninstall existing it should be safe to proceed even if i'm not back to the computer yet | 13:55 |
| *** yolanda has quit IRC | 13:56 | |
| *** otter768 has quit IRC | 13:57 | |
| *** aopincar has quit IRC | 13:57 | |
| *** wenlock has joined #openstack-infra | 13:58 | |
| doronshai | Whati want to achieve is that the jenkins build will start only when there are 2 votes of +2 | 13:59 |
| doronshai | what is the way to have it? is zull and gearman is a must? | 14:00 |
| *** EricGonczer_ has quit IRC | 14:00 | |
| *** yolanda has joined #openstack-infra | 14:01 | |
| *** yolanda has quit IRC | 14:01 | |
| *** wenlock has quit IRC | 14:03 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-graphite: Install pip data to a non-production location https://review.openstack.org/242842 | 14:03 |
| *** salv-orlando has quit IRC | 14:03 | |
| mordred | fungi: there you go | 14:03 |
| mordred | doronshai: are you talking about for a third-party CI system? | 14:04 |
| mordred | doronshai: like, for OpenStack, the thing you are describing is not a possibility | 14:05 |
| mordred | doronshai: here we run tests on patch upload, and then we run them again when the Approved vote happens before merging | 14:05 |
| mordred | doronshai: none of our tooling understands "take an action when a second +2 vote happens" - nor am I sure what good doing such a thing would do | 14:06 |
| *** yolanda has joined #openstack-infra | 14:06 | |
| mordred | since usually a second +2 is accompanied by a +A - so if you ran some tests for the first time in a 3rd party system then, nobody would ever see the results | 14:06 |
| mordred | in any case- if I understand what you are trying to accomplish better, I might be able to point you in the right direction | 14:06 |
| mordred | fungi: ^^ there is a patch that will, I think, help | 14:07 |
| *** yolanda has quit IRC | 14:07 | |
| doronshai | i mean for third party - in our lab | 14:09 |
| doronshai | what about the workflow custom label | 14:09 |
| doronshai | isn't there a condition on it to start the jenkins build? | 14:10 |
| mordred | sure. you can totally start a build when the workflow label is set to +1 | 14:12 |
| mordred | but if you run tests then, nobody will ever see the results | 14:12 |
| *** yolanda has joined #openstack-infra | 14:12 | |
| mordred | so it would be a waste of your resources | 14:12 |
| *** yfried has quit IRC | 14:12 | |
| *** baoli has joined #openstack-infra | 14:13 | |
| mordred | (also, yes, I _highly_ recommend using zuul to watch the gerrit events and trigger jenkins at times you configure it to) | 14:13 |
| mordred | fungi: /tmp/pip-HDH182-uninstall still exists | 14:14 |
| doronshai | mordred: why nobody will be able to see the results? | 14:15 |
| *** yolanda has quit IRC | 14:15 | |
| mordred | doronshai: because they will have already aproved the patch | 14:16 |
| mordred | so they are very unlikely to go back and look at additional tests that you ran after they did | 14:16 |
| mordred | where by very unlikely, I mean "nobody will ever go look at them" | 14:17 |
| mordred | the reason that we ask 3rd party ci to run tests on patchset-upload is so that the test results can advise the people doing the code review | 14:17 |
| doronshai | i see | 14:17 |
| doronshai | i want to have our local model a bit different than the original openstack - this is a side project which is not planned to become part of openstack | 14:18 |
| mordred | ah! gotcha | 14:18 |
| doronshai | since openstack implemented an amzing flow - i want to learn | 14:18 |
| mordred | so we basically have tests at those two times for different reasons | 14:19 |
| dims | doronshai if you dig into zuul, you will see that it uses gerrit stream-events to listen to events (https://github.com/openstack-infra/zuul/blob/master/zuul/trigger/gerrit.py#L80) | 14:19 |
| mordred | patchset-uploaded (check) tests are so that reviewers can not bother reviewing patches that don't pass tests | 14:19 |
| *** thorst has joined #openstack-infra | 14:20 | |
| *** yolanda has joined #openstack-infra | 14:20 | |
| *** thorst has quit IRC | 14:20 | |
| doronshai | so where can i find info about how to install and configure zull / gearman - i was loking for documentation and it was not quite clear to me what actually have to be sdone and what hardware is required.... | 14:20 |
| mordred | workflow-approved (gate) tests are to make sure that once it's reviewed, something didn't happen to cause it to start to fail due to new code that merged | 14:20 |
| mordred | asselin: ^^ you have a new customer | 14:21 |
| *** yolanda has quit IRC | 14:21 | |
| dims | doronshai http://docs.openstack.org/infra/system-config/running-your-own.html ? | 14:22 |
| *** baoli has quit IRC | 14:22 | |
| mordred | ya - that needs updated | 14:22 |
| doronshai | "Warning Parts of this file are out of date" | 14:22 |
| dims | mordred ah! | 14:22 |
| mordred | doronshai: https://github.com/openstack-infra/puppet-openstackci we have a puppet module which is for helping people do this - we need to update that doc to talk about using it | 14:22 |
| *** August1914 has joined #openstack-infra | 14:23 | |
| *** baoli has joined #openstack-infra | 14:23 | |
| doronshai | ok, and where can i find info about how to configure zull and such? | 14:24 |
| *** yolanda has joined #openstack-infra | 14:25 | |
| doronshai | any tip from where to start if i am not familiar with puppet? | 14:27 |
| *** yolanda has quit IRC | 14:27 | |
| *** thorst has joined #openstack-infra | 14:27 | |
| *** yolanda has joined #openstack-infra | 14:31 | |
| *** yolanda has quit IRC | 14:32 | |
| *** thorst has quit IRC | 14:32 | |
| mordred | dims: if you're around, would you mind https://review.openstack.org/#/c/242160/ ? | 14:34 |
| dims | mordred ack | 14:34 |
| mordred | dims: also - aren't we just supposed to approve the updates from the bot when they happen? https://review.openstack.org/#/c/232918 | 14:36 |
| dims | y, should have gotten in sooner (https://review.openstack.org/#/c/237894/19/upper-constraints.txt,cm) but the bot ran into trouble i have a review up (https://review.openstack.org/#/c/242798/) | 14:36 |
| *** yolanda has joined #openstack-infra | 14:36 | |
| mordred | ah! | 14:36 |
| dims | so the stable/liberty upper-constraints waiting for dhellmann to give the green signal | 14:36 |
| dims | i was talking about master | 14:37 |
| dims | during the summit lifeless was talking about having a release-constraints.txt in addition to a upper-constraints.txt so there's a bit more to do there i think | 14:37 |
| *** yolanda has quit IRC | 14:37 | |
| dims | mordred ^^ | 14:37 |
| mordred | oh - duh. that was a stable/liberty one. my bad | 14:38 |
| dims | :) | 14:38 |
| fungi | mordred: back, reviewing, looks like the delete finished | 14:41 |
| *** yolanda has joined #openstack-infra | 14:42 | |
| mordred | fungi: yes. | 14:43 |
| mordred | fungi: and I have run the changed commands in that puppet patch by hand and they worked | 14:43 |
| mordred | fungi: https://review.openstack.org/242842 | 14:43 |
| mordred | (and did not attempt to save the data dir) | 14:43 |
| *** yolanda has quit IRC | 14:44 | |
| *** yolanda has joined #openstack-infra | 14:49 | |
| *** HeOS has quit IRC | 14:49 | |
| *** yolanda has quit IRC | 14:49 | |
| *** dims has quit IRC | 14:50 | |
| *** dims has joined #openstack-infra | 14:52 | |
| *** yolanda has joined #openstack-infra | 14:54 | |
| *** yolanda has quit IRC | 14:56 | |
| *** dims has quit IRC | 14:56 | |
| *** doronshai has quit IRC | 14:58 | |
| mordred | fungi: I'm thinking we shoudl undisable puppet and run puppet agent on the host just to make sure that it's all good- then restart things - unless you think rebooting is necessary? | 15:00 |
| fungi | excellent. my only concern (noted in review) is that this may be insufficient to bootstrap a new server. we might need to add some additional file resources | 15:00 |
| *** yolanda has joined #openstack-infra | 15:00 | |
| mordred | (you mentioned a reboot above) | 15:00 |
| fungi | yep | 15:00 |
| mordred | fungi: those additional file resources are already in the puppet it turns out | 15:00 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-stackalytics: Create flock for crontab to avoid mulitple runs https://review.openstack.org/242848 | 15:00 |
| mordred | (sorry, I should have mentioned that - I tested what the install puts in that location, and it's just empty dirs, which we already create) | 15:01 |
| fungi | reboot is necessary any time you fill up the root filesystem | 15:01 |
| openstackgerrit | Merged openstack-infra/shade: Remove designate support https://review.openstack.org/242095 | 15:01 |
| mordred | kk | 15:01 |
| fungi | there's no telling what else it may have crashed, and also some inodes may no longer successfully unlink until that filesystem is unmounted | 15:02 |
| *** yolanda has quit IRC | 15:02 | |
| pabelanger | morning! | 15:02 |
| mordred | fungi: ++ | 15:02 |
| fungi | also a good way to catch filesystem corruption, since fsck will run if the kernel flagged it | 15:02 |
| mordred | fungi: I have noted the above about having tested what the installs do in the review | 15:02 |
| mordred | morning pabelanger | 15:02 |
| fungi | mordred: thanks. +3 | 15:03 |
| pabelanger | mordred: fungi: have some spare cycles to checkout the puppet-stackalytics patch I just posted? Add some locking support into crontab, and fixes multiple processors from spawning | 15:03 |
| openstackgerrit | Merged openstack-infra/nodepool: Fix provider manager reloading without jenkins https://review.openstack.org/233238 | 15:03 |
| fungi | let's let it merge, then reenable puppet, then reboot | 15:03 |
| fungi | pabelanger: perhaps in a bit | 15:04 |
| mordred | fungi: when that lands, I'll pull it onto thepuppetmaster, run a puppet agent --test by hand, then enable puppet | 15:04 |
| fungi | wfm | 15:04 |
| *** zhurong has quit IRC | 15:04 | |
| pabelanger | fungi: works for me | 15:04 |
| openstackgerrit | Merged openstack-infra/nodepool: Update tox.ini to current standards https://review.openstack.org/235403 | 15:04 |
| fungi | doronshai seems to have quit before anyone could point him to our zuul documentation | 15:05 |
| openstackgerrit | Merged openstack-infra/nodepool: Consume clouds.yaml from /etc/openstack https://review.openstack.org/240386 | 15:05 |
| fungi | or her | 15:05 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-stackalytics: Drop uwsgi process to 2 https://review.openstack.org/242850 | 15:06 |
| *** EricGonczer_ has joined #openstack-infra | 15:07 | |
| *** yolanda has joined #openstack-infra | 15:07 | |
| *** yolanda has quit IRC | 15:09 | |
| openstackgerrit | Merged openstack-infra/puppet-graphite: Install pip data to a non-production location https://review.openstack.org/242842 | 15:10 |
| *** EricGonczer_ has quit IRC | 15:11 | |
| *** yolanda has joined #openstack-infra | 15:12 | |
| *** yolanda has quit IRC | 15:13 | |
| fungi | mordred: ^ it merged | 15:14 |
| mordred | woot! | 15:14 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-stackalytics: Drop uwsgi process to 2 https://review.openstack.org/242850 | 15:15 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-stackalytics: Create flock for crontab to avoid mulitple runs https://review.openstack.org/242848 | 15:15 |
| pabelanger | fungi: mordred: sorry, I had a typo in first patch, do you mind reviewing again :( | 15:15 |
| *** sdake has joined #openstack-infra | 15:15 | |
| mordred | done | 15:16 |
| fungi | pabelanger: yep, saw the job failure and was just digging into the cause when you fixed it | 15:17 |
| *** yolanda has joined #openstack-infra | 15:18 | |
| pabelanger | fungi: mordred: Thanks! So, we're looking pretty good with stackalytics.o.o. So far everything is working as expected. Only concern for the moment is load issues, might need to extra tuning once more people start hitting the site | 15:19 |
| *** yolanda has quit IRC | 15:19 | |
| mordred | fungi: ok. I ran puppet on graphite and it succeeeded (and was a no-op as expected) | 15:20 |
| mordred | fungi: reboot? | 15:20 |
| fungi | pabelanger: or modification to use a real database | 15:20 |
| fungi | mordred: yep, go for it. if it doesn't come back up i can jump on the web console and troubleshoot | 15:20 |
| fungi | worst case we may need some manual fsck intervention | 15:20 |
| pabelanger | fungi: right. I suspect we'll have to make some adjustments to the code base. | 15:21 |
| mordred | kk | 15:21 |
| mordred | fungi: rebooting | 15:21 |
| *** Liuqing has joined #openstack-infra | 15:21 | |
| *** yolanda has joined #openstack-infra | 15:23 | |
| *** yolanda has quit IRC | 15:25 | |
| pabelanger | fungi: looks like graphite is down again too. OOM again? | 15:27 |
| fungi | pabelanger: we've been working on it all morning. the pip filled up the root filesystem | 15:28 |
| pabelanger | fungi: Ahh, well that explains it | 15:28 |
| fungi | mordred: looks like it came up and services are started | 15:29 |
| *** yolanda has joined #openstack-infra | 15:29 | |
| mordred | woot | 15:29 |
| *** yamamoto has quit IRC | 15:30 | |
| *** yolanda has quit IRC | 15:30 | |
| fungi | and now i need to stop computering for a bit and switch to weekend chores | 15:31 |
| fungi | bbiaw | 15:31 |
| mordred | ianw: https://review.openstack.org/#/c/237149/ is ready for review | 15:33 |
| *** dims has joined #openstack-infra | 15:33 | |
| openstackgerrit | Merged openstack-infra/puppet-stackalytics: Create flock for crontab to avoid mulitple runs https://review.openstack.org/242848 | 15:34 |
| *** yolanda has joined #openstack-infra | 15:35 | |
| *** bapalm has quit IRC | 15:35 | |
| *** bapalm has joined #openstack-infra | 15:36 | |
| *** yolanda has quit IRC | 15:36 | |
| openstackgerrit | Merged openstack-infra/puppet-stackalytics: Drop uwsgi process to 2 https://review.openstack.org/242850 | 15:38 |
| *** nadya has joined #openstack-infra | 15:39 | |
| *** yolanda has joined #openstack-infra | 15:40 | |
| *** yolanda has quit IRC | 15:43 | |
| *** nadya has quit IRC | 15:43 | |
| *** yolanda has joined #openstack-infra | 15:46 | |
| *** yolanda has quit IRC | 15:47 | |
| *** baoli has quit IRC | 15:50 | |
| *** yolanda has joined #openstack-infra | 15:51 | |
| *** otter768 has joined #openstack-infra | 15:53 | |
| *** yolanda has quit IRC | 15:53 | |
| *** baoli has joined #openstack-infra | 15:54 | |
| *** ayoung has joined #openstack-infra | 15:54 | |
| *** yolanda has joined #openstack-infra | 15:57 | |
| *** garyk has quit IRC | 15:57 | |
| *** otter768 has quit IRC | 15:58 | |
| *** yolanda has quit IRC | 15:58 | |
| *** maishsk has quit IRC | 15:58 | |
| *** wenlock has joined #openstack-infra | 16:00 | |
| *** yolanda has joined #openstack-infra | 16:02 | |
| *** wenlock has quit IRC | 16:04 | |
| *** yolanda has quit IRC | 16:04 | |
| *** kozhukalov_ has joined #openstack-infra | 16:04 | |
| *** maishsk has joined #openstack-infra | 16:05 | |
| *** yolanda has joined #openstack-infra | 16:08 | |
| *** maishsk has quit IRC | 16:09 | |
| *** yolanda has quit IRC | 16:09 | |
| *** yolanda has joined #openstack-infra | 16:14 | |
| *** e0ne has joined #openstack-infra | 16:16 | |
| *** yolanda has quit IRC | 16:16 | |
| *** thorst has joined #openstack-infra | 16:16 | |
| *** e0ne has quit IRC | 16:18 | |
| *** nadya has joined #openstack-infra | 16:19 | |
| *** yolanda has joined #openstack-infra | 16:19 | |
| *** yolanda has quit IRC | 16:20 | |
| *** thorst has quit IRC | 16:20 | |
| *** e0ne has joined #openstack-infra | 16:23 | |
| *** barnaby has joined #openstack-infra | 16:24 | |
| *** sdake has quit IRC | 16:24 | |
| *** yolanda has joined #openstack-infra | 16:25 | |
| *** yolanda has quit IRC | 16:26 | |
| *** asselin_ has joined #openstack-infra | 16:30 | |
| *** yolanda has joined #openstack-infra | 16:30 | |
| *** yamamoto has joined #openstack-infra | 16:31 | |
| *** maishsk has joined #openstack-infra | 16:31 | |
| *** yolanda has quit IRC | 16:32 | |
| openstackgerrit | Hongbin Lu proposed openstack/requirements: Add certifi to global requirements https://review.openstack.org/221107 | 16:33 |
| *** ayoung has quit IRC | 16:33 | |
| *** asselin_ has quit IRC | 16:35 | |
| *** yamamoto has quit IRC | 16:36 | |
| *** yolanda has joined #openstack-infra | 16:36 | |
| *** yolanda has quit IRC | 16:38 | |
| *** barnaby has quit IRC | 16:38 | |
| *** sshnaidm has quit IRC | 16:39 | |
| *** maishsk has quit IRC | 16:40 | |
| *** chlong has quit IRC | 16:41 | |
| *** yolanda has joined #openstack-infra | 16:42 | |
| *** yolanda has quit IRC | 16:43 | |
| *** e0ne has quit IRC | 16:45 | |
| *** yolanda has joined #openstack-infra | 16:47 | |
| *** yolanda has quit IRC | 16:48 | |
| *** e0ne has joined #openstack-infra | 16:49 | |
| *** yolanda has joined #openstack-infra | 16:53 | |
| *** e0ne has quit IRC | 16:54 | |
| *** yolanda has quit IRC | 16:54 | |
| *** yolanda has joined #openstack-infra | 16:58 | |
| *** e0ne has joined #openstack-infra | 16:58 | |
| *** yolanda has quit IRC | 16:59 | |
| *** kozhukalov_ has quit IRC | 17:01 | |
| *** yolanda has joined #openstack-infra | 17:04 | |
| *** yolanda has quit IRC | 17:05 | |
| *** abregman has quit IRC | 17:07 | |
| *** yolanda has joined #openstack-infra | 17:09 | |
| *** e0ne has quit IRC | 17:10 | |
| *** yolanda has quit IRC | 17:11 | |
| *** fedexo has joined #openstack-infra | 17:14 | |
| *** yolanda has joined #openstack-infra | 17:15 | |
| *** e0ne has joined #openstack-infra | 17:15 | |
| *** yolanda has quit IRC | 17:17 | |
| *** fedexo has quit IRC | 17:19 | |
| *** e0ne has quit IRC | 17:19 | |
| *** fedexo has joined #openstack-infra | 17:20 | |
| *** yolanda has joined #openstack-infra | 17:20 | |
| *** shashank_hegde has joined #openstack-infra | 17:21 | |
| *** yolanda has quit IRC | 17:23 | |
| *** e0ne has joined #openstack-infra | 17:23 | |
| *** yolanda has joined #openstack-infra | 17:26 | |
| *** e0ne has quit IRC | 17:27 | |
| *** yolanda has quit IRC | 17:27 | |
| *** Hal has joined #openstack-infra | 17:30 | |
| *** wenlock has joined #openstack-infra | 17:30 | |
| *** Hal is now known as Guest89252 | 17:31 | |
| *** yolanda has joined #openstack-infra | 17:31 | |
| *** e0ne has joined #openstack-infra | 17:32 | |
| *** yolanda has quit IRC | 17:34 | |
| *** _denisra has joined #openstack-infra | 17:34 | |
| *** Guest89252 has quit IRC | 17:36 | |
| *** yolanda has joined #openstack-infra | 17:37 | |
| *** yolanda has quit IRC | 17:39 | |
| *** tnarg has joined #openstack-infra | 17:40 | |
| *** wenlock has quit IRC | 17:42 | |
| *** yolanda has joined #openstack-infra | 17:42 | |
| *** nadya has quit IRC | 17:43 | |
| *** Liuqing has quit IRC | 17:43 | |
| *** yolanda has quit IRC | 17:43 | |
| *** e0ne has quit IRC | 17:46 | |
| *** yolanda has joined #openstack-infra | 17:48 | |
| *** [HeOS] has joined #openstack-infra | 17:48 | |
| *** yolanda has quit IRC | 17:49 | |
| *** e0ne has joined #openstack-infra | 17:50 | |
| *** ayoung has joined #openstack-infra | 17:51 | |
| *** dims has quit IRC | 17:51 | |
| *** yolanda has joined #openstack-infra | 17:53 | |
| *** otter768 has joined #openstack-infra | 17:54 | |
| *** yolanda has quit IRC | 17:54 | |
| *** e0ne has quit IRC | 17:57 | |
| *** otter768 has quit IRC | 17:58 | |
| *** yolanda has joined #openstack-infra | 17:59 | |
| *** yolanda has quit IRC | 18:00 | |
| *** e0ne has joined #openstack-infra | 18:01 | |
| *** fedexo has quit IRC | 18:01 | |
| *** yolanda has joined #openstack-infra | 18:04 | |
| *** yolanda has quit IRC | 18:07 | |
| *** e0ne has quit IRC | 18:08 | |
| *** kozhukalov_ has joined #openstack-infra | 18:09 | |
| *** yolanda has joined #openstack-infra | 18:10 | |
| *** [HeOS] has quit IRC | 18:11 | |
| *** yolanda has quit IRC | 18:11 | |
| *** [HeOS] has joined #openstack-infra | 18:12 | |
| *** e0ne has joined #openstack-infra | 18:12 | |
| *** yolanda has joined #openstack-infra | 18:16 | |
| *** e0ne has quit IRC | 18:16 | |
| *** tjones has joined #openstack-infra | 18:16 | |
| *** yolanda has quit IRC | 18:18 | |
| *** nadya has joined #openstack-infra | 18:20 | |
| *** yolanda has joined #openstack-infra | 18:21 | |
| *** yolanda has quit IRC | 18:24 | |
| *** Maky has left #openstack-infra | 18:26 | |
| *** dims has joined #openstack-infra | 18:27 | |
| *** yolanda has joined #openstack-infra | 18:27 | |
| *** nadya has quit IRC | 18:27 | |
| *** sergmelikyan has joined #openstack-infra | 18:29 | |
| *** yolanda has quit IRC | 18:29 | |
| *** yolanda has joined #openstack-infra | 18:33 | |
| *** [HeOS] has quit IRC | 18:33 | |
| *** yolanda has quit IRC | 18:35 | |
| *** btully has joined #openstack-infra | 18:37 | |
| *** yolanda has joined #openstack-infra | 18:38 | |
| *** btully has quit IRC | 18:42 | |
| *** yolanda has joined #openstack-infra | 18:44 | |
| *** yolanda has quit IRC | 18:45 | |
| *** yolanda has joined #openstack-infra | 18:49 | |
| *** yolanda has quit IRC | 18:52 | |
| *** yolanda has joined #openstack-infra | 18:55 | |
| *** yolanda has quit IRC | 18:57 | |
| *** August1914_ has joined #openstack-infra | 18:58 | |
| *** links has quit IRC | 18:59 | |
| *** August1914 has quit IRC | 18:59 | |
| *** August1914_ is now known as August1914 | 18:59 | |
| *** yolanda has joined #openstack-infra | 19:00 | |
| *** kevinbenton has quit IRC | 19:01 | |
| *** devl has quit IRC | 19:01 | |
| *** yolanda has quit IRC | 19:02 | |
| *** kevinbenton has joined #openstack-infra | 19:03 | |
| *** sergmelikyan has quit IRC | 19:04 | |
| *** dims has quit IRC | 19:05 | |
| *** yolanda has joined #openstack-infra | 19:06 | |
| *** yolanda has quit IRC | 19:09 | |
| *** sergmelikyan has joined #openstack-infra | 19:10 | |
| *** yolanda has joined #openstack-infra | 19:12 | |
| *** yolanda has quit IRC | 19:14 | |
| *** otter768 has joined #openstack-infra | 19:16 | |
| *** yolanda has joined #openstack-infra | 19:17 | |
| *** yfried has joined #openstack-infra | 19:19 | |
| *** yolanda has quit IRC | 19:19 | |
| *** shashank_hegde has quit IRC | 19:22 | |
| *** yolanda has joined #openstack-infra | 19:23 | |
| *** yolanda has quit IRC | 19:25 | |
| *** yolanda has joined #openstack-infra | 19:28 | |
| *** achanda has joined #openstack-infra | 19:29 | |
| *** sergmelikyan has quit IRC | 19:29 | |
| *** yolanda has quit IRC | 19:31 | |
| *** kozhukalov_ has quit IRC | 19:31 | |
| *** yolanda has joined #openstack-infra | 19:34 | |
| *** sergmelikyan has joined #openstack-infra | 19:35 | |
| *** yolanda has quit IRC | 19:35 | |
| *** kevinbenton_ has joined #openstack-infra | 19:36 | |
| *** kevinbenton has quit IRC | 19:39 | |
| *** yolanda has joined #openstack-infra | 19:40 | |
| *** slagle has joined #openstack-infra | 19:42 | |
| *** yolanda has quit IRC | 19:42 | |
| *** nadya has joined #openstack-infra | 19:43 | |
| *** FallenPegasus has joined #openstack-infra | 19:45 | |
| *** yolanda has joined #openstack-infra | 19:45 | |
| *** slagle has quit IRC | 19:47 | |
| *** yolanda has quit IRC | 19:47 | |
| openstackgerrit | Monty Taylor proposed openstack/requirements: Add requestsexceptions to global requirements https://review.openstack.org/241324 | 19:47 |
| *** slagle has joined #openstack-infra | 19:48 | |
| *** achanda has quit IRC | 19:50 | |
| *** yolanda has joined #openstack-infra | 19:51 | |
| *** sergmelikyan has quit IRC | 19:51 | |
| *** sdake has joined #openstack-infra | 19:51 | |
| *** yolanda has quit IRC | 19:53 | |
| openstackgerrit | Merged openstack-infra/project-config: Opening a repository for a ceilometer plugin for collectd. https://review.openstack.org/222763 | 19:56 |
| *** yolanda has joined #openstack-infra | 19:57 | |
| *** nadya has quit IRC | 19:57 | |
| *** sdake has quit IRC | 19:58 | |
| *** yolanda has quit IRC | 19:58 | |
| openstackgerrit | Merged openstack-infra/project-config: zuul/layout: fix puppet-integration jobs https://review.openstack.org/242040 | 19:59 |
| *** fawadkhaliq has joined #openstack-infra | 20:02 | |
| *** yolanda has joined #openstack-infra | 20:03 | |
| *** yolanda has quit IRC | 20:05 | |
| *** yolanda has joined #openstack-infra | 20:08 | |
| *** yolanda has quit IRC | 20:09 | |
| *** yolanda has joined #openstack-infra | 20:14 | |
| *** yolanda has quit IRC | 20:16 | |
| *** achanda has joined #openstack-infra | 20:17 | |
| *** otter768 has quit IRC | 20:18 | |
| *** jamielennox|away is now known as jamielennox | 20:19 | |
| *** yolanda has joined #openstack-infra | 20:19 | |
| *** wenlock has joined #openstack-infra | 20:19 | |
| *** hdd has joined #openstack-infra | 20:20 | |
| *** otter768 has joined #openstack-infra | 20:20 | |
| *** yolanda has quit IRC | 20:21 | |
| *** wenlock has quit IRC | 20:23 | |
| *** yolanda has joined #openstack-infra | 20:26 | |
| *** achanda has quit IRC | 20:26 | |
| *** yolanda has quit IRC | 20:26 | |
| *** otter768 has quit IRC | 20:30 | |
| *** yolanda has joined #openstack-infra | 20:31 | |
| *** yolanda has quit IRC | 20:32 | |
| *** hdd has quit IRC | 20:36 | |
| *** yolanda has joined #openstack-infra | 20:37 | |
| *** asselin_ has joined #openstack-infra | 20:38 | |
| *** yolanda has quit IRC | 20:38 | |
| *** asselin_ has quit IRC | 20:41 | |
| *** yolanda has joined #openstack-infra | 20:42 | |
| *** asselin_ has joined #openstack-infra | 20:44 | |
| *** yolanda has quit IRC | 20:44 | |
| *** tjones has quit IRC | 20:47 | |
| *** fawadkhaliq has quit IRC | 20:47 | |
| *** yolanda has joined #openstack-infra | 20:48 | |
| *** yolanda has quit IRC | 20:50 | |
| *** yolanda has joined #openstack-infra | 20:54 | |
| *** yolanda has quit IRC | 20:55 | |
| *** kozhukalov_ has joined #openstack-infra | 20:57 | |
| *** yolanda has joined #openstack-infra | 20:59 | |
| *** yolanda has quit IRC | 21:01 | |
| *** tlian has joined #openstack-infra | 21:02 | |
| *** kozhukalov_ has quit IRC | 21:03 | |
| *** tlian has quit IRC | 21:03 | |
| *** ddieterl_ has quit IRC | 21:04 | |
| *** yolanda has joined #openstack-infra | 21:05 | |
| *** yolanda has quit IRC | 21:06 | |
| *** rossella_s has quit IRC | 21:07 | |
| *** rossella_s has joined #openstack-infra | 21:07 | |
| *** yolanda has joined #openstack-infra | 21:10 | |
| *** yolanda has quit IRC | 21:12 | |
| *** sdake has joined #openstack-infra | 21:12 | |
| *** sdake has joined #openstack-infra | 21:13 | |
| *** asselin_ has quit IRC | 21:14 | |
| *** yolanda has joined #openstack-infra | 21:16 | |
| *** yolanda has quit IRC | 21:18 | |
| *** tlian has joined #openstack-infra | 21:18 | |
| *** yolanda has joined #openstack-infra | 21:21 | |
| *** yolanda has quit IRC | 21:23 | |
| *** tiswanso has joined #openstack-infra | 21:25 | |
| yolanda_ | jhesketh, around? i just did a resync of zuul downwstream, and i find some jobs failing with: | 21:26 |
| yolanda_ | File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py&q | 21:26 |
| yolanda_ | uot;, line 1528, in onMergeCompleted | 21:26 |
| yolanda_ | build_set.commit = item.change.newrev | 21:26 |
| yolanda_ | AttributeError: 'NullChange' object has no attribute 'newrev' | 21:26 |
| *** yolanda has joined #openstack-infra | 21:27 | |
| *** yolanda has quit IRC | 21:29 | |
| ruagair | o/ | 21:29 |
| *** yfried is now known as yfried|afk | 21:30 | |
| ruagair | Thanks mordred, tristanC, I've also been chatting to mhu as well. | 21:31 |
| *** yolanda has joined #openstack-infra | 21:32 | |
| *** yolanda has quit IRC | 21:34 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config: Add job for the ovs mechanism driver with native mode https://review.openstack.org/221143 | 21:36 |
| *** yolanda has joined #openstack-infra | 21:38 | |
| *** yolanda has quit IRC | 21:39 | |
| *** ddieterly has joined #openstack-infra | 21:41 | |
| *** yolanda_ has quit IRC | 21:42 | |
| *** yolanda has joined #openstack-infra | 21:42 | |
| *** yolanda_ has joined #openstack-infra | 21:43 | |
| *** yfried|afk is now known as yfried | 21:49 | |
| greghaynes | clarkb: I'm reviewing https://review.openstack.org/#/c/230023 - We dont actually run a fake zmq publisher in our tests AFAICT, so does the NodeUpdateListener just silently succeed on connect failure? | 21:50 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: Skip unnecessary tests for horizon docs/unittests changes https://review.openstack.org/242879 | 21:53 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: Skip unnecessary tests for horizon docs/unittests changes https://review.openstack.org/242879 | 21:55 |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: Skip unnecessary tests for horizon docs/unittests changes https://review.openstack.org/242879 | 21:56 |
| *** sdake_ has joined #openstack-infra | 22:00 | |
| *** sdake has quit IRC | 22:02 | |
| jhesketh | Morning | 22:02 |
| yolanda | hi jhesketh... so i have 2 problems with my downstream zuul. That is a problem with nullchange, but i think it's for an specific use case | 22:03 |
| yolanda | the other, is that i constantly have problems on zuul restarts. We have a set of canary jobs, experimentals, that are triggered periodically. These changes can have like 100,200 comments with recheck. I am thinking they are causing zuul starvation, have you ever found that? | 22:04 |
| yolanda | looking at zuul debug.log it's totally full of these changes | 22:04 |
| *** dims has joined #openstack-infra | 22:06 | |
| jhesketh | yolanda: I haven't seen anything like that sorry. It's quite possibly though | 22:07 |
| jhesketh | yolanda: are you able to clear out some of the comments to see if that helps? | 22:08 |
| jhesketh | I wonder if zuul pulls them all in every time there is a new one | 22:08 |
| yolanda | jhesketh, looks like, i can send you an extract of the logs | 22:09 |
| *** asselin_ has joined #openstack-infra | 22:11 | |
| *** tiswanso has quit IRC | 22:11 | |
| *** asselin__ has joined #openstack-infra | 22:11 | |
| jhesketh | Sure, but I don't know if I'll be able to do anything | 22:11 |
| *** dims has quit IRC | 22:11 | |
| yolanda | jhesketh, basically experimental jobs are queued, an zuul log full of messages with NOT_REGISTERED, that seem to be iterating each one of the comments on the experimental | 22:12 |
| jhesketh | Oh so you don't have workers for those jobs? | 22:13 |
| yolanda | yes, i should have, but looks as gearman not recognizing themn | 22:13 |
| yolanda | them | 22:13 |
| yolanda | it works fine in all other jobs, but these experimentals with lots of comments | 22:14 |
| jhesketh | Hmm okay | 22:14 |
| jhesketh | Have you connected to gearman and inspected the jobs+workers look as you expect? | 22:15 |
| yolanda | going to try, good idea | 22:15 |
| *** wenlock has joined #openstack-infra | 22:16 | |
| *** asselin_ has quit IRC | 22:20 | |
| *** asselin__ has quit IRC | 22:20 | |
| *** wenlock has quit IRC | 22:20 | |
| *** asselin_ has joined #openstack-infra | 22:21 | |
| *** asselin__ has joined #openstack-infra | 22:21 | |
| *** asselin_ has quit IRC | 22:26 | |
| *** nibalizer has joined #openstack-infra | 22:26 | |
| *** asselin_ has joined #openstack-infra | 22:27 | |
| *** asselin__ has quit IRC | 22:28 | |
| *** asselin__ has joined #openstack-infra | 22:29 | |
| *** asselin__ has quit IRC | 22:29 | |
| *** asselin_ has quit IRC | 22:29 | |
| *** asettle has joined #openstack-infra | 22:30 | |
| *** otter768 has joined #openstack-infra | 22:30 | |
| *** asselin__ has joined #openstack-infra | 22:32 | |
| *** sridhar_ram has joined #openstack-infra | 22:33 | |
| *** asselin__ has quit IRC | 22:33 | |
| *** asselin__ has joined #openstack-infra | 22:34 | |
| *** asselin_ has joined #openstack-infra | 22:34 | |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Builds and publishes Compute API Guide https://review.openstack.org/231000 | 22:34 |
| *** yfried is now known as yfried|afk | 22:35 | |
| *** asselin_ has quit IRC | 22:35 | |
| *** otter768 has quit IRC | 22:35 | |
| *** asselin__ has quit IRC | 22:35 | |
| yolanda | jhesketh, now hitting File "/usr/local/lib/python2.7/dist-packages/voluptuous.py", line 467, in validate_mapping | 22:36 |
| yolanda | raise MultipleInvalid(errors) | 22:36 |
| yolanda | voluptuous.MultipleInvalid: extra keys not allowed @ data['pipelines'][26]['ignore-dependencies'] | 22:36 |
| yolanda | i see that on merge-check, and i have the same on my zuul layout.yaml | 22:36 |
| yolanda | it should be supported properly right? | 22:36 |
| *** sdake_ has quit IRC | 22:40 | |
| *** tjones has joined #openstack-infra | 22:40 | |
| jhesketh | yolanda: check your zuul version? | 22:40 |
| yolanda | jhesketh, yep, bad version for that case | 22:42 |
| *** yfried|afk is now known as yfried | 22:48 | |
| clarkb | greghaynes yes thats a "feature" of zmq | 22:52 |
| clarkb | greghaynes the tests work by calling handlers directly and checking the side effects | 22:53 |
| greghaynes | Fun | 22:53 |
| greghaynes | I was wondering why we don't run a fake zmq publisher and push in to that | 22:53 |
| clarkb | I started with that but its far to async | 22:54 |
| clarkb | check older patchsets | 22:54 |
| lifeless | clarkb: there is an async facility built into testtools | 22:54 |
| lifeless | clarkb: would need just a little adaption for zmq's thing I suspect | 22:54 |
| lifeless | clarkb: (FWIW) | 22:54 |
| greghaynes | clarkb: yea, we would have to poll for the image to change or timeout. That is how a lot of the other tests are moving though | 22:56 |
| greghaynes | Or maybe that magic lifeless is talking about | 22:56 |
| *** tiswanso has joined #openstack-infra | 22:57 | |
| *** tiswanso has quit IRC | 23:03 | |
| *** yfried is now known as yfried|afk | 23:03 | |
| *** otter768 has joined #openstack-infra | 23:04 | |
| *** tjones has quit IRC | 23:06 | |
| *** sridhar_ram has quit IRC | 23:16 | |
| *** dims has joined #openstack-infra | 23:17 | |
| cody-somerville | yolanda: I've seen "AttributeError: 'NullChange' object has no attribute 'newrev'" for ages. It appears benign. | 23:20 |
| yolanda | cody-somerville, yep, that is triggered by our gozer-canary job | 23:20 |
| yolanda | and is causing that to stay there forever | 23:20 |
| yolanda | queued for 20 min | 23:20 |
| cody-somerville | No. I don't think that is causing that. It is benign. | 23:20 |
| cody-somerville | Happens for all periodic jobs. | 23:21 |
| cody-somerville | Unless something has since changed and now periodic jobs are broken in zuul all together. | 23:21 |
| yolanda | we'll need to trigger some other periodic to check | 23:22 |
| *** HeOS has joined #openstack-infra | 23:22 | |
| *** sc` has quit IRC | 23:24 | |
| yolanda | cody-somerville, for our gozer-canary i see | 23:24 |
| yolanda | 2015-11-08 23:24:25,417 DEBUG zuul.IndependentPipelineManager: Checking for changes needed by <NullChange for hp/gozer-canary>: | 23:24 |
| yolanda | 2015-11-08 23:24:25,417 DEBUG zuul.IndependentPipelineManager: Changeish does not support dependencies | 23:24 |
| yolanda | 2015-11-08 23:24:25,418 DEBUG zuul.IndependentPipelineManager: Finished queue processor: periodic-gozer-canary (changed: False) | 23:24 |
| *** achanda has joined #openstack-infra | 23:26 | |
| *** yamamoto has joined #openstack-infra | 23:26 | |
| openstackgerrit | James E. Blair proposed openstack/gertty: Add navigation to interactive search https://review.openstack.org/238263 | 23:28 |
| openstackgerrit | James E. Blair proposed openstack/gertty: Add interactive search to diff view https://review.openstack.org/238262 | 23:28 |
| openstackgerrit | James E. Blair proposed openstack/gertty: Add a simple kill ring https://review.openstack.org/238261 | 23:28 |
| openstackgerrit | James E. Blair proposed openstack/gertty: Separate search and refine search commands https://review.openstack.org/238260 | 23:28 |
| openstackgerrit | James E. Blair proposed openstack/gertty: Fix syncing changes with comments on a missing file https://review.openstack.org/242883 | 23:28 |
| openstackgerrit | James E. Blair proposed openstack/gertty: Add help entries for kill, yank, isearch https://review.openstack.org/242884 | 23:28 |
| openstackgerrit | James E. Blair proposed openstack/gertty: Support multiple key input https://review.openstack.org/242885 | 23:28 |
| *** yamamoto has quit IRC | 23:28 | |
| *** yamamoto has joined #openstack-infra | 23:28 | |
| *** yamamoto has quit IRC | 23:32 | |
| *** pahuang has quit IRC | 23:33 | |
| yolanda | jhesketh, we have several of: | 23:33 |
| yolanda | Traceback (most recent call last): | 23:33 |
| yolanda | File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 851, in run | 23:33 |
| yolanda | self.process_result_queue() | 23:33 |
| yolanda | File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 934, in process_result_queue | 23:33 |
| yolanda | self._doMergeCompletedEvent(event) | 23:33 |
| yolanda | File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 976, in _doMergeCompletedEvent | 23:33 |
| yolanda | pipeline.manager.onMergeCompleted(event) | 23:33 |
| yolanda | File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 1528, in onMergeCompleted | 23:33 |
| yolanda | build_set.commit = item.change.newrev | 23:33 |
| yolanda | AttributeError: 'NullChange' object has no attribute 'newrev' | 23:33 |
| yolanda | would you say that's benign? | 23:33 |
| *** asettle has quit IRC | 23:34 | |
| jhesketh | yolanda: hmm, doesn't lok benign... How long have you been seeing those for? | 23:36 |
| yolanda | i resynced today and it started | 23:36 |
| *** btully has joined #openstack-infra | 23:36 | |
| yolanda | changes on check, gate look fine | 23:36 |
| yolanda | it seems to happen on periodical jobs | 23:36 |
| yolanda | so, when you don't have a change related to tyat | 23:36 |
| yolanda | that | 23:36 |
| yolanda | elif event.updated: | 23:37 |
| yolanda | build_set.commit = item.change.newrev | 23:37 |
| yolanda | jhesketh, i was wondering if adding a check for item.change will be enough, because after that you find: | 23:37 |
| yolanda | if not build_set.commit: | 23:37 |
| yolanda | self.log.info("Unable to merge change %s" % item.change) | 23:37 |
| yolanda | self.pipeline.setUnableToMerge(item) | 23:37 |
| cody-somerville | You don't merge periodic jobs | 23:39 |
| yolanda | so seems that zuul is missing that use case | 23:40 |
| cody-somerville | I think the error is benign. I've seen it tons of times before. | 23:40 |
| yolanda | first time i've noticed that. I don't see jobs failing, that's right | 23:41 |
| jhesketh | cody-somerville: possibly is, although it's odd that it wasn't there before | 23:41 |
| yolanda | anyway, if we can avoid that zuul throws that, it will be less smary | 23:41 |
| yolanda | scary | 23:41 |
| jhesketh | I wonder if it's related to https://review.openstack.org/#/c/207094/9 | 23:41 |
| jhesketh | doesn't look like it should be though | 23:41 |
| yolanda | jhesketh, so cody is right, looking at older logs, it's present in all of our zuul logs | 23:45 |
| yolanda | i could file a simple change to avoid that,and just fall on "pipeline not able to merge" | 23:45 |
| yolanda | although for periodical, that's the expected behaviour... | 23:45 |
| *** dingyichen has joined #openstack-infra | 23:51 | |
| yolanda | jhesketh https://review.openstack.org/#/c/71628/7/zuul/scheduler.py | 23:52 |
| *** sflanigan_afk is now known as sflanigan | 23:55 | |
| *** August1914 has quit IRC | 23:56 | |
| *** Piet has quit IRC | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!