| fungi | zaro: i think jamespage will have to do it (or ken conley but i don't see him in here) | 00:00 |
|---|---|---|
| *** ZZelle_ has quit IRC | 00:00 | |
| *** serverascode____ is now known as serverascode | 00:01 | |
| openstackgerrit | Merged openstack-infra/project-config: Cleanup tempest-lib job list https://review.openstack.org/145593 | 00:02 |
| *** patrickeast has joined #openstack-infra | 00:05 | |
| *** dimtruck is now known as zz_dimtruck | 00:05 | |
| *** baoli has quit IRC | 00:05 | |
| *** mbacchi has quit IRC | 00:06 | |
| openstackgerrit | Merged openstack/requirements: Update oslo.middleware to 0.3.0 https://review.openstack.org/145594 | 00:06 |
| zaro | fungi: thanks. shall i just go ahead and use an openstack project? would hate to create a testing bug against one of those projects. | 00:08 |
| fungi | zaro: i suppose you could temporarily register a lp project corresponding to one of the gtest projects in review-dev | 00:10 |
| zaro | fungi: i think i would prefer using an openstack project :) | 00:12 |
| *** teran has joined #openstack-infra | 00:12 | |
| *** markvoelker has quit IRC | 00:14 | |
| zaro | fungi: i looked into removing local mysql from review-dev. not sure whether it's a good idea now. | 00:14 |
| *** marun has quit IRC | 00:14 | |
| zaro | fungi: puppet is all configured to make local mysql the default connection for gerrit. we've just overriden with connection to trove. | 00:15 |
| fungi | zaro: yeah, seems fine to leave it on there | 00:16 |
| openstackgerrit | Merged openstack/requirements: Add new angular deps to global-requirements.txt for Horizon https://review.openstack.org/139298 | 00:17 |
| *** shashankhegde has joined #openstack-infra | 00:18 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: create the ability to pass through devstack config https://review.openstack.org/145321 | 00:18 |
| *** andreykurilin has quit IRC | 00:18 | |
| openstackgerrit | Khai Do proposed stackforge/python-jenkins: this is for testing, DO NOT REVIEW https://review.openstack.org/145956 | 00:21 |
| *** sabeen has quit IRC | 00:22 | |
| ianw | clarkb / jeblair : http://nodepool.openstack.org/image.log <- not updating? | 00:25 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Broaden pattern match for jenkins-sudo-grep.sh https://review.openstack.org/145978 | 00:25 |
| fungi | jeblair: clarkb: ^ turns out that was also broken on trusty, for example http://logs.openstack.org/30/141130/10/check/gate-neutron-python27/f94ae9b/console.html | 00:26 |
| fungi | and with that, i'm heading offline for the nighth | 00:28 |
| *** sarob has quit IRC | 00:28 | |
| fungi | night | 00:28 |
| * fungi vanishes in a puff of logic | 00:28 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Cloner: handle missing ZUUL_BRANCH/_REF https://review.openstack.org/145980 | 00:32 |
| jeblair | clarkb, fungi, sdague: ^ that's the last bit the zuul cloner needed to support all the use cases we're doing in devstack-gate. after that lands i will start to remove the git operations from d-g. | 00:33 |
| *** sabari is now known as zz_sabari | 00:34 | |
| *** Kiall has quit IRC | 00:34 | |
| *** Kiall has joined #openstack-infra | 00:36 | |
| openstackgerrit | Khai Do proposed stackforge/python-jenkins: this is for testing, DO NOT REVIEW https://review.openstack.org/145956 | 00:36 |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config: Move Nova V2.1 job from experimental to check/gate https://review.openstack.org/145123 | 00:38 |
| jeblair | ianw: it looks current to me? | 00:38 |
| *** otter768 has joined #openstack-infra | 00:39 | |
| *** Masahiro has joined #openstack-infra | 00:40 | |
| ianw | jeblair: last entry is '2015-01-08 19:26:20,796 INFO nodepool.image.build.devstack-trusty-dib: ' | 00:42 |
| *** shayneburgess has quit IRC | 00:43 | |
| jeblair | ianw: yep, right after an exit 1 | 00:43 |
| *** asettle-mtg is now known as asettle | 00:43 | |
| *** Masahiro has quit IRC | 00:44 | |
| jeblair | ianw: switching back to the main debug log, it then logged: 2015-01-08 19:26:20,803 ERROR nodepool.DiskImageBuilderThread: Exception building DIB image: | 00:45 |
| jeblair | ianw: so i think that's correctly logged the end of a failed image build | 00:45 |
| *** wuhg has joined #openstack-infra | 00:45 | |
| *** yaguang has joined #openstack-infra | 00:46 | |
| ianw | jeblair: it hasn't successfully built a f21 image, i think i should be retrying | 00:46 |
| ianw | it should be retrying | 00:46 |
| *** teran_ has joined #openstack-infra | 00:46 | |
| ianw | all the f21 builds seem to stop with | 00:46 |
| ianw | 015-01-08 19:23:03,324 INFO nodepool.image.build.rax-iad.devstack-f21: + sudo /bin/bash /root/system-config/install_modules.sh | 00:46 |
| jeblair | ianw: oh, i bet they got stuck when clarkb restarted nodepool | 00:48 |
| *** tqtran_ has quit IRC | 00:48 | |
| *** sarob has joined #openstack-infra | 00:49 | |
| *** teran__ has joined #openstack-infra | 00:49 | |
| ianw | sounds likely... | 00:49 |
| *** teran has quit IRC | 00:49 | |
| jeblair | i'll see if deleting them from the command line fixes it. we may need to add something to nodepool to delete all building images on startup like we do with nodes | 00:49 |
| *** otter768 has quit IRC | 00:49 | |
| *** ivar-lazzaro has joined #openstack-infra | 00:50 | |
| *** teran_ has quit IRC | 00:51 | |
| jeblair | ianw: yeah, i think that's unsticking them | 00:51 |
| zaro | jeblair: i'm seeing this problem in review-dev, https://groups.google.com/d/msg/repo-discuss/rR_ouEeAQzU/bA70CYLT0t0J | 00:52 |
| zaro | you have any idea which would be the prefered solution there? | 00:53 |
| zaro | mordred, fungi: ^ maybe you know? relates to mysql | 00:54 |
| jeblair | zaro: unless i'm mistaken, that's been a perpetual problem on review-dev and will go away after a few failures. it never happens on prod because we're so busy. | 00:54 |
| jeblair | zaro: there's a pool of maybe 4 or 5 connections, and once they all fail, you're good for a while | 00:55 |
| *** ivar-lazzaro has quit IRC | 00:55 | |
| *** bhuvan_ has quit IRC | 00:55 | |
| zaro | it's comes up after leaving the UI idel for a few mins. | 00:56 |
| ianw | jeblair: ok, image.log doesn't have anything new, yet | 00:56 |
| zaro | browser refresh fixes it. | 00:56 |
| *** gmurphy has joined #openstack-infra | 00:56 | |
| *** alugovoi has quit IRC | 00:58 | |
| zaro | jeblair: so it's unlikely to happen on review.o.o but should we not apply one of those fixes to eliminate the problem? | 00:59 |
| *** Masahiro has joined #openstack-infra | 00:59 | |
| *** serverascode has left #openstack-infra | 00:59 | |
| jeblair | zaro: maybe, if you want to look into the implications of the autoreconnect flag. i have not used that with jdbc, so don't know how it will manifest | 01:00 |
| *** mfink has joined #openstack-infra | 01:03 | |
| zaro | i'm not real familiar with DBs so I thought i would just poke for advice on which suggestion in the thread would be best,, autoreconnect or removing the connection pool. | 01:04 |
| *** chuckC_ has joined #openstack-infra | 01:04 | |
| jeblair | zaro: i don't see a reason to remove the connection pool since that would have a small impact on performance and we're not hitting the problem in prod | 01:04 |
| jeblair | ianw: 2015-01-09 01:01:48,477 DEBUG nodepool.SnapshotImageUpdater: Image id: 4375 waiting for server 04692001-0b2f-4070-9bf7-f5797885bc3c | 01:05 |
| jeblair | ianw: the first one is just now getting a server | 01:05 |
| jeblair | ianw: should see something soon | 01:05 |
| *** xgerman has quit IRC | 01:05 | |
| *** ChuckC has quit IRC | 01:06 | |
| *** chuckC__ has joined #openstack-infra | 01:07 | |
| *** gyee has quit IRC | 01:08 | |
| *** dannywilson has quit IRC | 01:09 | |
| *** chuckC_ has quit IRC | 01:09 | |
| *** chuckC__ is now known as chuckC_ | 01:09 | |
| jeblair | zaro: the times in that error log are very short | 01:09 |
| jeblair | zaro: less than 5 minutes | 01:09 |
| jeblair | zaro: that's not the old issue at all. something has changed. | 01:09 |
| *** sputnik13 has quit IRC | 01:10 | |
| zaro | jeblair: here's the log from review-dev http://paste.openstack.org/show/156045/ | 01:11 |
| *** jyuso716 has joined #openstack-infra | 01:11 | |
| jeblair | zaro: yeah, i know, i was just looking at it. :) | 01:11 |
| jeblair | zaro: "175,201 milliseconds ago" is 3 minutes | 01:12 |
| zaro | yeah that sounds right. | 01:12 |
| jeblair | zaro: that's not the normal 12 hour timeout | 01:12 |
| zaro | i think it's just shorter idle now. but seems like same issue. | 01:13 |
| jeblair | zaro: the server determines the idle timeout | 01:13 |
| openstackgerrit | Travis Tripp proposed openstack/requirements: Remove old deprecated xstatic angular packages https://review.openstack.org/145096 | 01:14 |
| *** jyuso has quit IRC | 01:14 | |
| *** jyuso- is now known as jyuso | 01:14 | |
| zaro | jeblair: i assume you mean the db server? | 01:15 |
| jeblair | zaro: yep | 01:15 |
| *** patrickeast has quit IRC | 01:16 | |
| *** mtanino has quit IRC | 01:17 | |
| zaro | jeblair: are review & review-dev using the same Trove instance? | 01:18 |
| jeblair | zaro: i certainly hope not | 01:18 |
| *** patrickeast has joined #openstack-infra | 01:18 | |
| zaro | so does that mean we need to update the db to set higher timeout? | 01:19 |
| *** MaxV has joined #openstack-infra | 01:20 | |
| jeblair | zaro: | wait_timeout | 28800 | | 01:20 |
| jeblair | it's set to the default of 8 hours | 01:20 |
| zaro | hmm, i see. | 01:20 |
| jeblair | zaro: don't use review-dev for a few mins | 01:21 |
| *** lttrl has quit IRC | 01:23 | |
| jeblair | zaro: okay, you can use it again | 01:23 |
| jeblair | zaro: the connections last for exactly 120 seconds | 01:23 |
| jeblair | (of idle time) | 01:23 |
| *** lttrl has joined #openstack-infra | 01:24 | |
| *** MaxV has quit IRC | 01:24 | |
| *** nosnos has joined #openstack-infra | 01:27 | |
| jeblair | 01:27:00.071262 IP 10.188.32.101.mysql > review-dev.openstack.org.50982: Flags [F.], seq 36658, ack 8165, win 501, options [nop,nop,TS val 4139423225 ecr 2207611380], length 0 | 01:27 |
| jeblair | 01:27:00.109899 IP review-dev.openstack.org.50982 > 10.188.32.101.mysql: Flags [.], ack 36659, win 331, options [nop,nop,TS val 2207641365 ecr 4139423225], length 0 | 01:27 |
| jeblair | zaro: the disconnection is coming from the mysql server | 01:27 |
| *** lttrl has quit IRC | 01:29 | |
| zaro | interesting, sorry i can't be more of help | 01:30 |
| jeblair | i can't either, i have to run | 01:30 |
| *** spzala has quit IRC | 01:31 | |
| *** lttrl has joined #openstack-infra | 01:32 | |
| zaro | jeblair: line 176 https://gerrit.googlesource.com/gerrit/+/stable-2.9/gerrit-sshd/src/main/java/com/google/gerrit/sshd/SshDaemon.java | 01:34 |
| *** unicell has joined #openstack-infra | 01:35 | |
| zaro | not sure how that works yet, but might be a clue. | 01:35 |
| * zaro gotta go too | 01:35 | |
| *** lttrl has quit IRC | 01:36 | |
| *** melwitt has quit IRC | 01:36 | |
| *** signed8bit is now known as signed8bit_ZZZzz | 01:37 | |
| *** weshay has quit IRC | 01:40 | |
| *** banix has quit IRC | 01:40 | |
| *** wenlock has quit IRC | 01:42 | |
| *** chuckC_ has quit IRC | 01:42 | |
| *** dprince has joined #openstack-infra | 01:42 | |
| *** ivar-lazzaro has joined #openstack-infra | 01:45 | |
| *** bdpayne has quit IRC | 01:46 | |
| *** ivar-lazzaro has quit IRC | 01:50 | |
| *** yamahata has quit IRC | 01:53 | |
| *** dims__ has joined #openstack-infra | 01:55 | |
| *** dims__ has quit IRC | 01:55 | |
| *** dims__ has joined #openstack-infra | 01:55 | |
| *** xianghuihui has joined #openstack-infra | 01:59 | |
| *** xianghuihui has quit IRC | 02:05 | |
| *** amotoki has joined #openstack-infra | 02:05 | |
| *** dprince has quit IRC | 02:08 | |
| *** signed8bit_ZZZzz is now known as signed8bit | 02:11 | |
| *** eharney_ has quit IRC | 02:12 | |
| *** signed8bit has quit IRC | 02:13 | |
| *** shashankhegde has quit IRC | 02:13 | |
| *** ddieterly has joined #openstack-infra | 02:13 | |
| *** mohitsharma has joined #openstack-infra | 02:15 | |
| *** mohitsharma has quit IRC | 02:15 | |
| *** mohitsharma has joined #openstack-infra | 02:16 | |
| *** hdd has quit IRC | 02:17 | |
| *** oomichi has joined #openstack-infra | 02:18 | |
| *** asselin has quit IRC | 02:22 | |
| *** kmartin has quit IRC | 02:22 | |
| *** asselin has joined #openstack-infra | 02:23 | |
| *** kmartin has joined #openstack-infra | 02:23 | |
| *** sarob has quit IRC | 02:23 | |
| *** ddieterl_ has joined #openstack-infra | 02:25 | |
| *** ddieterly has quit IRC | 02:27 | |
| *** ddieterly has joined #openstack-infra | 02:27 | |
| *** jamielennox is now known as jamielennox|away | 02:30 | |
| *** ddieterl_ has quit IRC | 02:31 | |
| *** tkelsey has joined #openstack-infra | 02:32 | |
| *** pcaruana is now known as pcaruana|afk| | 02:34 | |
| *** jamielennox|away is now known as jamielennox | 02:37 | |
| *** tkelsey has quit IRC | 02:37 | |
| *** chuckC_ has joined #openstack-infra | 02:40 | |
| *** kaisers1 has joined #openstack-infra | 02:41 | |
| *** shashankhegde has joined #openstack-infra | 02:41 | |
| *** hdd has joined #openstack-infra | 02:42 | |
| *** ricolin has joined #openstack-infra | 02:43 | |
| *** kaisers has quit IRC | 02:43 | |
| *** amotoki has quit IRC | 02:45 | |
| *** sarob has joined #openstack-infra | 02:46 | |
| *** chlong has joined #openstack-infra | 02:49 | |
| *** otter768 has joined #openstack-infra | 02:50 | |
| *** jamielennox is now known as jamielennox|away | 02:51 | |
| *** otter768 has quit IRC | 02:54 | |
| *** ChuckC has joined #openstack-infra | 02:56 | |
| *** yamahata has joined #openstack-infra | 02:56 | |
| *** kmartin has quit IRC | 02:57 | |
| *** kmartin has joined #openstack-infra | 02:57 | |
| *** chuckC_ has quit IRC | 02:58 | |
| *** Ryan_Lane has quit IRC | 03:00 | |
| *** jamielennox|away is now known as jamielennox | 03:01 | |
| *** amotoki has joined #openstack-infra | 03:03 | |
| *** sarob has quit IRC | 03:04 | |
| *** andreaf has quit IRC | 03:04 | |
| *** andreaf has joined #openstack-infra | 03:04 | |
| *** chuckC_ has joined #openstack-infra | 03:05 | |
| *** hdd has quit IRC | 03:07 | |
| *** cpallares has quit IRC | 03:08 | |
| *** MaxV has joined #openstack-infra | 03:08 | |
| *** sarob has joined #openstack-infra | 03:10 | |
| *** MaxV has quit IRC | 03:13 | |
| *** patrickeast has quit IRC | 03:22 | |
| *** nosnos has quit IRC | 03:24 | |
| *** david-lyle has joined #openstack-infra | 03:29 | |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack/requirements: Avoid to use boto 2.35.0 for workaround https://review.openstack.org/146006 | 03:30 |
| *** MaxV has joined #openstack-infra | 03:31 | |
| *** otter768 has joined #openstack-infra | 03:36 | |
| *** MaxV has quit IRC | 03:36 | |
| *** harlowja is now known as harlowja_away | 03:38 | |
| *** harlowja_at_home has joined #openstack-infra | 03:39 | |
| *** dims__ has quit IRC | 03:40 | |
| *** dims__ has joined #openstack-infra | 03:41 | |
| *** sarob has quit IRC | 03:41 | |
| *** dims__ has quit IRC | 03:45 | |
| *** rwsu has quit IRC | 03:53 | |
| *** teran__ has quit IRC | 03:54 | |
| *** nosnos has joined #openstack-infra | 03:56 | |
| *** xgerman has joined #openstack-infra | 04:01 | |
| *** oomichi has quit IRC | 04:01 | |
| *** teran has joined #openstack-infra | 04:04 | |
| *** links has joined #openstack-infra | 04:07 | |
| *** bradjones has quit IRC | 04:12 | |
| *** ddieterly has quit IRC | 04:13 | |
| *** oomichi has joined #openstack-infra | 04:14 | |
| *** mwagner_lap has joined #openstack-infra | 04:14 | |
| *** rlandy has quit IRC | 04:17 | |
| *** david-lyle has quit IRC | 04:17 | |
| *** shashankhegde has quit IRC | 04:21 | |
| *** stevemar has joined #openstack-infra | 04:23 | |
| *** shayneburgess has joined #openstack-infra | 04:31 | |
| *** tkelsey has joined #openstack-infra | 04:33 | |
| *** tkelsey has quit IRC | 04:38 | |
| *** harlowja_at_home has quit IRC | 04:49 | |
| *** harlowja_at_home has joined #openstack-infra | 04:57 | |
| *** lingxian has joined #openstack-infra | 05:02 | |
| *** zxiiro has quit IRC | 05:03 | |
| lingxian | hi, there, could somebody provide some articles about how to setup my own CI infrastrcture(including gerrit)? I found the posts writen by jaypipes, but some contents are outdated. (e.g. http://www.joinfu.com/2014/02/setting-up-an-external-openstack-testing-system/) | 05:06 |
| *** harlowja_at_home has quit IRC | 05:07 | |
| *** garyh has quit IRC | 05:09 | |
| charz | lingxian: what problems that you face? more detail will be great. | 05:11 |
| *** asettle has quit IRC | 05:15 | |
| lingxian | charz: jaypipes's article talks about setup CI using upstream Gerrit. But I want setup everything necessary that can work together, say gerrit, zuul, JJB, jenkins. | 05:15 |
| lingxian | charz, and what I did is install gerrit by hand, and then using jaypipes's git repo to setup others. | 05:16 |
| charz | lingxian: Do you create a wiki page of your CI server in openstack/3party CI wiki? | 05:18 |
| *** hdd has joined #openstack-infra | 05:19 | |
| *** yfried|afk is now known as yfried | 05:19 | |
| lingxian | charz, no. Maybe I don't make me understood. I just want a OpenStack CI infra used inside my company, it has nothing to do with the upstream. | 05:20 |
| *** nosnos has quit IRC | 05:21 | |
| *** nosnos has joined #openstack-infra | 05:21 | |
| charz | lingxian: ok, The document of you pasted is setup a 3party CI and work with upstream. | 05:23 |
| charz | lingxian: You also need to fetch code from upstream, right? | 05:24 |
| lingxian | charz, the code is hosted in our git server inside. | 05:24 |
| charz | lingxian: ok, I got it. | 05:25 |
| lingxian | charz, :) | 05:25 |
| lingxian | charz, what's your suggestion? | 05:25 |
| *** nosnos has quit IRC | 05:26 | |
| charz | lingxian: I think that's no document to describe that. You need to find source code of accout stackforge in github or wiki. | 05:26 |
| *** xgerman has quit IRC | 05:27 | |
| charz | lingxian: Make sure you can retrieve push log from internal gerrit server, and you can flow the steps of document to setup others. | 05:29 |
| *** bhunter71 has quit IRC | 05:29 | |
| lingxian | charz, so, what I want to do is install gerrit by myself, then install others using jaypipes' git repo(just need to modify something), to see if it works. | 05:29 |
| lingxian | charz, yes, I can see the output through 'ssh -p 29418 kong@review.kong.org gerrit stream-events' | 05:30 |
| charz | lingxian: ok, that's good. | 05:30 |
| charz | lingxian: try setup a job with gerrit plugin on jenkins. | 05:31 |
| *** yfried is now known as yfried|afk | 05:32 | |
| lingxian | charz, ok, when jenkins is installed after the puppet task, I'll check. | 05:33 |
| lingxian | charz, thanks! | 05:33 |
| charz | lingxian: np :) | 05:33 |
| charz | lingxian: r you run whole OpenStack CI in your internal server? | 05:34 |
| lingxian | charz, yes | 05:34 |
| lingxian | charz, is it workable? | 05:35 |
| charz | lingxian: That's a huge project, it must needs a lot of engineers. | 05:35 |
| charz | lingxian: I think that's workable, but how many servers on it? | 05:36 |
| lingxian | charz, AFAIK, gerrit, zuul, jjb, jenkins(master+slave) will do everything I need. But I'm not sure. | 05:37 |
| lingxian | charz, and the nodepool | 05:37 |
| charz | lingxian: cool! | 05:37 |
| *** otter768 has quit IRC | 05:38 | |
| lingxian | charz, I can ignore the real devstack-gate in the early stage. I just want the workflow is oK | 05:39 |
| *** MarkAtwood has joined #openstack-infra | 05:39 | |
| *** Longgeek has joined #openstack-infra | 05:40 | |
| lingxian | charz, say, I push a patch to gerrit, zuul got the event and dicide which jobs to run, and jenkins to his job | 05:40 |
| lingxian | charz, that's it for the first step. | 05:41 |
| lingxian | charz, and then I can consider the complacated parts of the infra | 05:41 |
| *** ddieterly has joined #openstack-infra | 05:43 | |
| *** jyuso has quit IRC | 05:44 | |
| *** ddieterly has quit IRC | 05:45 | |
| *** ddieterly has joined #openstack-infra | 05:45 | |
| *** ddieterly has quit IRC | 05:50 | |
| *** nosnos has joined #openstack-infra | 05:53 | |
| *** mrmartin has joined #openstack-infra | 06:03 | |
| *** MarkAtwood has quit IRC | 06:05 | |
| *** mohitsharma has quit IRC | 06:05 | |
| *** tomoe has joined #openstack-infra | 06:09 | |
| *** mrmartin has quit IRC | 06:14 | |
| *** ivar-lazzaro has joined #openstack-infra | 06:21 | |
| *** zz_avozza is now known as avozza | 06:21 | |
| *** ivar-lazzaro has quit IRC | 06:27 | |
| *** teran_ has joined #openstack-infra | 06:32 | |
| *** pcrews has quit IRC | 06:32 | |
| *** mrmartin has joined #openstack-infra | 06:33 | |
| *** vigneshvar has joined #openstack-infra | 06:34 | |
| *** teran has quit IRC | 06:34 | |
| openstackgerrit | Merged openstack-infra/project-config: Make the Horizon integration tests job voting https://review.openstack.org/145477 | 06:34 |
| *** sdake_ has joined #openstack-infra | 06:36 | |
| *** obondarev has joined #openstack-infra | 06:39 | |
| *** obondarev has quit IRC | 06:39 | |
| *** TravT_ has quit IRC | 06:41 | |
| *** ildikov has quit IRC | 06:41 | |
| *** TravT has joined #openstack-infra | 06:41 | |
| *** sarob has joined #openstack-infra | 06:41 | |
| *** ddieterly has joined #openstack-infra | 06:44 | |
| *** sarob has quit IRC | 06:46 | |
| *** camunoz has quit IRC | 06:46 | |
| *** nosnos has quit IRC | 06:48 | |
| *** ddieterly has quit IRC | 06:49 | |
| *** reed has quit IRC | 06:49 | |
| *** nosnos has joined #openstack-infra | 06:50 | |
| *** thingee has quit IRC | 06:56 | |
| *** Longgeek has quit IRC | 06:58 | |
| *** dims__ has joined #openstack-infra | 07:00 | |
| *** mrunge has joined #openstack-infra | 07:02 | |
| *** mohitsharma has joined #openstack-infra | 07:06 | |
| *** dims__ has quit IRC | 07:06 | |
| *** mohitsharma has quit IRC | 07:10 | |
| *** Longgeek has joined #openstack-infra | 07:11 | |
| *** jamielennox is now known as jamielennox|away | 07:15 | |
| *** mohitsharma has joined #openstack-infra | 07:18 | |
| *** tkelsey has joined #openstack-infra | 07:18 | |
| *** liam_ has joined #openstack-infra | 07:19 | |
| *** liam_ is now known as Guest14772 | 07:20 | |
| *** berendt has joined #openstack-infra | 07:27 | |
| *** ildikov has joined #openstack-infra | 07:28 | |
| *** otter768 has joined #openstack-infra | 07:39 | |
| *** yfried|afk is now known as yfried | 07:39 | |
| *** alugovoi has joined #openstack-infra | 07:41 | |
| *** ddieterly has joined #openstack-infra | 07:44 | |
| *** otter768 has quit IRC | 07:44 | |
| *** coolsvap|afk is now known as coolsvap | 07:49 | |
| *** ddieterly has quit IRC | 07:49 | |
| *** hdd has quit IRC | 07:49 | |
| *** armax has quit IRC | 07:50 | |
| *** yfried is now known as yfried|afk | 07:54 | |
| *** Guest14772 has quit IRC | 07:55 | |
| *** pblaho_ has joined #openstack-infra | 07:55 | |
| *** jcoufal has joined #openstack-infra | 08:04 | |
| *** dhritishikhar has joined #openstack-infra | 08:06 | |
| *** ddieterly has joined #openstack-infra | 08:13 | |
| *** MaxV has joined #openstack-infra | 08:14 | |
| *** bhunter71 has joined #openstack-infra | 08:16 | |
| *** mpaolino has joined #openstack-infra | 08:17 | |
| *** ddieterly has quit IRC | 08:18 | |
| *** MaxV has quit IRC | 08:18 | |
| *** MaxV has joined #openstack-infra | 08:18 | |
| *** afazekas_ has joined #openstack-infra | 08:18 | |
| *** MaxV has quit IRC | 08:20 | |
| *** markus_z has joined #openstack-infra | 08:23 | |
| *** mrmartin has quit IRC | 08:23 | |
| *** e0ne has joined #openstack-infra | 08:24 | |
| *** yamahata has quit IRC | 08:24 | |
| *** hashar has joined #openstack-infra | 08:24 | |
| *** ivar-lazzaro has joined #openstack-infra | 08:25 | |
| *** e0ne has quit IRC | 08:27 | |
| *** tomoe has quit IRC | 08:27 | |
| *** alugovoi has quit IRC | 08:28 | |
| *** ivar-lazzaro has quit IRC | 08:30 | |
| *** vigneshvar has quit IRC | 08:30 | |
| *** vigneshvar has joined #openstack-infra | 08:31 | |
| *** dhritishikhar has quit IRC | 08:34 | |
| *** mpaolino has quit IRC | 08:40 | |
| *** luqas has joined #openstack-infra | 08:40 | |
| *** oomichi has quit IRC | 08:41 | |
| *** Hal_ has joined #openstack-infra | 08:43 | |
| *** yfried|afk is now known as yfried | 08:47 | |
| *** bhunter71 has quit IRC | 08:50 | |
| *** mpaolino has joined #openstack-infra | 08:52 | |
| *** MaxV has joined #openstack-infra | 08:53 | |
| *** dalgaaf has joined #openstack-infra | 08:57 | |
| *** mrmartin has joined #openstack-infra | 08:59 | |
| *** jpich has joined #openstack-infra | 09:01 | |
| *** Masahiro has quit IRC | 09:04 | |
| *** jistr has joined #openstack-infra | 09:05 | |
| *** Masahiro has joined #openstack-infra | 09:05 | |
| *** HeOS has joined #openstack-infra | 09:08 | |
| *** ankit_ag has joined #openstack-infra | 09:09 | |
| *** Longgeek_ has joined #openstack-infra | 09:10 | |
| *** Longgeek has quit IRC | 09:11 | |
| *** boris-42 has quit IRC | 09:13 | |
| *** berendt has quit IRC | 09:13 | |
| *** berendt has joined #openstack-infra | 09:15 | |
| *** derekh has joined #openstack-infra | 09:22 | |
| openstackgerrit | Mike Heald proposed openstack-infra/infra-specs: Proposal for a 'fail-fast' option on jobs https://review.openstack.org/129255 | 09:27 |
| *** marun has joined #openstack-infra | 09:28 | |
| *** ssam2 has joined #openstack-infra | 09:30 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 09:31 | |
| *** berendt has quit IRC | 09:32 | |
| *** dtantsur|afk is now known as dtantsur | 09:33 | |
| *** ZZelle has quit IRC | 09:34 | |
| *** ZZelle has joined #openstack-infra | 09:35 | |
| *** hashar has quit IRC | 09:36 | |
| *** mpaolino has quit IRC | 09:36 | |
| *** hashar has joined #openstack-infra | 09:37 | |
| openstackgerrit | Cyril Roelandt proposed openstack-infra/project-config: Make the py33 gate voting for neutronclient https://review.openstack.org/145430 | 09:38 |
| *** berendt has joined #openstack-infra | 09:40 | |
| *** otter768 has joined #openstack-infra | 09:40 | |
| *** stevemar has quit IRC | 09:44 | |
| *** andreaf has quit IRC | 09:44 | |
| *** smoser has quit IRC | 09:45 | |
| *** otter768 has quit IRC | 09:45 | |
| *** andreaf has joined #openstack-infra | 09:47 | |
| *** mpaolino has joined #openstack-infra | 09:48 | |
| *** andreaf has quit IRC | 09:49 | |
| *** andreaf has joined #openstack-infra | 09:49 | |
| *** smoser has joined #openstack-infra | 09:52 | |
| openstackgerrit | Roman Prykhodchenko proposed openstack-infra/project-config: Add Fuel Client to Stackforge https://review.openstack.org/145843 | 09:57 |
| *** mohitsharma has quit IRC | 09:59 | |
| *** jp_at_hp has joined #openstack-infra | 10:05 | |
| *** KanagarajM has joined #openstack-infra | 10:07 | |
| ankit_ag | Hello All tempest tests failing on https://review.openstack.org/#/c/145738/ | 10:09 |
| ankit_ag | due to eeror message <Response><Errors><Error><Code>AuthFailure</Code><Message>Signature not provided</Message></Error></Errors><RequestID>req-6a3ea6ef-0658-4aaf-8996-0ef1277d9e0d</RequestID></Response> | 10:09 |
| openstackgerrit | Roman Prykhodchenko proposed openstack-infra/project-config: Add Fuel Client to Stackforge https://review.openstack.org/145843 | 10:09 |
| ankit_ag | I could not found any relative bug, do you guys have any idea | 10:10 |
| afazekas_ | Is the boto version changed on pypi ? | 10:11 |
| afazekas_ | I just seen this kind of issue.. | 10:11 |
| *** afazekas_ is now known as afazekas | 10:11 | |
| *** avozza is now known as zz_avozza | 10:16 | |
| afazekas | https://pypi.python.org/pypi/boto/ it is just released | 10:16 |
| openstackgerrit | afazekas proposed openstack/requirements: Deny boto==0.35.0 https://review.openstack.org/146049 | 10:19 |
| *** Masahiro has quit IRC | 10:26 | |
| *** ihrachyshka has joined #openstack-infra | 10:26 | |
| *** wznoinsk has joined #openstack-infra | 10:27 | |
| *** ivar-lazzaro has joined #openstack-infra | 10:27 | |
| *** zz_avozza is now known as avozza | 10:28 | |
| *** ivar-lazzaro has quit IRC | 10:32 | |
| *** e0ne has joined #openstack-infra | 10:36 | |
| gilliard | afazekas: not sure if we have a policy to only deny reqs that we _know_ are broken. | 10:38 |
| gilliard | hence my comment on your patch. | 10:39 |
| *** ldnunes has joined #openstack-infra | 10:40 | |
| *** unicell has quit IRC | 10:43 | |
| *** andreykurilin has joined #openstack-infra | 10:44 | |
| *** gilliard is now known as gilllliard | 10:46 | |
| *** unicell has joined #openstack-infra | 10:50 | |
| *** hashar has quit IRC | 10:54 | |
| afazekas | I see the issue with 2.34.0, the issue might be somewhere else | 10:57 |
| *** pelix has joined #openstack-infra | 10:59 | |
| *** jlibosva has joined #openstack-infra | 11:00 | |
| *** aysyd has joined #openstack-infra | 11:02 | |
| gilllliard | 2.34.0 ? | 11:02 |
| afazekas | I logged in to other wise broken setup :( , but yes I saw 2.34.0 | 11:03 |
| *** e0ne is now known as e0ne_ | 11:03 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/storyboard-webclient: Add maxlength validation for fields https://review.openstack.org/146055 | 11:05 |
| *** pc_m has joined #openstack-infra | 11:09 | |
| *** KanagarajM has quit IRC | 11:12 | |
| *** links has quit IRC | 11:13 | |
| *** e0ne_ has quit IRC | 11:13 | |
| *** alexpilotti has joined #openstack-infra | 11:14 | |
| *** rfolco has joined #openstack-infra | 11:18 | |
| *** amotoki_ has joined #openstack-infra | 11:19 | |
| afazekas | gilllliard, looks like the gate working with 146049 , is 'boto>=2.32.1,<2.35.0' ok as version spec ? | 11:21 |
| Daviey | Isn't this a bit much? From: "A mailing list for the OpenStack Stable Branch test reports. <openstack-stable-maint@lists.openstack.org>" | 11:22 |
| gilllliard | afazekas: works for me! I don't know the policy - I was just asking. | 11:22 |
| gilllliard | Because I think they are likely to release 2.35.x soon (perhaps) which would break us again | 11:23 |
| *** rfolco has quit IRC | 11:23 | |
| *** e0ne has joined #openstack-infra | 11:23 | |
| *** rfolco has joined #openstack-infra | 11:25 | |
| *** adalbas has joined #openstack-infra | 11:26 | |
| *** Masahiro has joined #openstack-infra | 11:27 | |
| afazekas | gilllliard: we usually open a ticket when something fails, in this case very likely it will be fixed in the next release | 11:29 |
| *** rfolco has quit IRC | 11:30 | |
| *** rfolco has joined #openstack-infra | 11:30 | |
| *** Masahiro has quit IRC | 11:31 | |
| *** mpaolino has quit IRC | 11:34 | |
| *** rfolco has quit IRC | 11:35 | |
| *** rfolco has joined #openstack-infra | 11:35 | |
| *** rfolco has quit IRC | 11:39 | |
| *** ssam2 has quit IRC | 11:40 | |
| *** ssam2 has joined #openstack-infra | 11:40 | |
| *** otter768 has joined #openstack-infra | 11:41 | |
| *** tomoe has joined #openstack-infra | 11:44 | |
| openstackgerrit | afazekas proposed openstack/requirements: Limit boto version to <2.35.0 https://review.openstack.org/146049 | 11:44 |
| *** otter768 has quit IRC | 11:46 | |
| *** mohitsharma has joined #openstack-infra | 11:46 | |
| *** avozza is now known as zz_avozza | 11:49 | |
| *** andreykurilin has quit IRC | 11:49 | |
| *** jcoufal_ has joined #openstack-infra | 11:50 | |
| *** cdent has joined #openstack-infra | 11:50 | |
| *** mfink has quit IRC | 11:50 | |
| openstackgerrit | Radomir Dopieralski proposed openstack-infra/project-config: Enable Selenium tests for stackforge/tuskar-ui-extras https://review.openstack.org/146061 | 11:52 |
| *** rfolco has joined #openstack-infra | 11:52 | |
| *** jcoufal has quit IRC | 11:52 | |
| *** tomoe has quit IRC | 11:52 | |
| *** nosnos has quit IRC | 11:54 | |
| *** rfolco has quit IRC | 11:56 | |
| *** erlon_ has joined #openstack-infra | 11:57 | |
| *** rfolco has joined #openstack-infra | 11:57 | |
| *** e0ne is now known as e0ne_ | 11:58 | |
| *** mrmartin has quit IRC | 11:58 | |
| *** arxcruz has joined #openstack-infra | 11:59 | |
| *** rfolco has quit IRC | 12:02 | |
| *** mohitsharma has quit IRC | 12:03 | |
| *** e0ne_ has quit IRC | 12:03 | |
| *** rdopiera has joined #openstack-infra | 12:05 | |
| rdopiera | is Andreas Jaeger here? | 12:05 |
| *** EmilienM|afk is now known as EmilienM | 12:07 | |
| *** mohitsharma has joined #openstack-infra | 12:09 | |
| openstackgerrit | Radomir Dopieralski proposed openstack-infra/project-config: Enable Selenium tests for stackforge/tuskar-ui-extras https://review.openstack.org/146061 | 12:11 |
| *** e0ne has joined #openstack-infra | 12:13 | |
| *** rfolco has joined #openstack-infra | 12:14 | |
| *** dizquierdo has joined #openstack-infra | 12:15 | |
| *** boris-42 has joined #openstack-infra | 12:15 | |
| *** ldnunes has quit IRC | 12:17 | |
| *** rfolco has quit IRC | 12:18 | |
| *** ldnunes has joined #openstack-infra | 12:20 | |
| *** Longgeek_ has quit IRC | 12:21 | |
| *** ldnunes has quit IRC | 12:22 | |
| *** mrmartin has joined #openstack-infra | 12:22 | |
| *** ldnunes has joined #openstack-infra | 12:24 | |
| *** rfolco has joined #openstack-infra | 12:30 | |
| *** ivar-lazzaro has joined #openstack-infra | 12:31 | |
| *** chlong has quit IRC | 12:32 | |
| *** jp_at_hp has quit IRC | 12:33 | |
| *** mohitsharma has quit IRC | 12:34 | |
| *** rfolco has quit IRC | 12:35 | |
| *** jp_at_hp has joined #openstack-infra | 12:35 | |
| *** ivar-lazzaro has quit IRC | 12:36 | |
| *** rfolco has joined #openstack-infra | 12:36 | |
| *** hashar has joined #openstack-infra | 12:38 | |
| *** erlon_ is now known as erlon | 12:39 | |
| *** rfolco has quit IRC | 12:41 | |
| *** jedimike has joined #openstack-infra | 12:41 | |
| *** rfolco has joined #openstack-infra | 12:41 | |
| *** Longgeek has joined #openstack-infra | 12:47 | |
| *** EmilienM is now known as EmilienM|afk | 12:52 | |
| *** jp_at_hp has quit IRC | 12:55 | |
| *** rfolco has quit IRC | 12:57 | |
| *** jp_at_hp has joined #openstack-infra | 12:59 | |
| *** jp_at_hp has quit IRC | 12:59 | |
| *** jp_at_hp has joined #openstack-infra | 12:59 | |
| *** dhritishikhar has joined #openstack-infra | 12:59 | |
| *** Masahiro has joined #openstack-infra | 12:59 | |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config: Add networking-cisco-apic to stackforge https://review.openstack.org/145928 | 13:01 |
| *** johnthetubaguy is now known as zz_johnthetubagu | 13:02 | |
| *** MarkAtwood has joined #openstack-infra | 13:03 | |
| *** MarkAtwood has quit IRC | 13:03 | |
| *** Masahiro has quit IRC | 13:04 | |
| *** baoli has joined #openstack-infra | 13:07 | |
| *** baoli_ has joined #openstack-infra | 13:08 | |
| *** baoli_ has quit IRC | 13:08 | |
| *** EmilienM|afk is now known as EmilienM | 13:09 | |
| *** baoli_ has joined #openstack-infra | 13:09 | |
| *** baoli has quit IRC | 13:12 | |
| *** bswartz has quit IRC | 13:14 | |
| *** radez_g0n3 is now known as radez | 13:19 | |
| *** rdopiera has quit IRC | 13:19 | |
| *** rdopiera has joined #openstack-infra | 13:19 | |
| *** dprince has joined #openstack-infra | 13:21 | |
| *** rlandy has joined #openstack-infra | 13:24 | |
| *** mpaolino has joined #openstack-infra | 13:24 | |
| openstackgerrit | Radomir Dopieralski proposed openstack-infra/project-config: Enable Selenium tests for stackforge/tuskar-ui-extras https://review.openstack.org/146061 | 13:27 |
| *** hashar has quit IRC | 13:27 | |
| *** rdopiera has quit IRC | 13:27 | |
| *** dims__ has joined #openstack-infra | 13:28 | |
| jaypipes | lingxian: pong. sorry, our timezones are quite offset. can you email me at jaypipes@gmail.com and I will try to help you on your CI tasks as much as I can. thx! | 13:29 |
| *** rdopiera has joined #openstack-infra | 13:29 | |
| *** ddieterly has joined #openstack-infra | 13:31 | |
| *** dims_ has joined #openstack-infra | 13:33 | |
| *** berendt has quit IRC | 13:34 | |
| *** dkranz has joined #openstack-infra | 13:34 | |
| *** hashar has joined #openstack-infra | 13:34 | |
| *** dims__ has quit IRC | 13:35 | |
| *** mbacchi has joined #openstack-infra | 13:35 | |
| *** dkliban_afk is now known as dkliban | 13:36 | |
| *** _nadya_ has joined #openstack-infra | 13:37 | |
| *** dims_ has quit IRC | 13:37 | |
| *** ankit_ag has quit IRC | 13:38 | |
| *** _nadya_ has quit IRC | 13:40 | |
| *** dims__ has joined #openstack-infra | 13:41 | |
| *** otter768 has joined #openstack-infra | 13:42 | |
| openstackgerrit | Radomir Dopieralski proposed openstack-infra/project-config: Enable Selenium tests for stackforge/tuskar-ui-extras https://review.openstack.org/146061 | 13:45 |
| *** otter768 has quit IRC | 13:46 | |
| openstackgerrit | Radomir Dopieralski proposed openstack-infra/project-config: Enable Selenium tests for stackforge/tuskar-ui-extras https://review.openstack.org/146061 | 13:48 |
| *** weshay has joined #openstack-infra | 13:49 | |
| *** _nadya_ has joined #openstack-infra | 13:51 | |
| *** dkranz has quit IRC | 13:52 | |
| *** pcaruana|afk| has quit IRC | 13:53 | |
| *** annegentle has joined #openstack-infra | 13:54 | |
| *** _nadya_ has joined #openstack-infra | 13:54 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:55 | |
| ZZelle | Hi everyone | 13:57 |
| ZZelle | It seems there is a trouble with boto in neutron tests | 13:58 |
| rdopiera | what is the nature of said problem? | 13:59 |
| ZZelle | rdopiera, http://logs.openstack.org/84/133484/17/check/check-tempest-dsvm-neutron-full/068768b/console.html | 13:59 |
| ZZelle | rdopiera, boto version has been updated on pypi yesterday | 14:00 |
| salv-orlando | ZZelle: http://goo.gl/LyrMMy | 14:01 |
| *** zz_johnthetubagu is now known as johnthetubaguy | 14:02 | |
| salv-orlando | I am pretty sure people here already know what's going on - it might be a coincidence but failures started at midnight | 14:02 |
| salv-orlando | ZZelle: not just neutron - all boto tests fail. | 14:03 |
| salv-orlando | regardless of the job where they are executed | 14:03 |
| ZZelle | salv-orlando, indeed it seems tempest tests are no more working | 14:04 |
| *** vigneshvar has quit IRC | 14:04 | |
| salv-orlando | ZZelle: afazekas has a bug report -> https://bugs.launchpad.net/tempest/+bug/1408987 | 14:05 |
| uvirtbot | Launchpad bug 1408987 in nova "tempest failing with boto==2.35.0" [Critical,Confirmed] | 14:05 |
| salv-orlando | ZZelle: and a patch which is in the gate queue... so nothing to be worried about, it's going to be fixed soon | 14:05 |
| *** rdopiera has left #openstack-infra | 14:06 | |
| *** mattfarina has joined #openstack-infra | 14:07 | |
| *** bswartz has joined #openstack-infra | 14:08 | |
| *** mfink has joined #openstack-infra | 14:08 | |
| *** coolsvap is now known as coolsvap|afk | 14:10 | |
| *** ddieterly has quit IRC | 14:11 | |
| *** signed8bit has joined #openstack-infra | 14:11 | |
| ZZelle | salv-orlando, great | 14:11 |
| *** dkranz has joined #openstack-infra | 14:11 | |
| mpaolino | ZZelle, salv-orlando I have the same problem with https://review.openstack.org/#/c/108714/1 | 14:12 |
| ZZelle | salv-orlando, so it's a good time for a beer :) | 14:12 |
| ZZelle | mpaolino, patience is the response :) | 14:12 |
| mpaolino | should we "recheck" later? | 14:12 |
| mpaolino | or the tests will be triggered automagically? :) | 14:13 |
| ZZelle | mpaolino, no, we should wait and then recheck manually | 14:14 |
| salv-orlando | mpaolino: wait for this to merge otherwise rechecks are always going to fail https://review.openstack.org/#/c/146049/ | 14:15 |
| mpaolino | thanks guys | 14:15 |
| *** dustins has joined #openstack-infra | 14:19 | |
| *** coolsvap|afk is now known as coolsvap | 14:19 | |
| *** ryanpetrello has joined #openstack-infra | 14:20 | |
| *** amotoki_ has quit IRC | 14:20 | |
| *** e0ne is now known as e0ne_ | 14:20 | |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config: Enable functional testing job for ironic-discoverd https://review.openstack.org/146092 | 14:21 |
| openstackgerrit | Merged openstack/requirements: Limit boto version to <2.35.0 https://review.openstack.org/146049 | 14:26 |
| *** miqui_ has quit IRC | 14:26 | |
| *** _nadya_ has quit IRC | 14:30 | |
| *** e0ne_ has quit IRC | 14:31 | |
| *** mriedem has joined #openstack-infra | 14:32 | |
| *** ivar-lazzaro has joined #openstack-infra | 14:33 | |
| *** dhritishikhar has quit IRC | 14:33 | |
| *** ivar-lazzaro has quit IRC | 14:37 | |
| *** yfried has quit IRC | 14:38 | |
| *** amitgandhinz has joined #openstack-infra | 14:40 | |
| *** teran_ has quit IRC | 14:41 | |
| *** shayneburgess has quit IRC | 14:42 | |
| *** ddieterly has joined #openstack-infra | 14:43 | |
| *** armax has joined #openstack-infra | 14:46 | |
| *** Masahiro has joined #openstack-infra | 14:48 | |
| *** mjturek has joined #openstack-infra | 14:50 | |
| *** dannywilson has joined #openstack-infra | 14:52 | |
| *** Masahiro has quit IRC | 14:52 | |
| *** pcaruana|afk| has joined #openstack-infra | 14:52 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for infra/pypi mirror bug 1355050 https://review.openstack.org/145808 | 14:53 |
| uvirtbot | Launchpad bug 1355050 in openstack-ci "markupsafe package not found" [Medium,Triaged] https://launchpad.net/bugs/1355050 | 14:53 |
| *** xyang0 has joined #openstack-infra | 14:54 | |
| *** rfolco has joined #openstack-infra | 14:54 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: add query for bug 1408814 https://review.openstack.org/145969 | 14:56 |
| uvirtbot | Launchpad bug 1408814 in cinder "Image could not be found after upload. The image may have been deleted during the upload." [Undecided,New] https://launchpad.net/bugs/1408814 | 14:56 |
| *** rickerc has quit IRC | 14:56 | |
| *** zxiiro has joined #openstack-infra | 14:58 | |
| *** rickerc has joined #openstack-infra | 14:59 | |
| *** rfolco has quit IRC | 14:59 | |
| *** dannywilson has quit IRC | 14:59 | |
| *** rfolco has joined #openstack-infra | 14:59 | |
| fungi | clarkb: jeblair: mordred: if you're okay with 145876 and 145839 then i'll go ahead and do the paste db migration maintenance this afternoon or tomorrow | 15:00 |
| *** annegentle has quit IRC | 15:01 | |
| *** achanda has joined #openstack-infra | 15:02 | |
| *** annegentle has joined #openstack-infra | 15:02 | |
| *** packet has joined #openstack-infra | 15:03 | |
| *** pradk has joined #openstack-infra | 15:07 | |
| mordred | fungi: in 145876, you refer to server package but the patch is removing python-mysqldb which is the python module | 15:08 |
| fungi | mordred: oh, you are correct... thanks | 15:09 |
| fungi | mordred: lemme see if i can confirm what package puppet was complaining about | 15:09 |
| *** banix has joined #openstack-infra | 15:09 | |
| *** chuckC_ has quit IRC | 15:10 | |
| mordred | well, I think it's gonna be a conflict with include mysql::pythin | 15:10 |
| mordred | fungi: so I'm likely just quibbling over the commit message - but I wanted to make sure | 15:10 |
| mordred | the other one looks great | 15:10 |
| fungi | yeah, i think the patch is correct and i need to fix the commit message, but double-checking it all now that i clearly was coding under the influence of cold meds when i wrote that :/ | 15:11 |
| openstackgerrit | Merged openstack-infra/project-config: Set up venv for zuul-swift-logs https://review.openstack.org/142327 | 15:11 |
| mordred | ++ | 15:11 |
| fungi | i think include mysql::python is cargo culted actually and i can drop it and abandon 145876 | 15:12 |
| *** TheJulia has quit IRC | 15:13 | |
| mordred | oh cool! | 15:13 |
| *** reed has joined #openstack-infra | 15:14 | |
| *** reed has quit IRC | 15:14 | |
| *** reed has joined #openstack-infra | 15:14 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add a local MySQL database to paste.o.o https://review.openstack.org/145839 | 15:16 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: MySQL database cleanup for paste.o.o https://review.openstack.org/145840 | 15:16 |
| fungi | added one other minor cleanup while i was at it | 15:16 |
| sdague | salv-orlando: https://review.openstack.org/#/c/145430/3 is that something the neutron team is on board with? I don't recognize the commit author, and it's not got a very clear message | 15:18 |
| *** banix has quit IRC | 15:19 | |
| openstackgerrit | Merged openstack-infra/project-config: Publish yaml2ical to PyPI https://review.openstack.org/145243 | 15:19 |
| *** esker has joined #openstack-infra | 15:20 | |
| *** bhunter71 has joined #openstack-infra | 15:22 | |
| *** mrunge has quit IRC | 15:22 | |
| openstackgerrit | Merged openstack-infra/project-config: Use large-ops-testing to test reducing the number of services https://review.openstack.org/145367 | 15:23 |
| *** tonytan4ever has joined #openstack-infra | 15:24 | |
| openstackgerrit | Roman Prykhodchenko proposed openstack-infra/project-config: Add Fuel Client to Stackforge https://review.openstack.org/145843 | 15:25 |
| *** shayneburgess has joined #openstack-infra | 15:25 | |
| *** bswartz has quit IRC | 15:28 | |
| mordred | fungi: the gozer folks were doing a log review thing yesterday that's part of internal HP needs - as part of it, they ran a report on the gerrit db that I found interesting, so I adapted it a little and put it into the file override_report.sql in root's homedir on review.o.o | 15:31 |
| mordred | fungi: basically, it's a report of all of the times someone force-merged something bypassing the gate | 15:32 |
| *** dmsimard_away is now known as dmsimard | 15:32 | |
| mordred | it's kinda a fun output to look at | 15:32 |
| fungi | mordred: oh! neat idea | 15:32 |
| salv-orlando | sdague: first time I see that patch. I don't think this is a move agreed with the rest of them but non necessarily a bad one. Can infra-team wait for input from me, mestery, markmcclain, or armax before merging? | 15:33 |
| mordred | yah - I _almost_ want to make it slightly better and publish it as an HTML report anyone can look at | 15:33 |
| *** bswartz has joined #openstack-infra | 15:33 | |
| armax | salv-orlando: what patch? | 15:33 |
| mestery | armax: https://review.openstack.org/#/c/145430/3 | 15:33 |
| mordred | (it would be nicer if the corresponding comment on the force-merge vote was included too) | 15:33 |
| *** ihrachyshka has quit IRC | 15:34 | |
| *** andreaf has quit IRC | 15:34 | |
| fungi | agreed | 15:35 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for boto 2.35 auth failure bug 1408987 https://review.openstack.org/146118 | 15:35 |
| uvirtbot | Launchpad bug 1408987 in nova "tempest failing with boto==2.35.0" [Critical,Confirmed] https://launchpad.net/bugs/1408987 | 15:35 |
| mordred | fungi: but I don't _actually_ want to spend the effort to do that :) | 15:35 |
| *** vhoward has joined #openstack-infra | 15:36 | |
| fungi | and yeah, a quick review of the results looks reasonably like what i would expect | 15:37 |
| openstackgerrit | Randy Bertram proposed openstack/requirements: Add term.js to global requirements https://review.openstack.org/145825 | 15:37 |
| fungi | roughly matches the people, frequency and projects i think it should | 15:37 |
| *** wenlock has joined #openstack-infra | 15:37 | |
| *** arielb has joined #openstack-infra | 15:38 | |
| *** arielb has quit IRC | 15:38 | |
| *** arielb has joined #openstack-infra | 15:38 | |
| fungi | for example, sahara force-merges around the time it was renamed from savanna, et cetera | 15:38 |
| *** teran has joined #openstack-infra | 15:38 | |
| fungi | requirements and devstack force-merges when setuptools 8 had us horribly broken | 15:38 |
| *** achanda_ has joined #openstack-infra | 15:41 | |
| *** achanda has quit IRC | 15:41 | |
| *** cpallares has joined #openstack-infra | 15:42 | |
| *** otter768 has joined #openstack-infra | 15:43 | |
| *** teran has quit IRC | 15:43 | |
| fungi | mordred: if you're in the mood to look at more puppet, 144367 is needed so that i can make another try on deploying storyboard-dev | 15:44 |
| mordred | looking | 15:44 |
| fungi | once that's in place i'll take another stab at bringing it up with 140466 | 15:45 |
| *** doug-fish has joined #openstack-infra | 15:46 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for boto 2.35 auth failure bug 1408987 https://review.openstack.org/146118 | 15:46 |
| uvirtbot | Launchpad bug 1408987 in nova "tempest failing with boto==2.35.0" [Critical,Confirmed] https://launchpad.net/bugs/1408987 | 15:46 |
| mordred | fungi: I do agree with tim's comment about the comment - but not so much that I think we shoudl block that patch | 15:46 |
| fungi | mordred: yeah, i figured i'd tweak the commen to be more accurate if the change ended up needing another patchset | 15:47 |
| *** marun has quit IRC | 15:47 | |
| *** otter768 has quit IRC | 15:47 | |
| *** pblaho_ is now known as pblaho | 15:48 | |
| mordred | +3 | 15:48 |
| jeblair | fungi: i approved the add-local-db change | 15:49 |
| openstackgerrit | Merged openstack-infra/system-config: Add a local MySQL database to paste.o.o https://review.openstack.org/145839 | 15:49 |
| jeblair | openstackgerrit makes jeblair redundant | 15:50 |
| fungi | jeblair: i saw--thanks. i'll get going on the migration prep once that's showing up on the server | 15:50 |
| *** afazekas has quit IRC | 15:50 | |
| fungi | heh, and that ;) | 15:50 |
| openstackgerrit | Merged openstack-infra/project-config: Prepare project-config for puppet module split https://review.openstack.org/140523 | 15:50 |
| fungi | have you been replaced by a small shell^H^H^H^H^Hpython script? | 15:50 |
| jeblair | and i wrote it :( | 15:50 |
| fungi | i think i saw that movie | 15:51 |
| fungi | come with me if you want to live | 15:51 |
| *** sputnik13 has joined #openstack-infra | 15:51 | |
| *** nelsnelson has joined #openstack-infra | 15:53 | |
| mordred | jeblair: oh! you're here today | 15:53 |
| mordred | jeblair: got a sec to talk about the exception hiding stuff in shade? the other comments are easy to either address or respond to in the review | 15:54 |
| *** jokke_ has joined #openstack-infra | 15:54 | |
| *** bnemec is now known as beekneemech | 15:56 | |
| jeblair | mordred: sure. i'm sorta here. packing and flying later. | 15:56 |
| openstackgerrit | Merged openstack-infra/puppet-storyboard: Allow overriding the cert/key file paths https://review.openstack.org/144367 | 15:57 |
| *** zz_dimtruck is now known as dimtruck | 15:57 | |
| *** achanda_ has quit IRC | 15:58 | |
| mordred | jeblair: k - so - in general there's two ideas driving that pattern across the library | 15:58 |
| mordred | jeblair: one is that there are more than one place where shade may be using one of many different backend clients based on what it's detected about your cloud - so the amount of information you'd need to catch the right base exception is a bit extreme (could be glanceclient, could be novaclient, etc) | 15:59 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix puppet gate check jobs naming https://review.openstack.org/139539 | 16:00 |
| mordred | jeblair: the other is the eventually I want shade to use openstacksdk instead of python-*client - but for people to not have to rewrite consuming code | 16:00 |
| mordred | _however_ I hate exception hiding :) | 16:00 |
| mordred | so I've been trying to debug log whenever we do that - and maybe even should just attach the original exception object to the raised exception? | 16:01 |
| *** dhritishikhar has joined #openstack-infra | 16:01 | |
| mordred | jeblair: so do you think the combo of exception logging when they're masked and attaching the original exception object would be good enough to handle your concern? | 16:02 |
| jeblair | mordred: yeah, if you do it in such a way that when you repr() the ecxception, the original gets printed too, that could aid in debugging... but i think the issue i raised in line 687 of 145603 is still an issue | 16:02 |
| mordred | looking at that one | 16:02 |
| *** annegentle has quit IRC | 16:03 | |
| mordred | jeblair: nod - I'll take a stab at a different approach to that issue | 16:03 |
| *** dhritishikhar has quit IRC | 16:03 | |
| jeblair | mordred: ok. i think to generalize, it's two sides of the same coin -- you have to be careful not to mask debug info from users, and also when exceptions are used for fine-grained error reporting, not mask that from developers | 16:04 |
| mordred | yup. totally agree | 16:05 |
| mordred | jeblair: also - I tested minimal installs of ubuntu, fedora and centos and all had md5sum and figured it would be more efficient than hashlib - but I suppose if the task at hand is "upload this image to the cloud" perhaps the extra 20 seconds of md5 processing isn't particularly concerning? | 16:05 |
| jeblair | mordred: also, i do like your goals. i think it's probably worth doing. it just might (<sigh>) be a bit of work | 16:05 |
| *** openstackgerrit has quit IRC | 16:05 | |
| *** openstackgerrit has joined #openstack-infra | 16:06 | |
| mordred | yah. golly, if only there were a consistent way to deal with this already ... | 16:06 |
| *** bcourt has quit IRC | 16:07 | |
| mordred | jeblair: re: etag in swift - I had originally tested some code that added the md5 as a property to the swift object - I could just go back to doing that instead | 16:07 |
| *** dhritishikhar has joined #openstack-infra | 16:07 | |
| jeblair | mordred: the oracle of stackoverflow suggested it would take like 2 seconds | 16:07 |
| mordred | and that means I could use not-md5 | 16:07 |
| *** TravT has left #openstack-infra | 16:07 | |
| *** bcourt has joined #openstack-infra | 16:07 | |
| *** che-arne has joined #openstack-infra | 16:07 | |
| mordred | jeblair: wel, md5sum itself on a centos image created by dib takes more than 2 | 16:07 |
| fungi | yeah, read speed on the file will be the killer | 16:08 |
| mordred | it takes 29 seconds to calculate the md5 of centos.vhd with md5sum | 16:08 |
| *** yaguang has quit IRC | 16:09 | |
| fungi | since you have to read the entirety of it to generate a checksum | 16:09 |
| jeblair | mordred: i think we ought to try not-md5. it's easy to say "oh this isn't crypto". but who knows what might be built on this and where some attacker might change an image or something with an md5 collision and then this would not notice | 16:09 |
| mordred | ++ | 16:09 |
| mordred | jeblair: which one do you think we should use? | 16:09 |
| jeblair | that would be embarassing | 16:09 |
| jeblair | i think sha256 is the cool kid now, right? | 16:09 |
| fungi | honestly, i expect just about any checksumming algorithm to be ~= timewise, because you're going to be i/o bound on the file | 16:10 |
| *** otter768 has joined #openstack-infra | 16:10 | |
| fungi | so, yeah, sha256 should be fine | 16:10 |
| fungi | (technically that's sha2-256) | 16:10 |
| jeblair | that's the one | 16:10 |
| clarkb | you need md5 anyways because thats what glance checks iirc | 16:10 |
| mordred | clarkb: this is all shade | 16:11 |
| jeblair | which is sad | 16:11 |
| clarkb | talking to glance right? | 16:11 |
| jeblair | i think clarkb was referring to the upload integrity check? | 16:11 |
| mordred | this is how shade knows whether to re-upload | 16:11 |
| clarkb | yes | 16:11 |
| fungi | glance will probably add a more modern checksum algorithm eventually--there is a bug open for that | 16:11 |
| jeblair | it will probably not cost us anything more to generate 2 hashes at the same time | 16:12 |
| jeblair | since we are more likely io-bound than cpu bound on that | 16:12 |
| *** Ala has quit IRC | 16:13 | |
| mordred | ++ | 16:13 |
| mordred | if I'm doing it with hashlib, I'll absolutely do that | 16:13 |
| mordred | in fact, then I can check to see if the swift etag matches _and_ if the user-provided sha256 matches - because also it'll be the same cost to do so | 16:13 |
| fungi | hrm, there's a good chance it's going to want to read the file twice to be able to do that | 16:13 |
| mordred | fungi: the hashlib approach is to read the file in 8k chunks | 16:14 |
| fungi | (to be able to generate two different checksums i mean) | 16:14 |
| mordred | fungi: and then update the hash | 16:14 |
| *** patrickeast has joined #openstack-infra | 16:14 | |
| mordred | so I should be able to update two different hashes in that loop | 16:14 |
| *** mtanino has joined #openstack-infra | 16:14 | |
| fungi | mordred: it can reuse the same read to update two hash objects? | 16:14 |
| fungi | yeah, that should be fine then | 16:14 |
| mordred | should be able to | 16:14 |
| mordred | I'll test real quick | 16:14 |
| jeblair | mordred: on my machine, md5 of a 950MB trusty iso takes 1.5s with the command line, and 1.5s with python @ 4096b chunks | 16:15 |
| jeblair | (with a warm io cache) | 16:16 |
| *** coolsvap is now known as coolsvap|afk | 16:16 | |
| jeblair | er wow | 16:17 |
| jeblair | sha256 command line takes 4.1 seconds, python takes 3.7 | 16:17 |
| openstackgerrit | Marton Kiss proposed openstack-infra/system-config: Backup groups.o.o with bup https://review.openstack.org/146126 | 16:17 |
| jeblair | python is faster? ^ | 16:18 |
| mordred | wow | 16:18 |
| jeblair | that's pretty much additive. doing both in python is 5.6s | 16:18 |
| *** rfolco has quit IRC | 16:19 | |
| jeblair | 8192b chunks is slightly faster at 4.9s | 16:19 |
| mordred | jeblair: I was doing my md5 calc of the 1.6G centos image on a cloud server :) | 16:19 |
| *** ssam2 has quit IRC | 16:20 | |
| jeblair | mordred: yeah, not only might the cpu be slower, but you might not be able to fit it all in ram to keep the io cache warm | 16:20 |
| *** annegentle has joined #openstack-infra | 16:20 | |
| *** ssam2 has joined #openstack-infra | 16:20 | |
| mordred | jeblair: a single loop in python using 1M chunks can calculate both checksums in 1.5 minutes | 16:20 |
| jeblair | mordred: but that's an issue no matter how it's generated, so as long as you're consistent, comparing the two approaches should be fine | 16:20 |
| *** xgerman has joined #openstack-infra | 16:20 | |
| jeblair | mordred: much more than 4096 isn't going to get you much and is just going to waste ram | 16:21 |
| *** e0ne has joined #openstack-infra | 16:21 | |
| jeblair | 8192 seems to be the sweet spot | 16:21 |
| mordred | jeblair: yah - I was doing 1M because that was in the copy-pasta | 16:21 |
| mordred | jeblair: re-doing with 8192 | 16:21 |
| openstackgerrit | Roman Prykhodchenko proposed openstack-infra/project-config: Add Fuel Client to Stackforge https://review.openstack.org/145843 | 16:22 |
| jeblair | http://paste.openstack.org/show/156106/ | 16:22 |
| *** marcusvrn has quit IRC | 16:25 | |
| *** ildikov has quit IRC | 16:27 | |
| *** ddieterly has quit IRC | 16:27 | |
| mordred | jeblair: same time/cost for 8k | 16:28 |
| jeblair | yeah, so i'd go with that so you're not eating ram | 16:29 |
| *** bswartz has quit IRC | 16:30 | |
| *** marcusvrn has joined #openstack-infra | 16:30 | |
| *** Hal_ has quit IRC | 16:30 | |
| *** otter768 has quit IRC | 16:30 | |
| cody-somerville | Hey folks. Were the sqlalchemy dependency updated to handle new pip on the stable branches as well or was a different strategy taken there? | 16:30 |
| *** bswartz has joined #openstack-infra | 16:31 | |
| fungi | cody-somerville: we updated the sqlalchemy dependency strings in global-requirements.txt on all branches | 16:31 |
| fungi | cody-somerville: though the exact strings used differed between stable and master | 16:31 |
| *** bswartz has quit IRC | 16:32 | |
| *** bswartz has joined #openstack-infra | 16:33 | |
| *** annegentle has quit IRC | 16:33 | |
| *** ddieterly has joined #openstack-infra | 16:33 | |
| cody-somerville | fungi: And to be clear, no attempts to pin pip to older version was used or done anywhere, right? | 16:33 |
| fungi | cody-somerville: it was temporarily but we've been able to revert that everywhere now afaik | 16:34 |
| *** bcourt has quit IRC | 16:34 | |
| fungi | er, we didn't pin pip but we did pin setuptools | 16:34 |
| fungi | and have since rolled that back now that things are working | 16:34 |
| fungi | cody-somerville: the sqlalchemy requirements issue was already solved by the time pip updated. we went through all that to accommodate setuptools 8.0 | 16:35 |
| *** andreykurilin has joined #openstack-infra | 16:35 | |
| fungi | which led the pip/virtualenv update a couple weeks | 16:35 |
| *** annegentle has joined #openstack-infra | 16:35 | |
| *** zz_avozza is now known as avozza | 16:36 | |
| *** stevemar has joined #openstack-infra | 16:36 | |
| *** Masahiro has joined #openstack-infra | 16:37 | |
| *** rfolco has joined #openstack-infra | 16:37 | |
| *** dansmith is now known as superdan | 16:37 | |
| sdague | cody-somerville: yeh, it mostly requires synchronizing all the stable trees for projects I think | 16:38 |
| *** teran has joined #openstack-infra | 16:39 | |
| *** pcrews has joined #openstack-infra | 16:40 | |
| *** dimtruck is now known as zz_dimtruck | 16:41 | |
| *** rfolco has quit IRC | 16:41 | |
| *** Masahiro has quit IRC | 16:41 | |
| *** zz_dimtruck is now known as dimtruck | 16:42 | |
| *** dangers_away is now known as dangers | 16:42 | |
| *** hdd has joined #openstack-infra | 16:43 | |
| *** teran has quit IRC | 16:44 | |
| *** marun has joined #openstack-infra | 16:44 | |
| *** dannywilson has joined #openstack-infra | 16:47 | |
| *** bcourt has joined #openstack-infra | 16:48 | |
| *** annegent_ has joined #openstack-infra | 16:48 | |
| *** TheJulia has joined #openstack-infra | 16:49 | |
| *** banix has joined #openstack-infra | 16:49 | |
| *** markus_z has quit IRC | 16:49 | |
| *** dkranz has quit IRC | 16:49 | |
| *** marun has quit IRC | 16:50 | |
| *** amrith has joined #openstack-infra | 16:50 | |
| *** annegentle has quit IRC | 16:50 | |
| amrith | jeblair, ping. | 16:50 |
| *** dannywilson has quit IRC | 16:50 | |
| *** dannywilson has joined #openstack-infra | 16:51 | |
| *** vigneshvar has joined #openstack-infra | 16:51 | |
| amrith | mordred, ping. | 16:52 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Set the Gerrit change screen default to OLD_UI. https://review.openstack.org/145909 | 16:53 |
| *** mfink has quit IRC | 16:53 | |
| mordred | amrith: sup? | 16:53 |
| *** rwsu has joined #openstack-infra | 16:53 | |
| amrith | mordred, thx. looking into an issue in trove. | 16:54 |
| amrith | building guest image, I get "ImportError: No module named unittest2" | 16:54 |
| amrith | I saw some talk about this 12/16 on openstack-infra | 16:54 |
| amrith | betwee you and jeblair | 16:54 |
| *** Guest47811 has joined #openstack-infra | 16:54 | |
| amrith | was wondering what y'all may have found | 16:54 |
| amrith | and how you got around it | 16:54 |
| amrith | I'm getting it on an install of oslo.concurrency | 16:54 |
| amrith | which requires | 16:54 |
| amrith | testtools -> unittest2 | 16:54 |
| jeblair | amrith: i think the solution is to install a new version of virtualenv, which will have a new setuptools and then it should work | 16:55 |
| amrith | http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2014-12-16.log approx 2014-12-16T22:34:51 | 16:55 |
| *** Guest47811 is now known as rackertom | 16:55 | |
| mordred | amrith: yes, what jeblair said | 16:55 |
| *** dannywilson has quit IRC | 16:55 | |
| amrith | we currently have pip install -U 'setuptools<8.0' | 16:56 |
| amrith | pip install virtualenv | 16:56 |
| amrith | what's the preference on setuptools version? | 16:56 |
| *** avozza is now known as zz_avozza | 16:56 | |
| *** annegentle has joined #openstack-infra | 16:57 | |
| *** yamahata has joined #openstack-infra | 16:57 | |
| mordred | amrith: virtualenv bundles setuptools - do you need a new version of virtualenv | 16:57 |
| mordred | s/do/so/ | 16:57 |
| mordred | so do a pip install -U virtualenv | 16:58 |
| mordred | you also should not be pinning setuptools to less than 8.0 | 16:58 |
| mordred | amrith: where is that? | 16:58 |
| mordred | amrith: _definitely_ remove the <8.0 from your pip install -U setuptools | 16:58 |
| *** sdake has quit IRC | 16:58 | |
| amrith | am looking for that | 16:58 |
| amrith | I see it in my log | 16:59 |
| amrith | trying to see where it comes from | 16:59 |
| amrith | it is in generating a guest image for trove | 16:59 |
| *** fthigv has joined #openstack-infra | 16:59 | |
| *** sdake has joined #openstack-infra | 16:59 | |
| mordred | amrith: is this driven by devstack? | 16:59 |
| *** sdake has quit IRC | 16:59 | |
| *** sdake has joined #openstack-infra | 16:59 | |
| amrith | yes, this part would be | 17:00 |
| mordred | because there _was_ a period where we were pinning setuptols in devstack | 17:00 |
| amrith | looking, one second | 17:00 |
| mordred | so you could have a _very_ out of date devstack :) | 17:00 |
| zaro | morning | 17:00 |
| mordred | mornign zaro | 17:00 |
| *** annegent_ has quit IRC | 17:00 | |
| *** marcusvrn has quit IRC | 17:01 | |
| *** marcusvrn1 has joined #openstack-infra | 17:01 | |
| clarkb | its also redstack and dib | 17:01 |
| amrith | my devstack isn't all that old | 17:01 |
| yjiang5 | fungi: Hi, I have a question to the gerrit stream event. I have a local mirror, which will pull the repo from http://git.openstack.org/ immediately after get the change-merged event. However, most of the time, it will get result of "Already up-to-date". So want to know if the event is sent out before or after the corresponding change are pushed to the openstack/github mirror? | 17:01 |
| amrith | and I'm building image through redstack | 17:01 |
| amrith | which invokes devstack for this part | 17:01 |
| sdague | redstack ? | 17:02 |
| *** dkranz has joined #openstack-infra | 17:03 | |
| amrith | redstack | 17:03 |
| amrith | trove-integration/scripts/redstack | 17:03 |
| *** jlibosva has quit IRC | 17:03 | |
| clarkb | sdague fork thing of devstack from the reddwarf days | 17:03 |
| sdague | ah, gotcha | 17:03 |
| *** TravT has joined #openstack-infra | 17:03 | |
| amrith | convenience for development in my sandbox | 17:03 |
| *** EmilienM is now known as EmilienM|afk | 17:04 | |
| amrith | quick and dirty way for me to build a dev environment | 17:04 |
| *** kgiusti has quit IRC | 17:04 | |
| clarkb | yjiang5 I think they happen concurrently | 17:04 |
| clarkb | yjiang5 unfortunately we need a newer gerrit to get replication events but zaro is testing that upgrade now | 17:05 |
| *** jgrimm is now known as zz_jgrimm | 17:05 | |
| *** kgiusti has joined #openstack-infra | 17:06 | |
| *** alugovoi has joined #openstack-infra | 17:06 | |
| yjiang5 | clarkb: ok, so it means highly possible my git will be one changeset delay, till the replication event is done, right? | 17:06 |
| *** sabeen1 has joined #openstack-infra | 17:06 | |
| clarkb | yjiang5 yes | 17:06 |
| *** ivar-lazzaro has joined #openstack-infra | 17:07 | |
| yjiang5 | clarkb: thanks for your help a lot. Currently I will workaround this issue by sleep 10 seconds before pull the git. Hope 10 seconds will be enough. | 17:07 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for ceilometer oslo.utils bug 1408737 https://review.openstack.org/146136 | 17:07 |
| uvirtbot | Launchpad bug 1408737 in ceilometer "broken unit tests after oslo.utils update" [High,In progress] https://launchpad.net/bugs/1408737 | 17:07 |
| amrith | mordred, jeblair ... brand new devstack and still the same. https://gist.github.com/amrith/7543ce6320f10ec8e296, I'll try and figure out where this version pinning is coming from. thx | 17:09 |
| *** Ryan_Lane has joined #openstack-infra | 17:11 | |
| *** dimtruck is now known as zz_dimtruck | 17:11 | |
| amrith | mordred, jeblair looks like it comes from tripleo-image-elements/elements/pip-and-virtualenv/install.d/get-pip-py-source-install/01-install-pip | 17:13 |
| *** shashankhegde has joined #openstack-infra | 17:14 | |
| *** bswartz1 has joined #openstack-infra | 17:14 | |
| *** bswartz has quit IRC | 17:16 | |
| *** gmurphy has left #openstack-infra | 17:16 | |
| *** beekneemech has quit IRC | 17:17 | |
| *** bnemec has joined #openstack-infra | 17:18 | |
| *** marcusvrn1 has quit IRC | 17:18 | |
| *** zz_dimtruck is now known as dimtruck | 17:19 | |
| *** cpallares has quit IRC | 17:19 | |
| *** garyh has joined #openstack-infra | 17:19 | |
| zaro | clarkb: i'm wondering if the replication issue we've been having is from setting replicationDelay to too low of value? | 17:20 |
| mordred | amrith: ah - then that's almost certainly a bug in t-i-e | 17:20 |
| *** MaxV has quit IRC | 17:20 | |
| mordred | SpamapS, greghaynes: ^^ | 17:20 |
| *** otter768 has joined #openstack-infra | 17:20 | |
| zaro | clarkb: read lockErrorMaxRetries https://gerrit.googlesource.com/plugins/replication/+/stable-2.9/src/main/resources/Documentation/config.md | 17:20 |
| amrith | mordred, what version of setuptools are you expecting? | 17:20 |
| amrith | seems like it is endinng up with 7.0 | 17:20 |
| amrith | which is now up to 11.3 ish ;) | 17:21 |
| zaro | clarkb: the default is 15 seconds. | 17:21 |
| mordred | amrith: we expect latest setuptools | 17:22 |
| *** dims__ has quit IRC | 17:22 | |
| amrith | ok, thanks! | 17:22 |
| greghaynes | aroo? | 17:22 |
| *** achanda has joined #openstack-infra | 17:22 | |
| mordred | greghaynes: you're pinning setuptools it seems in t-i-e? | 17:22 |
| *** dims__ has joined #openstack-infra | 17:22 | |
| greghaynes | https://review.openstack.org/#/c/145027/ | 17:22 |
| mordred | amrith: ^^ there's your fix | 17:23 |
| amrith | mordred, thanks! | 17:23 |
| clarkb | zaro we can probably bump that value. should dig through git logs to see if they tell us why its set low though | 17:23 |
| *** marcusvrn has joined #openstack-infra | 17:23 | |
| *** ldnunes has quit IRC | 17:24 | |
| *** ZZelle is now known as ZZelle|away | 17:24 | |
| *** ldnunes has joined #openstack-infra | 17:24 | |
| zaro | clarkb: i'll take a look | 17:25 |
| *** melwitt has joined #openstack-infra | 17:26 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for ceilometer oslo.utils bug 1408737 https://review.openstack.org/146136 | 17:26 |
| uvirtbot | Launchpad bug 1408737 in ceilometer "broken unit tests after oslo.utils update" [High,In progress] https://launchpad.net/bugs/1408737 | 17:26 |
| *** dims__ has quit IRC | 17:27 | |
| *** dizquierdo has quit IRC | 17:28 | |
| *** jedimike has quit IRC | 17:28 | |
| *** arxcruz has quit IRC | 17:28 | |
| jeblair | zaro, clarkb: i think we should have the delay to be low (eg, current value of 1 second), and retries to be higher than the default of 0, (eg, suggestion of 3) | 17:28 |
| jeblair | zaro, clarkb: is that what you are discussing? or are you proposing increasing the delay? | 17:29 |
| *** bhuvan_ has joined #openstack-infra | 17:29 | |
| *** bhuvan_ has joined #openstack-infra | 17:29 | |
| clarkb | jeblair possibly both? not sure how long these locks will be contended for | 17:30 |
| *** gyee has joined #openstack-infra | 17:30 | |
| *** e0ne is now known as e0ne_ | 17:30 | |
| *** jistr_ has joined #openstack-infra | 17:31 | |
| *** dtantsur_ has joined #openstack-infra | 17:31 | |
| jeblair | i want our git.o.o mirrors to be useful and fast, so i want to keep the delay low. i don't mind the extra traffic. | 17:31 |
| *** pblaho_ has joined #openstack-infra | 17:31 | |
| *** jcoufal has joined #openstack-infra | 17:32 | |
| *** ryanpetrello has quit IRC | 17:33 | |
| *** signed8bit has quit IRC | 17:33 | |
| *** jcoufal_ has quit IRC | 17:33 | |
| *** palar has joined #openstack-infra | 17:34 | |
| *** jcoufal_ has joined #openstack-infra | 17:34 | |
| *** dtantsur has quit IRC | 17:34 | |
| *** pblaho has quit IRC | 17:34 | |
| *** jaypipes is now known as leakypipes | 17:34 | |
| *** jistr has quit IRC | 17:35 | |
| *** pblaho__ has joined #openstack-infra | 17:37 | |
| *** pblaho__ is now known as pblah | 17:37 | |
| *** pblaho_ has quit IRC | 17:37 | |
| *** pblah is now known as pblaho | 17:37 | |
| *** jcoufal has quit IRC | 17:37 | |
| *** pblaho has quit IRC | 17:37 | |
| *** dtantsur_ has quit IRC | 17:37 | |
| *** jistr_ has quit IRC | 17:37 | |
| *** pblaho has joined #openstack-infra | 17:37 | |
| *** Ryan_Lane has quit IRC | 17:38 | |
| *** hdd has quit IRC | 17:39 | |
| *** otter768 has quit IRC | 17:39 | |
| zaro | isn't setting replicationDelay low and retries high opposite to what the documentation recommends? | 17:39 |
| *** aysyd has quit IRC | 17:40 | |
| zaro | jeblair: ohh actually i got mixed up. i think your suggestion is reasonable. | 17:41 |
| *** Longgeek has quit IRC | 17:42 | |
| *** jcoufal_ has quit IRC | 17:42 | |
| *** yamahata has quit IRC | 17:42 | |
| *** dtantsur has joined #openstack-infra | 17:42 | |
| *** hdd has joined #openstack-infra | 17:42 | |
| *** aysyd has joined #openstack-infra | 17:42 | |
| *** Longgeek has joined #openstack-infra | 17:44 | |
| *** andreykurilin has quit IRC | 17:44 | |
| *** chianingwang has joined #openstack-infra | 17:44 | |
| *** andreykurilin has joined #openstack-infra | 17:45 | |
| *** otter768 has joined #openstack-infra | 17:45 | |
| *** tkelsey has quit IRC | 17:45 | |
| *** ryanpetrello has joined #openstack-infra | 17:45 | |
| *** sputnik13 has quit IRC | 17:47 | |
| *** bcourt has quit IRC | 17:48 | |
| *** hashar has quit IRC | 17:48 | |
| *** dims__ has joined #openstack-infra | 17:49 | |
| *** e0ne_ is now known as e0ne | 17:50 | |
| *** dimtruck is now known as zz_dimtruck | 17:51 | |
| amrith | jeblair, mordred ... I cherry-picked that change and re-ran redstack, failed in the same way. I now have setuptools (latest). will investigate | 17:51 |
| *** derekh has quit IRC | 17:51 | |
| *** dannywilson has joined #openstack-infra | 17:51 | |
| zaro | clarkb, jeblair : opps, i referenced the wrong doc. ver 2.8.4 doesn't have the e | 17:51 |
| *** dannywilson has quit IRC | 17:51 | |
| zaro | lockErrorMaxRetries option. https://gerrit.googlesource.com/plugins/replication/+/v2.8.4/src/main/resources/Documentation/config.md | 17:52 |
| *** zz_dimtruck is now known as dimtruck | 17:52 | |
| *** dannywilson has joined #openstack-infra | 17:52 | |
| *** _shaps_ has quit IRC | 17:53 | |
| *** Longgeek has quit IRC | 17:53 | |
| *** dims__ is now known as dimsum__ | 17:53 | |
| *** amitgandhinz has quit IRC | 17:53 | |
| *** sputnik13 has joined #openstack-infra | 17:54 | |
| *** Longgeek has joined #openstack-infra | 17:54 | |
| *** otter768 has quit IRC | 17:55 | |
| *** shashankhegde has quit IRC | 17:57 | |
| mordred | amrith: did you also update virtualenv to latest? | 17:57 |
| mordred | amrith: or are you using virtualenv at all? | 17:57 |
| reed | are there already plans to remove this page and use infra manual instead? https://wiki.openstack.org/wiki/Gerrit_Workflow | 17:57 |
| amrith | mordred, I'm using virtualenv but I'm using redstack to build a new image. so in creating a new image it gets the latest virtualenv also | 17:57 |
| amrith | will get you versions in a second | 17:58 |
| mordred | amrith: ok. can you verify that it's latest virtualenv? cool | 17:58 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: update replication setting for review-dev.o.o match review.o.o https://review.openstack.org/146149 | 17:59 |
| *** Longgeek has quit IRC | 17:59 | |
| *** hdd has quit IRC | 17:59 | |
| *** bcourt has joined #openstack-infra | 18:00 | |
| krotscheck | jeblair: Comments on yolanda’s core promotion have closed, and we’ve got no concerns. What’s the next step? | 18:00 |
| *** Longgeek has joined #openstack-infra | 18:00 | |
| fungi | throw a party? | 18:01 |
| yolanda | fungi, sure! :) | 18:01 |
| krotscheck | Party at fungi’s place! | 18:02 |
| *** Longgeek has quit IRC | 18:02 | |
| fungi | i think someone just adds her to the core group for that project and jeblair follows up to the nomination thread with a confirmation (or delegates it) | 18:02 |
| krotscheck | That’s a projects-config thing? | 18:03 |
| fungi | krotscheck: i recommend coming in by seaplane | 18:03 |
| *** e0ne has quit IRC | 18:03 | |
| *** aysyd has quit IRC | 18:03 | |
| *** AJaeger has joined #openstack-infra | 18:03 | |
| *** Longgeek has joined #openstack-infra | 18:04 | |
| *** Longgeek has quit IRC | 18:04 | |
| fungi | krotscheck: it happens by editing https://review.openstack.org/#/admin/groups/187,members though right now it looks like that's not a self-managed group (owned by administrators). not sure if that was intentional | 18:04 |
| *** annegentle has quit IRC | 18:05 | |
| krotscheck | fungi: hrm. Yeah, I can’t edit that. | 18:05 |
| *** mfink has joined #openstack-infra | 18:05 | |
| pleia2 | haha, woke up congested, my husband goes "you look and sound horrible" - such a charmer :) and I might have a cold | 18:05 |
| *** aysyd has joined #openstack-infra | 18:05 | |
| *** Longgeek has joined #openstack-infra | 18:05 | |
| *** tonytan4ever has quit IRC | 18:07 | |
| *** tonytan4ever has joined #openstack-infra | 18:08 | |
| AJaeger | Hi, I just noticed that logfiles are not complete. Is that a known issue with our swift logs? | 18:10 |
| *** dtantsur is now known as dtantsur|afk | 18:10 | |
| fungi | AJaeger: yeah, it seems there may be a race with log uploading where jenkins hasn't finished recording the console output. there was some discussion yesterday on ways to mitigate | 18:11 |
| AJaeger | fungi: thanks | 18:11 |
| fungi | best contender was to insert a canary at the end of the job output and then keep polling (with some reasonable timeout) until that pattern is found | 18:12 |
| AJaeger | that sounds ugly ;( | 18:12 |
| *** tonytan4ever has quit IRC | 18:12 | |
| fungi | welcome to jenkins integration! | 18:12 |
| *** tqtran has joined #openstack-infra | 18:12 | |
| *** achanda has quit IRC | 18:13 | |
| AJaeger | ;) | 18:13 |
| *** chianingwang has quit IRC | 18:13 | |
| *** ivar-lazzaro has quit IRC | 18:14 | |
| tqtran | Hello, can i bug someone to take a look at https://review.openstack.org/#/c/145572/? We're trying to move along some of the 3rd party JS dependencies in horizon. Any help on this is greatly appreciated. | 18:14 |
| dkranz | fungi: Is https://review.openstack.org/#/c/145901/ supposed to fix the gate problem with tempest? Cause it looks like stable branches still fail there. | 18:14 |
| fungi | sdague: ^ ? | 18:15 |
| *** kgiusti has quit IRC | 18:16 | |
| fungi | dkranz: i _think_ it needs a similar patch in nova stable branches | 18:16 |
| *** jpich has quit IRC | 18:16 | |
| *** kgiusti has joined #openstack-infra | 18:16 | |
| fungi | not actually following that issue very closely today | 18:16 |
| fungi | so i coule be way off base | 18:16 |
| fungi | dkranz: or maybe just requirements stable branches will be enough | 18:17 |
| clarkb | fungi that sounds right. need to roll up a lot of these fixes with grenade and branchless tempest | 18:18 |
| fungi | aha, the juno patch hasn't landed for that yet... https://review.openstack.org/#/q/I2f9c26168c1296cd8dc19dc59ae5d2831f90bd0d,n,z | 18:19 |
| dkranz | clarkb: that looks like https://review.openstack.org/#/c/146064/ but check-requirements-integration-dsvm is failing | 18:19 |
| dkranz | clarkb: maybe this from the log: 'oslo.context' is not in global-requirements.txt | 18:20 |
| fungi | http://logs.openstack.org/64/146064/3/check/check-requirements-integration-dsvm/7c61b03/console.html#_2015-01-09_15_46_16_859 "'oslo.context' is not in global-requirements.txt" | 18:20 |
| fungi | yep | 18:20 |
| fungi | looks like oslo.log is in juno requirements and has added a requirement for oslo.context which is not in juno global requirements | 18:21 |
| clarkb | its all documented in the fix for the fix for the fix :) | 18:21 |
| fungi | hah | 18:21 |
| *** marcusvrn has quit IRC | 18:22 | |
| *** jp_at_hp has quit IRC | 18:23 | |
| dkranz | fungi, clarkb : So the "cherry pick" was actually no | 18:24 |
| dkranz | not | 18:24 |
| fungi | i take it oslo.log is lacking requirements checks | 18:24 |
| *** ssam2 has quit IRC | 18:24 | |
| openstackgerrit | Merged openstack/requirements: Use oslo.utils without namespace package https://review.openstack.org/145790 | 18:24 |
| fungi | no, wait, that would be a lack of stable backward-compat checks for juno i guess | 18:24 |
| fungi | though i suppose we don't do backward compatibility requirements checks to catch this | 18:25 |
| *** Masahiro has joined #openstack-infra | 18:25 | |
| openstackgerrit | Merged openstack/requirements: provide more meaningful output on update.py https://review.openstack.org/145925 | 18:26 |
| fungi | so, based on a cursory time trial, it looks like i can accomplish the paste db migration in ~2 minutes if all goes smoothly, and can cut back in under a minute if it fails for some reason | 18:28 |
| *** Masahiro has quit IRC | 18:30 | |
| openstackgerrit | Khai Do proposed openstack-infra/project-config: Update zuul layout to match production https://review.openstack.org/145934 | 18:31 |
| fungi | the plan is: 1. disable puppet agent, 2. remove the trove hostname from hiera, 3. stop apache, 4. mysqldump the trove db, 5. read the dump into the local db, 6. reenable puppet agent and apply, 7. start apache, 8. test | 18:31 |
| *** jcoufal has joined #openstack-infra | 18:31 | |
| fungi | assuming the test fails: 9. replace the trove hostname in hiera, 10. update puppet again | 18:32 |
| clarkb | when does point at localhost happen? | 18:32 |
| fungi | teh global site manifest provides localhost as a fallback value if the db hostname key is not present in hiera | 18:32 |
| clarkb | gotcha | 18:33 |
| fungi | http://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp#n250 | 18:33 |
| *** zz_sabari is now known as sabari | 18:33 | |
| fungi | that's what that does, right? or am i misunderstanding how that works? | 18:33 |
| *** ivar-lazzaro has joined #openstack-infra | 18:33 | |
| dkranz | clarkb, fungi: is any one working on unblocking the gate? I would help but not sure what exactly to do. | 18:34 |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/storyboard: Explicit Event API https://review.openstack.org/145874 | 18:34 |
| openstackgerrit | Merged openstack/requirements: update test-requirements to current https://review.openstack.org/145931 | 18:34 |
| sdague | dkranz: it's possible that stable/juno is wedged on 2 things needing to happen at once | 18:34 |
| fungi | dkranz: sdague had a thread about it on the -dev ml, but not sure if he's still actively poking the fixes | 18:34 |
| fungi | oh, there's an sdague | 18:34 |
| dkranz | ouch | 18:34 |
| sdague | yeh, sorry, was just scraping snow off the driveway | 18:35 |
| dkranz | sdague: :) | 18:35 |
| openstackgerrit | Joe Gordon proposed openstack/requirements: Update to latest hacking (0.10.0) https://review.openstack.org/145567 | 18:35 |
| sdague | the master and icehouse fixes look like they are in | 18:35 |
| clarkb | fungi ya that is how the hiera lookup should work | 18:35 |
| sdague | I haven't been following the juno fixes, one of the stable maint folks hopefully will | 18:36 |
| *** jogo is now known as flashgordon | 18:37 | |
| *** amitgandhinz has joined #openstack-infra | 18:37 | |
| fungi | sdague: looks like it's presently deadlocked on the fact that oslo.log is in stable/juno projects.txt and has added a requirement for oslo.context which is not in stable/juno global-requirements.txt | 18:37 |
| fungi | or something along those lines | 18:38 |
| fungi | adam_g: ^ | 18:38 |
| *** tonytan4ever has joined #openstack-infra | 18:38 | |
| fungi | flaper87: ^ | 18:39 |
| *** marcusvrn has joined #openstack-infra | 18:39 | |
| *** Ryan_Lane has joined #openstack-infra | 18:40 | |
| *** tonytan4ever has quit IRC | 18:40 | |
| *** mpaolino has quit IRC | 18:40 | |
| *** tonytan4ever has joined #openstack-infra | 18:41 | |
| amrith | mordred, I'm using virtualenv-12.0.5 (latest). | 18:42 |
| amrith | mordred, ^^ | 18:42 |
| *** dkranz has quit IRC | 18:44 | |
| fungi | so anyway, given that i can migrate paste in such a short window, i'm tempted to just go ahead and do it, not bother scheduling/announcing anything | 18:46 |
| adam_g | sdague, fungi, will take a look shortly | 18:46 |
| fungi | or i can make openstackstatus announce it for me via notice to the irc channels and not bother with e-mail or alert/ok | 18:47 |
| fungi | that seems like a sane compromise | 18:47 |
| clarkb | fungi ya I think considering it fails enough on its own a short window like that is fine | 18:47 |
| *** dustins has quit IRC | 18:51 | |
| *** bnemec is now known as beekneemech | 18:51 | |
| *** harlowja_away is now known as harlowja | 18:52 | |
| *** pblaho has quit IRC | 18:54 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 18:57 | |
| fungi | #status notice paste.openstack.org is going offline for a database migration (duration: ~2 minutes) | 18:57 |
| openstackstatus | fungi: sending notice | 18:57 |
| -openstackstatus- NOTICE: paste.openstack.org is going offline for a database migration (duration: ~2 minutes) | 18:58 | |
| *** dannywilson has quit IRC | 18:59 | |
| openstackstatus | fungi: finished sending notice | 18:59 |
| *** shashankhegde has joined #openstack-infra | 18:59 | |
| fungi | it's migrated and back online now | 19:00 |
| fungi | http://paste.openstack.org/show/156124/ is a paste i saved right before i stopped apache | 19:00 |
| *** annegentle has joined #openstack-infra | 19:00 | |
| *** dannywilson has joined #openstack-infra | 19:01 | |
| mordred | fungi: it seems to be working | 19:01 |
| fungi | yep, and i saved 156125 right after apache came back up. the trove db has 156124 but not 156125 while the local mysql db has both | 19:02 |
| fungi | so looks like it was successful | 19:02 |
| *** alexpilotti has quit IRC | 19:03 | |
| *** yamahata has joined #openstack-infra | 19:05 | |
| *** e0ne has joined #openstack-infra | 19:06 | |
| *** dustins has joined #openstack-infra | 19:07 | |
| *** ivar-lazzaro has quit IRC | 19:07 | |
| *** dalgaaf has quit IRC | 19:07 | |
| *** sabari is now known as zz_sabari | 19:07 | |
| fungi | clarkb: oh, not sure if you saw 145978 last night, my next installment in the "you can't sudo here" saga | 19:08 |
| *** xyang0 has quit IRC | 19:08 | |
| adam_g | fungi, https://review.openstack.org/#/c/146180/ + https://review.openstack.org/#/c/146179/1 not certain whether the latter is needed but seems to make sense | 19:09 |
| *** imcsk8 has quit IRC | 19:09 | |
| *** imcsk8 has joined #openstack-infra | 19:10 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/infra-specs: StoryBoard Streaming API specification https://review.openstack.org/105252 | 19:10 |
| fungi | adam_g: you'll likely need to switch the order on them to get them to pass | 19:10 |
| krotscheck | fungi: Looks like your recheck on https://review.openstack.org/#/c/140466/ passed. | 19:11 |
| *** kgiusti has quit IRC | 19:11 | |
| fungi | adam_g: i think the requirements integration job for juno is broken until 146180 is in place, so 146179 won't succeed | 19:11 |
| fungi | krotscheck: yep, i'm about to switch gears to storyboard-dev testing now that the paste migration is wrapped up | 19:11 |
| adam_g | fungi, ah, right. ill break em up | 19:11 |
| krotscheck | fungi: Thanks :D | 19:11 |
| zaro | can someone help me debug zuul? | 19:11 |
| zaro | it doesn't seem like zuul is sending jobs to gearman | 19:12 |
| * krotscheck is still doing gerrit things. Will hopefully be done by lunch | 19:12 | |
| *** jcoufal has quit IRC | 19:12 | |
| zaro | fungi: help with zuul? | 19:12 |
| fungi | zaro: does the debug log indicate it's initiating jobs? | 19:13 |
| *** jcoufal has joined #openstack-infra | 19:13 | |
| *** signed8bit has joined #openstack-infra | 19:13 | |
| zaro | fungi: i see this in the zuul log, "2015-01-09 19:09:08,492 INFO zuul.Gerrit: Updating information for 5306,6" | 19:13 |
| zaro | that's it. | 19:13 |
| fungi | zaro: oh, actually an easy cause is that you might have forgotten to start the zuul-merger daemon? | 19:13 |
| fungi | zuul won't submit any jobs until the necessary merges for them complete | 19:14 |
| zaro | fungi: nope zuul merger service is started and running. however i don't see anything in the merger logs | 19:14 |
| fungi | zaro: and it's configured to use the built-in gearman service? have you checked the gearman status output? | 19:15 |
| *** fthigv has quit IRC | 19:15 | |
| *** vponomaryov has quit IRC | 19:15 | |
| zaro | i'm not sure how the zuul merger is configured. but i'm looking at the gearman status and i never see the job getting added to the gearman queue. | 19:16 |
| zaro | so maybe zuul merger is where i should look? | 19:16 |
| zaro | this is on review-dev, so maybe merger didn't get setup? | 19:16 |
| fungi | that's possible... check the zuul.conf merger section | 19:17 |
| *** AJaeger has quit IRC | 19:17 | |
| *** arielb has quit IRC | 19:17 | |
| fungi | maybe this is a newish version of zuul with an out of date config | 19:17 |
| zaro | zuul version is 2.0.0.312.g2b4890f | 19:18 |
| ekarlso- | anyone tell me what is wrong here ? https://review.openstack.org/#/c/145561/ | 19:18 |
| *** yamahata has quit IRC | 19:18 | |
| fungi | ekarlso-: the log got truncated uploading to swift. i'll see if i can find the full output | 19:19 |
| zaro | fungi: well there is a merger section, so looks like the configuration is there and it seems good to me. | 19:20 |
| *** mrunge has joined #openstack-infra | 19:21 | |
| fungi | ekarlso-: i wound up rechecking it to see if we can get a full log from the next run. the log was a day old so not likely possible to dig out of whichever jenkins master it ran on at this point | 19:21 |
| fungi | zaro: and you see a zuul-merger process in the process table and a merger worker available in the gearman status? | 19:22 |
| zaro | fungi: actually the setting zuul_url=http://zuul-dev.openstack.org/p seems incorrect. should the 'p' be there at the end? | 19:22 |
| zaro | fungi: yes, i see merger:merge and merger:update | 19:23 |
| fungi | oh, when you said review-dev you meant zuul-dev? i thought you just had some personal zuul server connected to review-dev. i'll take a look on zuul-dev myself | 19:23 |
| *** yamahata has joined #openstack-infra | 19:23 | |
| zaro | yes, zuul-dev | 19:24 |
| *** EmilienM|afk is now known as EmilienM | 19:24 | |
| fungi | yep, i agree there's one merger worker showing available | 19:25 |
| fungi | zaro: what job are you expecting to trigger? | 19:25 |
| zaro | fungi: gtest | 19:27 |
| zaro | *gtest/gtest-org | 19:27 |
| *** hdd has joined #openstack-infra | 19:27 | |
| clarkb | fungi: I had not. looking now | 19:27 |
| *** Longgeek has quit IRC | 19:27 | |
| *** ddieterly has quit IRC | 19:27 | |
| fungi | zaro: yeah, so your change is got gtest-org/gtest which is in the projects list in layout.yaml and has noop pseudo-jobs defined for all pipelines | 19:28 |
| zaro | fungi: ohh actually i think zuul-dev is just out of date. looking at site.pp i see it's probably missing a few thingys. | 19:28 |
| *** dustins has quit IRC | 19:28 | |
| *** Longgeek has joined #openstack-infra | 19:28 | |
| fungi | zaro: aha, probably a good avenue to check down, yes | 19:29 |
| *** ddieterly has joined #openstack-infra | 19:29 | |
| pleia2 | should the dev servers .pp files typically be kept up to date with the prod ones? (it's something I've thought about when reviewing prod changes, but never did more than think :)) | 19:30 |
| zaro | fungi: thanks for your time. i shore up zuul-dev and see if i can get it to work. | 19:30 |
| *** dustins has joined #openstack-infra | 19:30 | |
| *** dkranz has joined #openstack-infra | 19:32 | |
| *** zz_sabari is now known as sabari | 19:32 | |
| fungi | zaro: you bet--let me know if you hit another dead end | 19:32 |
| *** bswartz1 has quit IRC | 19:32 | |
| fungi | krotscheck: i have a new storyboard-dev building now, based on 140466 applied to a dev env. should know if it's functional shortly | 19:32 |
| *** Longgeek has quit IRC | 19:33 | |
| krotscheck | fungi: Great. | 19:33 |
| *** sabari is now known as zz_sabari | 19:33 | |
| *** wuhg has quit IRC | 19:34 | |
| *** pelix has quit IRC | 19:34 | |
| fungi | pleia2: generally would be a good idea (with some notable exceptions like when we leave out some support intentionally, or are testing some specific divergence on the dev server) | 19:34 |
| *** vigneshvar has quit IRC | 19:34 | |
| * pleia2 nods | 19:34 | |
| clarkb | dib builds are currently not working on new nodepool so I am looking into that | 19:35 |
| clarkb | hopefully can get hpcloud using dib images soon | 19:35 |
| *** marcusvrn has quit IRC | 19:35 | |
| openstackgerrit | Merged openstack-infra/project-config: Broaden pattern match for jenkins-sudo-grep.sh https://review.openstack.org/145978 | 19:35 |
| fungi | krotscheck: it's 2001:4800:7818:104:be76:4eff:fe04:cc25 104.130.158.229 | 19:37 |
| fungi | krotscheck: built with no obvious errors | 19:37 |
| *** zz_sabari is now known as sabari | 19:38 | |
| krotscheck | fungi: Checking | 19:38 |
| fungi | self-signed cert and redirect from http->https seem to work as designed | 19:39 |
| krotscheck | fungi: Indeed. Just logged in without an issue. | 19:39 |
| krotscheck | And it autogranted me admin. Nice. | 19:39 |
| fungi | it knows what's good for it | 19:40 |
| krotscheck | I’m a little concerned that this will make the draft build no longer a seamless experience. | 19:40 |
| clarkb | mordred: you still around? it almost looks like we are leaking stuff into the cache via nodepool/elements/cache-devstack/extra-data.d/50-early-source-repo | 19:40 |
| krotscheck | Originally it was: Click on link, it works! | 19:40 |
| clarkb | mordred: I will probably end up removing the cahce but maybe you want to look first? | 19:40 |
| fungi | anyway, i think this is working enough that we can go ahead and approve 140466 if we can get a second pair of eye on it | 19:40 |
| *** sabari is now known as zz_sabari | 19:41 | |
| fungi | krotscheck: well, the self-signed cert is something you'll only need to accept once | 19:41 |
| krotscheck | Now, it’s going to be click on link, watch-storyboard-stutter-because-you-don’t-know-that-you-have-to-authorize-snakeoil. | 19:41 |
| clarkb | fungi: sure I can take a look | 19:41 |
| *** bhunter71 has quit IRC | 19:41 | |
| *** zz_sabari is now known as sabari | 19:41 | |
| clarkb | fungi: any cahnce you want to look at dib failures on nodepool before I clean out its caceh? | 19:41 |
| fungi | clarkb: any chance there's something there which isn't obvious from the image update log? | 19:42 |
| krotscheck | fungi: True, but it won’t be obvious for users using the draft build, since it’s sitting on a domain with a signed cert. | 19:42 |
| clarkb | fungi: the error is 'Permission denied' trying to access a thing in the cache | 19:42 |
| fungi | clarkb: i would probably just try to fire off a new image build and see where it dies, honestly | 19:42 |
| fungi | clarkb: oh | 19:42 |
| fungi | too fun | 19:42 |
| fungi | looking | 19:42 |
| clarkb | but the cache path looks fine permissions too | 19:42 |
| clarkb | however that repo cache is dirty with all these manifest files that come out of our elements | 19:43 |
| clarkb | krotscheck: fungi in 140466 I think you need to update storyboard.o.o's node {} block to tell it where to get its cert files? | 19:44 |
| clarkb | we default to undef in that cahgne but don't update storyboard.o.o | 19:44 |
| *** kgiusti has joined #openstack-infra | 19:44 | |
| clarkb | or am I missing something with how that fits together? | 19:44 |
| *** TravT has quit IRC | 19:45 | |
| fungi | clarkb: in puppet-storyboard if the content is non-empty but the filename is undef then it has a fallback to a hard-coded default filename | 19:46 |
| fungi | except for the chain file | 19:46 |
| anteaya | has anyone seen mordred around? | 19:47 |
| fungi | er, no, even for the chain file, the logic around what to do with that if the content is undef is just a little more convoluted | 19:47 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Update query for ec2 bug 1408987 https://review.openstack.org/146188 | 19:47 |
| uvirtbot | Launchpad bug 1408987 in nova "tempest failing with boto==2.35.0" [Critical,In progress] https://launchpad.net/bugs/1408987 | 19:47 |
| fungi | anteaya: yep, he's been in and out today | 19:47 |
| anteaya | I am stuck in australia they won't let me transfere to my nz flight | 19:47 |
| fungi | oh, that's no good | 19:47 |
| clarkb | anteaya: he was around earlier today but hasn't popped in for a bit | 19:47 |
| anteaya | I don't go throough the round machine | 19:48 |
| anteaya | so I can not go to nz | 19:48 |
| anteaya | but I don't know if I can get home | 19:48 |
| clarkb | fungi: thanks, I will approve shortly | 19:48 |
| clarkb | as soon as I double check that for my own sanity | 19:48 |
| anteaya | I tried texting him | 19:48 |
| fungi | clarkb: i'm trying to double-check that myswlf as well | 19:48 |
| *** cdent has quit IRC | 19:49 | |
| fungi | clarkb: it looks like you may be right, if we pass in an explicit undef | 19:49 |
| clarkb | anteaya: they don't have an opt out option in AU? /me is trying to remember what perth was like | 19:49 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Update query for ec2 bug 1408987 https://review.openstack.org/146188 | 19:50 |
| uvirtbot | Launchpad bug 1408987 in nova "tempest failing with boto==2.35.0" [Critical,In progress] https://launchpad.net/bugs/1408987 | 19:50 |
| anteaya | clarkb: I'm told they don't | 19:50 |
| anteaya | it didn't come up in perth or brisbane | 19:50 |
| anteaya | perhaps just sydney has a round machine | 19:50 |
| anteaya | I don't know | 19:51 |
| *** mrunge has quit IRC | 19:51 | |
| *** sputnik13 has quit IRC | 19:51 | |
| clarkb | fungi: ya I think the issue there is we are overriding the default | 19:52 |
| clarkb | fungi: because an explicit undef overrides | 19:52 |
| *** palar has quit IRC | 19:52 | |
| *** vigneshvar has joined #openstack-infra | 19:53 | |
| fungi | anteaya: clarkb: looks like they changed it a couple years ago to be non-opt-out http://www.lifehacker.com.au/2013/01/you-cant-opt-out-of-airport-body-scans-in-australia/ | 19:53 |
| *** dprince has quit IRC | 19:53 | |
| clarkb | crinkle: nibalizer: https://review.openstack.org/#/c/140466/ am I wrong about the default behavior for default params there | 19:54 |
| mordred | anteaya: hey! yikes ... | 19:54 |
| anteaya | yeah | 19:54 |
| *** e0ne has quit IRC | 19:54 | |
| mordred | anteaya: I don't know much about what the options are there ... any chance you've tried pinging mikal or lifeless? they both know many things | 19:54 |
| mordred | also - let me poke stewart | 19:55 |
| anteaya | thanks | 19:55 |
| anteaya | sitting and waiting | 19:55 |
| mordred | anteaya: oh - it's 7am there - I probably shouldn't text stewart ... | 19:55 |
| anteaya | ah | 19:55 |
| mordred | maybe lifeless is awake though | 19:55 |
| anteaya | thanks | 19:55 |
| *** otter768 has joined #openstack-infra | 19:56 | |
| anteaya | they are trying to understand my reasons | 19:56 |
| anteaya | they laws don't allow a person to make choices about their own health apparently | 19:56 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Update query for ec2 bug 1408987 https://review.openstack.org/146188 | 19:58 |
| uvirtbot | Launchpad bug 1408987 in nova "tempest failing with boto==2.35.0" [Critical,In progress] https://launchpad.net/bugs/1408987 | 19:58 |
| *** TravT has joined #openstack-infra | 19:59 | |
| *** TravT_ has joined #openstack-infra | 19:59 | |
| *** TravT has quit IRC | 20:01 | |
| *** otter768 has quit IRC | 20:01 | |
| *** bhunter71 has joined #openstack-infra | 20:05 | |
| clarkb | mordred: did you see the thing about the dib cache having trouble on nodepool.o.o? | 20:06 |
| clarkb | mordred: wanted to give you a chance to debug before I rm -rf source_repositories and make it start over | 20:06 |
| openstackgerrit | Joe Gordon proposed openstack-infra/project-config: Reduce number of services running in large-ops https://review.openstack.org/146192 | 20:07 |
| *** marun has joined #openstack-infra | 20:07 | |
| mordred | clarkb: I did not | 20:07 |
| mordred | clarkb: /me reads scrollback | 20:07 |
| clarkb | nodepool.image.build.devstack-trusty-dib: error: cannot open /opt/dib_cache/source-repositories/ci_sandbox_6a5dd9d33a0c8ef15cf705206fa0d8765f0da322/.git/FETCH_HEAD: Permission denied | 20:08 |
| fungi | clarkb: i don't suppose we're running into a max nodes condition there? (~104k inodes in that one directory level) | 20:08 |
| *** annegentle has quit IRC | 20:08 | |
| clarkb | fungi: ya would that manifest as permission denied? | 20:08 |
| mordred | clarkb: when you say "leaking stuff into the cache" - what do you mean | 20:08 |
| fungi | oh, this is in dib_cache not dib_tmp | 20:08 |
| flashgordon | sdague: good news, the large-ops change worked | 20:08 |
| clarkb | mordred: in /opt/dib_cache/source-repositories we have a bunch of dib git manifest files | 20:09 |
| *** andreykurilin has quit IRC | 20:09 | |
| clarkb | mordred: so the git caching thing you have that does the extradata dib magic isn't cleaning up after itself | 20:09 |
| fungi | clarkb: .git is root-owned there... was it trying to fetch as root? | 20:09 |
| mordred | hrm. | 20:09 |
| *** andreykurilin has joined #openstack-infra | 20:09 | |
| mordred | rm -f $GIT_MANIFEST | 20:09 |
| clarkb | fungi: hrm it shouldn't but maybe something did? | 20:09 |
| mattoliverau | Morning all | 20:10 |
| clarkb | mordred: I think the problem is we abort before that point due to the permissions issue | 20:10 |
| mordred | gotcha | 20:10 |
| *** signed8b_ has joined #openstack-infra | 20:10 | |
| *** dhritishikhar has quit IRC | 20:10 | |
| fungi | mattoliverau: have any suggestions for anteaya's au airport body scanner dilemma? | 20:10 |
| clarkb | fungi: mordred: I think the simplest thing now is probably to nuke source-repositories and let it rebuild and see if we can narrow down if it happens again | 20:10 |
| mordred | okie | 20:10 |
| mordred | I'm ok with that | 20:11 |
| zaro | fungi: did a detailed check of zuul-dev configuration, seems ok. i thought it needed to connect to a zuul merger server but it looks like zuul_dev.pp installs the merger on localhost. | 20:11 |
| anteaya | might have to go through it today to avoid prison | 20:11 |
| clarkb | fungi: anything else you want to check before I do that? | 20:11 |
| *** signed8bit has quit IRC | 20:11 | |
| anteaya | since I don't have an au visa | 20:11 |
| fungi | clarkb: yeah, the only other thing i can think of is if we had parallel threads trying to access that (for a while the source repos stuff was not concurrency-safe and would break if we ran multiple dib image creations concurrently) | 20:12 |
| anteaya | but I will be getting a medical certificate in future | 20:12 |
| fungi | s/threads/processes | 20:12 |
| *** annegentle has joined #openstack-infra | 20:12 | |
| anteaya | since if I have one that says I can't go throught the scanner I don't have to go through | 20:12 |
| zaro | fungi: it seems like zuul picks up gerrit changes because i see it in the zuul.log -> "Updating information for 5306" | 20:12 |
| clarkb | fungi: mordred also clean up dib_tmp while I am at it | 20:13 |
| clarkb | I will go ahead and start this | 20:13 |
| zaro | fungi: i'm not seeing any updates in the merger debug logs. | 20:13 |
| fungi | zaro: well, since it has no jobs defined for that change's project it's not actually going to do any work requests (aside from the merge i think) | 20:13 |
| mikal | anteaya: so yeah, if you're talking about the full body scans I don't think they're optional | 20:13 |
| *** aysyd has quit IRC | 20:13 | |
| lifeless | anteaya: hi | 20:14 |
| mikal | http://travelsecure.infrastructure.gov.au/bodyscanners/ | 20:14 |
| *** Masahiro has joined #openstack-infra | 20:14 | |
| lifeless | the US has mitigating things like the constitution for some of the crazy.. once it escapes into the wider world, it usually doesn't. | 20:14 |
| mikal | Quote from the FAQ: | 20:14 |
| *** SotK has quit IRC | 20:14 | |
| mikal | "If a person refuses to undergo a body scan, and they have no medical or physical condition which prevents them for undertaking a body scan, they will be refused clearance and not allowed to pass through the screening point. They therefore will not be allowed to board their aircraft. This policy not only applies to passengers, but also to pilots, other aircrew, government officers and airport staff." | 20:14 |
| mattoliverau | anteaya: what, that's annoying, I thought you could refuse and they'd find some other way. Damn Sydney.. I apologise for my dumb country :( | 20:14 |
| *** ldnunes has quit IRC | 20:15 | |
| fungi | clarkb: okay | 20:15 |
| anteaya | lifeless: hi | 20:15 |
| mattoliverau | Well there you go then | 20:15 |
| mikal | Ahhh, interesting. We have the same ban as the EU on x-ray scanners, but this isn't an x-ray machine apparently. | 20:15 |
| *** SotK has joined #openstack-infra | 20:16 | |
| anteaya | yeah that is what they are telling me | 20:16 |
| anteaya | mikal: that is there point | 20:16 |
| anteaya | with no scientific studies to back it up | 20:16 |
| *** shashankhegde has quit IRC | 20:16 | |
| *** bswartz has joined #openstack-infra | 20:16 | |
| *** sputnik13 has joined #openstack-infra | 20:17 | |
| mikal | anteaya: yeah, I get your objection (I have similar concerns) but I don't see any provision in these rules for the AU govt to let you opt out | 20:17 |
| zaro | fungi: argg! i completely missed that. | 20:18 |
| anteaya | mikal: medical cert | 20:18 |
| timrc | Somewhere Bruce Schneier is sighing. | 20:18 |
| anteaya | which I don't have but am going to get | 20:18 |
| fungi | zaro: well, i'd still expect zuul to update the change with a passing comment or something | 20:18 |
| *** andreaf has joined #openstack-infra | 20:18 | |
| *** annegentle has quit IRC | 20:19 | |
| fungi | zaro: it's configured to operate on the change in that pipeline, just not configured to run any jobs for it | 20:19 |
| clarkb | ok those dirs are cleaned up, now we wait for nodepool to start the next image build | 20:19 |
| *** Masahiro has quit IRC | 20:19 | |
| fungi | clarkb: is it possible the error they failed on was the source repos caching concurrency issue? | 20:20 |
| zaro | do you need to restart zuul to reload layout.yaml? | 20:20 |
| clarkb | fungi: maybe? I wouldn't expect permission denied from that though | 20:20 |
| clarkb | fungi: iirc it manifested in the past as "I don't know what this ref is" | 20:20 |
| mikal | anteaya: there will be an airport doctor. You could ask to talk to them? | 20:21 |
| *** ZZelle has joined #openstack-infra | 20:21 | |
| anteaya | oh good question | 20:21 |
| anteaya | I'm asking | 20:22 |
| fungi | zaro: i think it automatically notices if you change the layout.yaml, but you could check the puppet module to see if it has a special exec subscribed to that file | 20:24 |
| *** annegentle has joined #openstack-infra | 20:25 | |
| *** baoli has joined #openstack-infra | 20:25 | |
| *** baoli_ has quit IRC | 20:25 | |
| *** dimtruck is now known as zz_dimtruck | 20:27 | |
| lifeless | jamielennox|away: sorry, didn't see the context from your ^ | 20:27 |
| anteaya | mikal: they tell me there is no doctor | 20:27 |
| anteaya | they are offering to call me an ambulance | 20:28 |
| *** zz_dimtruck is now known as dimtruck | 20:28 | |
| *** spzala has joined #openstack-infra | 20:28 | |
| anteaya | I don't need an ambulance | 20:28 |
| *** baoli has quit IRC | 20:28 | |
| *** dustins has quit IRC | 20:29 | |
| *** baoli has joined #openstack-infra | 20:29 | |
| lifeless | ok planes | 20:30 |
| *** dimtruck is now known as zz_dimtruck | 20:30 | |
| *** pack3t has joined #openstack-infra | 20:31 | |
| *** sabari is now known as zz_sabari | 20:35 | |
| *** alugovoi has quit IRC | 20:36 | |
| notmyname | anyone else on the 9pm flight out of SFO to AUK today? | 20:36 |
| *** dhritishikhar has joined #openstack-infra | 20:36 | |
| *** alugovoi has joined #openstack-infra | 20:36 | |
| *** thingee has joined #openstack-infra | 20:37 | |
| clarkb | hrm those dib image builds are still running. I need to run errands and will check on them after | 20:38 |
| mikal | I have to drop offline for my flight | 20:38 |
| mikal | I will check IRC later | 20:38 |
| jeblair | anteaya: i did a quick search and do not see a workable SYD->NZ cruise ship option | 20:40 |
| jeblair | notmyname: i depart SFO at 7:45pm (for AKL) | 20:41 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Added manifest for storyboard-dev.openstack.org https://review.openstack.org/140466 | 20:41 |
| fungi | clarkb: i think that ^ solves it? | 20:41 |
| clarkb | fungi: yup | 20:42 |
| crinkle | was just going to comment on that | 20:43 |
| fungi | clarkb: as for the dib builds, if it _is_ the concurrency issue, the outcome i witnessed was that all but one of them would fail with the same error, whichever one managed to win the race for the lock | 20:43 |
| clarkb | fungi: all three are building according to nodepool dib-image-list | 20:43 |
| *** unicell has quit IRC | 20:43 | |
| fungi | yeah, in my case it was around the point where they all tried to clone nova about the same moment | 20:44 |
| fungi | took a while for it to get to that point | 20:44 |
| *** armax has quit IRC | 20:46 | |
| *** hdd has quit IRC | 20:48 | |
| *** shashankhegde has joined #openstack-infra | 20:52 | |
| *** zz_avozza is now known as avozza | 20:52 | |
| *** avozza is now known as zz_avozza | 20:54 | |
| *** armax has joined #openstack-infra | 20:54 | |
| openstackgerrit | Joe Gordon proposed openstack/requirements: Update to latest hacking (0.10.0) https://review.openstack.org/145567 | 20:56 |
| *** armax has quit IRC | 20:59 | |
| *** mrmartin has quit IRC | 21:00 | |
| *** armax has joined #openstack-infra | 21:02 | |
| *** annegent_ has joined #openstack-infra | 21:03 | |
| *** annegentle has quit IRC | 21:04 | |
| *** baoli has quit IRC | 21:06 | |
| *** baoli has joined #openstack-infra | 21:08 | |
| *** ryanpetrello_ has joined #openstack-infra | 21:10 | |
| *** ryanpetrello has quit IRC | 21:12 | |
| *** ryanpetrello_ is now known as ryanpetrello | 21:12 | |
| *** achanda has joined #openstack-infra | 21:13 | |
| *** bswartz has quit IRC | 21:14 | |
| *** bswartz has joined #openstack-infra | 21:16 | |
| *** baoli has quit IRC | 21:17 | |
| *** achanda has quit IRC | 21:18 | |
| *** baoli has joined #openstack-infra | 21:18 | |
| openstackgerrit | Endre Karlson proposed openstack-infra/project-config: Add python-cueclient https://review.openstack.org/145561 | 21:22 |
| *** zul has quit IRC | 21:22 | |
| ekarlso- | can I get a signoff on ^ | 21:23 |
| *** zz_dimtruck is now known as dimtruck | 21:23 | |
| *** zz_sabari is now known as sabari | 21:30 | |
| *** krtaylor has quit IRC | 21:35 | |
| *** TravT_ has quit IRC | 21:36 | |
| *** annegent_ has quit IRC | 21:37 | |
| *** rlandy has quit IRC | 21:37 | |
| *** armax has quit IRC | 21:37 | |
| *** TravT has joined #openstack-infra | 21:37 | |
| *** mjturek has quit IRC | 21:39 | |
| *** mjturek has joined #openstack-infra | 21:39 | |
| *** TravT has quit IRC | 21:41 | |
| *** TravT has joined #openstack-infra | 21:41 | |
| *** dhritishikhar has quit IRC | 21:42 | |
| *** armax has joined #openstack-infra | 21:44 | |
| *** dhritishikhar has joined #openstack-infra | 21:45 | |
| *** zz_gondoi is now known as gondoi | 21:45 | |
| *** krtaylor has joined #openstack-infra | 21:47 | |
| *** zz_avozza is now known as avozza | 21:47 | |
| *** andreykurilin has quit IRC | 21:51 | |
| *** andreykurilin has joined #openstack-infra | 21:51 | |
| *** shashankhegde has quit IRC | 21:53 | |
| *** bswartz has quit IRC | 21:55 | |
| *** otter768 has joined #openstack-infra | 21:57 | |
| *** dhritishikhar has quit IRC | 21:59 | |
| zaro | fungi: what's the chain of events that should happen after | 21:59 |
| dimsum__ | looks like we defined "The OpenStack way"! | 22:00 |
| zaro | fungi: zuul adds a trigger event? | 22:01 |
| *** otter768 has quit IRC | 22:01 | |
| *** hdd has joined #openstack-infra | 22:01 | |
| *** dannywilson has quit IRC | 22:02 | |
| *** bcourt has quit IRC | 22:03 | |
| *** Masahiro has joined #openstack-infra | 22:03 | |
| *** thingee has quit IRC | 22:03 | |
| *** baoli has quit IRC | 22:05 | |
| *** baoli has joined #openstack-infra | 22:05 | |
| *** jamielennox|away has quit IRC | 22:05 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:05 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 22:05 | |
| *** mbacchi has quit IRC | 22:06 | |
| *** jpeeler has quit IRC | 22:06 | |
| *** Masahiro has quit IRC | 22:08 | |
| *** hemna has quit IRC | 22:08 | |
| *** jamielennox|away has joined #openstack-infra | 22:09 | |
| *** jamielennox|away is now known as jamielennox | 22:09 | |
| *** shashankhegde has joined #openstack-infra | 22:09 | |
| *** tonytan4ever has quit IRC | 22:11 | |
| *** armax has quit IRC | 22:11 | |
| *** Hal_ has joined #openstack-infra | 22:14 | |
| *** TravT_ has joined #openstack-infra | 22:15 | |
| *** TravT has quit IRC | 22:17 | |
| *** pc_m has quit IRC | 22:19 | |
| *** dimsum__ has quit IRC | 22:19 | |
| *** dimsum__ has joined #openstack-infra | 22:20 | |
| fungi | zaro: the sequence seems to be: Adding trigger event, Done adding trigger event, Run handler awake, Fetching trigger event, Processing trigger event | 22:20 |
| *** baoli has quit IRC | 22:20 | |
| jeblair | fungi, zaro: something i can help with? | 22:20 |
| fungi | jeblair: troubleshooting the zuul installation on zuul-dev.openstack.org | 22:21 |
| fungi | which is hooked to the upgraded review-dev | 22:21 |
| *** baoli has joined #openstack-infra | 22:21 | |
| zaro | jeblair: i'm trying to debug zuul-dev it doesn't seem to be sending jobs to gearman | 22:22 |
| zaro | when i push a change to review-dev.o.o the last message in zuul debug log is Done adding trigger event: <TriggerEvent patchset-created gtest-org/gtest master 5306,8> | 22:23 |
| *** baoli has quit IRC | 22:23 | |
| zaro | i'm not sure what should happen next. | 22:23 |
| jeblair | 2015-01-09 19:09:08,610 DEBUG zuul.Scheduler: Project gtest-org/gtest not found | 22:24 |
| jeblair | that means there is no configuration for that project in the layout | 22:24 |
| *** baoli has joined #openstack-infra | 22:24 | |
| *** dimsum__ has quit IRC | 22:24 | |
| fungi | interestingly it's in the projects list | 22:24 |
| jeblair | mind if i restart it? | 22:25 |
| fungi | comparing the layout.yaml for zuul and zuul-dev i think the latter needs a lot of updates | 22:25 |
| zaro | ohh wait, the layout file changed. is puppet still running? | 22:26 |
| jeblair | fungi: probably. i don't see the final configuration in the debug log, so i suspect it's invalid in some way | 22:26 |
| *** jpeeler has joined #openstack-infra | 22:26 | |
| *** jpeeler has quit IRC | 22:26 | |
| *** jpeeler has joined #openstack-infra | 22:26 | |
| zaro | i updated the layout.yaml and it somehow changed back to original. | 22:27 |
| *** bswartz has joined #openstack-infra | 22:27 | |
| jeblair | then i think puppet is running :) | 22:27 |
| fungi | if you haven't deactivated puppet agent, it will keep reverting whatever you change in there | 22:27 |
| fungi | as that file is definitely managed by puppet | 22:27 |
| zaro | i thought i did, it's 'sudo service puppet stop ' right? | 22:28 |
| *** bhuvan_ is now known as bhuvan | 22:28 | |
| *** annegentle has joined #openstack-infra | 22:28 | |
| fungi | zaro: no, we don't run puppet agent as a service, haven't for months | 22:28 |
| fungi | puppet agent --disable | 22:28 |
| *** Longgeek has joined #openstack-infra | 22:28 | |
| jeblair | sigh. i thought someone was going to document that. | 22:28 |
| fungi | zaro: i think probably for sanity's sake, the next step ought to be porting the current production config and layout to the dev environment | 22:29 |
| zaro | ughh! actually clark told me that, but nibilizer said it's sudo command so i thought both were same. | 22:29 |
| *** mattfarina has quit IRC | 22:29 | |
| *** TravT_ has quit IRC | 22:29 | |
| *** patrickeast has quit IRC | 22:30 | |
| *** baoli has quit IRC | 22:30 | |
| zaro | fungi: it's in review https://review.openstack.org/#/c/145934/ | 22:30 |
| *** sabeen1 has quit IRC | 22:30 | |
| *** armax has joined #openstack-infra | 22:30 | |
| fungi | oh, awesome. looking | 22:30 |
| *** otter768 has joined #openstack-infra | 22:31 | |
| zaro | command to start zuul is '/etc/init.d/zuul' start and '/etc/init.d/zuul-merger' start correct? | 22:31 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Document how to disable/enable puppet https://review.openstack.org/146216 | 22:32 |
| *** gondoi is now known as zz_gondoi | 22:32 | |
| clarkb | it should already be documented | 22:32 |
| *** Longgeek has quit IRC | 22:32 | |
| jeblair | clarkb: what should? | 22:33 |
| *** banix has quit IRC | 22:33 | |
| jeblair | oh, it's here: http://ci.openstack.org/puppet.html#disabling-puppet-on-nodes | 22:34 |
| *** annegentle has quit IRC | 22:34 | |
| zaro | jeblair: only way i was able to get review-dev to _not_ timeout is to change the connectionpool setting to false. | 22:35 |
| *** packet has quit IRC | 22:35 | |
| *** pack3t is now known as packet | 22:35 | |
| jeblair | zaro: please keep track of that problem. we need to solve it correctly before going into production. | 22:35 |
| jeblair | i don't think disabling the connection pool without understanding what is going on or why is okay | 22:36 |
| jeblair | (i think it's fine to do that to proceed with your work, but we need to start keeping a "punch list" of things like that) | 22:36 |
| *** packet has quit IRC | 22:37 | |
| *** mriedem has quit IRC | 22:38 | |
| zaro | yeah, i'm keeping track of it all. i'll put it on etherpad shortly so you can see. | 22:38 |
| *** teran has joined #openstack-infra | 22:40 | |
| clarkb | hrm those image builds are still going | 22:41 |
| mordred | clarkb: didn't we blow away the dib source-repositories cache earlier? | 22:42 |
| clarkb | mordred: yes | 22:43 |
| clarkb | but thats all in the cache and shouldn't affect anything | 22:43 |
| clarkb | also the build sseem to have gotten stuck before that | 22:43 |
| *** teran has quit IRC | 22:44 | |
| *** HeOS has quit IRC | 22:44 | |
| clarkb | I don't see any exceptions grepping for devstack-trusty-dib. Now to grep for more targetted info | 22:45 |
| clarkb | http://paste.openstack.org/show/156150/ | 22:48 |
| *** sabari is now known as zz_sabari | 22:49 | |
| clarkb | I wonder if we dont handle that particular error properly in nodepool | 22:50 |
| mordred | clarkb: that seems less than informative | 22:50 |
| *** signed8b_ is now known as signed8bit_ZZZzz | 22:50 | |
| *** baoli has joined #openstack-infra | 22:50 | |
| jeblair | mordred, clarkb: the actual error will probably be in the image build log | 22:50 |
| *** esker has quit IRC | 22:50 | |
| jeblair | i think the usage of "unable to create /path" is more "i couldn't build this image" not "there is a problem opening this file". | 22:51 |
| clarkb | jeblair: right I know the actual error. Its the permissions denied thing | 22:51 |
| clarkb | jeblair: yup | 22:51 |
| clarkb | the major problem is that nodepool still thinks it is running those builds so cleanup wasn't properly | 22:51 |
| jeblair | ah, gotcha | 22:52 |
| *** sarob has joined #openstack-infra | 22:52 | |
| *** andreaf has quit IRC | 22:52 | |
| clarkb | I think I may see the problem | 22:52 |
| *** pradk has quit IRC | 22:52 | |
| *** armax has quit IRC | 22:53 | |
| clarkb | it was sufficient to simply return when catching that exception have to remove the previously commited data for the building state | 22:53 |
| clarkb | *wasn't sufficient | 22:53 |
| *** otter768 has quit IRC | 22:53 | |
| *** dimtruck is now known as zz_dimtruck | 22:54 | |
| *** mfink has quit IRC | 22:54 | |
| mordred | nod | 22:55 |
| mordred | you know - there are times when I think we should use fixtures for more things in non-testing land | 22:55 |
| fungi | or exit traps | 22:55 |
| jeblair | i disagree. there's a really simple exception handling structure in nodepool. | 22:58 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/nodepool: Raise a more descriptive KeyError for debugging https://review.openstack.org/146219 | 22:58 |
| *** jcoufal has quit IRC | 23:00 | |
| jeblair | basically, things should try to do something, and if they fail, try to undo it, and then let the top level catch every exception so that it logs it and tries again | 23:00 |
| clarkb | ya this was tricky becaues before we weren't handling the error and my fix was insufficient | 23:01 |
| *** EmilienM is now known as EmilienM|afk | 23:01 | |
| clarkb | that'll learn me | 23:01 |
| jeblair | this is what exceptions are for -- cleanup and state change on error. we're not exiting the program or thread, so fixtures and exit traps are not a good fit. | 23:02 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Cleanup DIB images in DB if they fail to build https://review.openstack.org/146220 | 23:02 |
| *** zz_sabari is now known as sabari | 23:02 | |
| fungi | i was referring to the possibility of exit traps in diskimage-builder, to optionally clean up when it fails | 23:03 |
| jeblair | i'm told dib cleans up after itself | 23:03 |
| *** mfink has joined #openstack-infra | 23:03 | |
| clarkb | ya it should. I think the particular case here where it didn't is a bug in the git repo element | 23:04 |
| clarkb | that mordred added | 23:04 |
| clarkb | in the psat we always marked it ready so we would transition state. Sometimes to the wrong state. Removing the db row should be what we want | 23:06 |
| *** baoli has quit IRC | 23:07 | |
| *** baoli has joined #openstack-infra | 23:08 | |
| morganfainberg | sdague, thanks for all the responses on that thread: KSC and middleware | 23:09 |
| morganfainberg | sdague, really appreciate it :) | 23:09 |
| morganfainberg | sdague, helps a lot with trying to get all of this cleaned up / resolved in a consistent manner (and one that wont break *theworld*) | 23:10 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Make DIB exceptions more robust https://review.openstack.org/146222 | 23:12 |
| *** armax has joined #openstack-infra | 23:13 | |
| jeblair | clarkb: lgtm (though there was a typo in commit msg i think). i added another (untested) patch to bring logging and exception patterns more inline. | 23:14 |
| clarkb | ya I am looking at your change now | 23:16 |
| *** amitgandhinz has quit IRC | 23:16 | |
| *** dkliban is now known as dkliban_afk | 23:16 | |
| *** dannywilson has joined #openstack-infra | 23:18 | |
| openstackgerrit | Merged openstack-infra/nodepool: Cleanup DIB images in DB if they fail to build https://review.openstack.org/146220 | 23:19 |
| fungi | in 146222 the DiskImageBuilder.run() method no longer sets self.disk_image at all. is that going to cause a problem or was it not necessary to track in the object? | 23:19 |
| clarkb | I thought it wasn't needed in the class object | 23:21 |
| *** zz_dimtruck is now known as dimtruck | 23:21 | |
| fungi | yeah, i'm not finding any other places accessing that attribute | 23:22 |
| fungi | lgtm | 23:22 |
| clarkb | I can do a quick restart and delete the three rows in the db when those merge | 23:22 |
| *** armax has quit IRC | 23:23 | |
| sdague | morganfainberg: no prob | 23:27 |
| openstackgerrit | Merged openstack-infra/nodepool: Make DIB exceptions more robust https://review.openstack.org/146222 | 23:27 |
| *** ChuckC_ has joined #openstack-infra | 23:28 | |
| clarkb | cool once puppet update sthe nodepool install I will do a restart | 23:28 |
| *** banix has joined #openstack-infra | 23:29 | |
| *** ChuckC has quit IRC | 23:29 | |
| *** baoli has quit IRC | 23:30 | |
| * jeblair transits | 23:33 | |
| jeblair | have a nice weekend | 23:33 |
| *** jeblair is now known as corvus | 23:33 | |
| *** vigneshvar has quit IRC | 23:34 | |
| *** hdd has quit IRC | 23:34 | |
| *** ryanpetrello has quit IRC | 23:36 | |
| *** ryanpetrello has joined #openstack-infra | 23:37 | |
| clarkb | ok nodepool is up to date going to restart it now | 23:37 |
| *** ChuckC_ has quit IRC | 23:39 | |
| openstackgerrit | Joshua Harlow proposed openstack-infra/project-config: Add debtcollector project configuration https://review.openstack.org/146225 | 23:40 |
| *** dimsum__ has joined #openstack-infra | 23:40 | |
| *** dimsum__ has quit IRC | 23:41 | |
| *** ryanpetrello has quit IRC | 23:42 | |
| *** dimsum__ has joined #openstack-infra | 23:42 | |
| *** ivar-lazzaro has joined #openstack-infra | 23:43 | |
| *** pcaruana|afk| has quit IRC | 23:46 | |
| zaro | fungi: i can't figure out why zuul says it cannot find gtest-org/gtest, layout looks correct to me. | 23:46 |
| zaro | is there a way to query zuul for the projects it knows about? | 23:46 |
| fungi | it enumerates the configuration it has parsed when it starts, in the debug loh | 23:47 |
| fungi | log | 23:47 |
| *** mriedem has joined #openstack-infra | 23:47 | |
| *** dmsimard is now known as dmsimard_away | 23:47 | |
| fungi | neat, we have a zombified zuul-server process there too | 23:47 |
| fungi | mind if i stop all the processes and try to start them fresh? | 23:48 |
| zaro | nope, go ahead | 23:48 |
| fungi | oh, yeah, also missing pidfile | 23:48 |
| zaro | i probably started them all wrong. | 23:49 |
| fungi | it's still not writing a pidfile either, so it's likely pretty ill. checking the logs now | 23:51 |
| *** Masahiro has joined #openstack-infra | 23:52 | |
| *** sarob has quit IRC | 23:54 | |
| *** sarob_ has joined #openstack-infra | 23:54 | |
| *** dangers is now known as dangers_away | 23:54 | |
| *** ddieterly has quit IRC | 23:54 | |
| *** banix has quit IRC | 23:55 | |
| openstackgerrit | Joshua Harlow proposed openstack-infra/project-config: Add debtcollector project configuration https://review.openstack.org/146225 | 23:55 |
| *** teran has joined #openstack-infra | 23:56 | |
| mriedem | anyone else see this before? http://logs.openstack.org/64/146064/3/check/check-requirements-integration-dsvm/692929f/console.html#_2015-01-09_21_29_23_058 | 23:56 |
| *** Masahiro has quit IRC | 23:56 | |
| mriedem | logstash shows this quite a bit and it's not always a failure | 23:56 |
| mriedem | blocking the backport of the pin to boto on stable/juno - which blocks all other stable/juno nova changes | 23:56 |
| *** ChuckC_ has joined #openstack-infra | 23:57 | |
| *** dimsum__ has quit IRC | 23:58 | |
| *** dimsum__ has joined #openstack-infra | 23:58 | |
| fungi | mriedem: i believe someone filed a bug against httpretty earlier in the week noting that it is unable to properly wheel | 23:58 |
| mriedem | looking | 23:59 |
| clarkb | no I did't because httpretty's latest release is able to do it | 23:59 |
| clarkb | the version we pin to is not | 23:59 |
| fungi | https://github.com/gabrielfalcao/HTTPretty/issues/149 | 23:59 |
| *** melwitt has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!