| *** tphummel has joined #openstack-infra | 00:03 | |
| *** Sukhdev has joined #openstack-infra | 00:03 | |
| *** pradk has quit IRC | 00:03 | |
| *** pahuang has joined #openstack-infra | 00:05 | |
| *** sdague has quit IRC | 00:06 | |
| openstackgerrit | Merged openstack-infra/subunit2sql: Add unit test for api.update_run https://review.openstack.org/277717 | 00:10 |
|---|---|---|
| asselin_ | can anyone explain http://graphite.openstack.org/ data? I'm looking at stats.nodepool.task.bluebox-sjc1.AddFloatingIPTask but seems the data is of by an order of magnitude | 00:12 |
| *** Qiming has quit IRC | 00:16 | |
| SpamapS | asselin_: what is your evidence it is off? | 00:16 |
| SpamapS | asselin_: seems to mirror CreateServerTask | 00:16 |
| asselin_ | SpamapS, trying to make sense of the data in my nodepool running. log files are showing different times than statsd | 00:16 |
| asselin_ | ran task <nodepool.provider_manager.GetFloatingIPTask object at 0x7f0bec411b50> in 0.705707073212s | 00:17 |
| asselin_ | but I don't see anything over 0.5s | 00:17 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql: Add missing release notes before the 1.3.0 release https://review.openstack.org/281590 | 00:18 |
| SpamapS | asselin_: 205ms could just be bad network latency? | 00:18 |
| asselin_ | SpamapS, ?? | 00:19 |
| mordred | asselin_: I think the data is in miliseconds | 00:20 |
| *** aeng has quit IRC | 00:20 | |
| *** sabeen3 has quit IRC | 00:20 | |
| mordred | or something | 00:20 |
| SpamapS | asselin_: the difference between 0.5s and 0.7s is 200ms | 00:20 |
| *** tzn has joined #openstack-infra | 00:20 | |
| asselin_ | SpamapS, right...seems to me that the time in the log message should match the time in graphite regardless of network latency | 00:20 |
| SpamapS | asselin_: that's not a safe assumption. graphite summarizes. | 00:21 |
| *** aeng has joined #openstack-infra | 00:21 | |
| asselin_ | yeah that's the part I'm trying to understand....i think something's off still... | 00:23 |
| zaro | jeblair: RE:not being able to search https://review.openstack.org/#/c/13725 | 00:23 |
| zaro | jeblair: looks like there's some inconsistency with that change. | 00:23 |
| *** KimJ has quit IRC | 00:24 | |
| *** mdenny has quit IRC | 00:24 | |
| zaro | doing 'git fetch https://review.openstack.org/openstack-infra/system-config refs/changes/25/13725/8 && git checkout FETCH_HEAD' returns 'fatal: Couldn't find remote ref refs/changes/25/13725/8' | 00:24 |
| zaro | jeblair: you might want to give this a try: https://gerrit-documentation.storage.googleapis.com/Documentation/2.11/rest-api-changes.html#fix-change | 00:25 |
| *** ccrouch1 has joined #openstack-infra | 00:25 | |
| *** Sukhdev has quit IRC | 00:26 | |
| *** tzn has quit IRC | 00:26 | |
| *** ccrouch has quit IRC | 00:27 | |
| *** pahuang has quit IRC | 00:28 | |
| *** ybathia has joined #openstack-infra | 00:29 | |
| *** ybathia has quit IRC | 00:29 | |
| openstackgerrit | Merged openstack/requirements: bump upper constraint for keystoneclient https://review.openstack.org/281030 | 00:30 |
| *** zz_dimtruck is now known as dimtruck | 00:31 | |
| *** sridhar_ram1 has joined #openstack-infra | 00:31 | |
| *** ashleighfarnham has quit IRC | 00:32 | |
| asselin_ | SpamapS, so I see this in my log file, but nothing of that size in graphite... | 00:32 |
| asselin_ | 2016-02-18 00:31:28,115 DEBUG nodepool.ProviderManager: Manager mpc ran task <nodepool.provider_manager.DeleteFloatingIPTask object at 0x7f0be80464d0> in 113.744323969s | 00:32 |
| asselin_ | 2016-02-18 00:31:28,116 DEBUG nodepool.ProviderManager: nodepool.task.mpc.DeleteFloatingIPTask 113744 | 00:32 |
| *** harlowja_at_home has quit IRC | 00:33 | |
| *** harlowja_at_home has joined #openstack-infra | 00:33 | |
| *** sridhar_ram has quit IRC | 00:33 | |
| asselin_ | my graphite is showing everthing < 0.15s.....??? | 00:33 |
| asselin_ | that's a few orders of magnitude off.... | 00:35 |
| *** ybathia has joined #openstack-infra | 00:39 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/zmq-event-publisher: Update links in pom.xml https://review.openstack.org/281540 | 00:40 |
| *** _amrith_ is now known as amrith | 00:40 | |
| nibalizer | clarkb: so that ^^ + the wiki page I created (linked) should fix the zmq-e-p problem | 00:40 |
| clarkb | nibalizer: you are too kind | 00:41 |
| clarkb | nibalizer: we will have to do a new release once that merges in order to push back to jenkins but that is easy enough | 00:42 |
| *** qb_ has joined #openstack-infra | 00:42 | |
| *** thorst has joined #openstack-infra | 00:42 | |
| openstackgerrit | Merged openstack-infra/subunit2sql: Add missing release notes before the 1.3.0 release https://review.openstack.org/281590 | 00:42 |
| clarkb | actually I can probably just approve that since it is mostly bookeeping will go ahea dand do so | 00:43 |
| clarkb | nibalizer: then one of us can push the tag to release | 00:43 |
| *** Sukhdev has joined #openstack-infra | 00:43 | |
| *** xarses has quit IRC | 00:45 | |
| *** pahuang has joined #openstack-infra | 00:45 | |
| *** rcernin has quit IRC | 00:46 | |
| *** dimtruck is now known as zz_dimtruck | 00:46 | |
| clarkb | jeblair: any opposition to me just reapproving https://review.openstack.org/#/c/273703/13 ? the dsvm test that failed did so because snapshot build had not completed in time | 00:47 |
| openstackgerrit | Merged openstack-infra/zmq-event-publisher: Update links in pom.xml https://review.openstack.org/281540 | 00:48 |
| clarkb | nibalizer: ^ you want to tag and push? | 00:48 |
| *** harlowja_at_home has quit IRC | 00:48 | |
| *** harlowja_at_home has joined #openstack-infra | 00:49 | |
| asselin_ | clarkb, nibalizer could one of you give me some of this output? tail -f /var/log/nodepool/debug.log | grep "ran task" | 00:50 |
| clarkb | asselin_: how much do you want? | 00:51 |
| asselin_ | clarkb, just enough to sanity check the data in graphite | 00:51 |
| clarkb | asselin_: http://paste.openstack.org/show/487357/ thats about a minutes worth | 00:52 |
| asselin_ | thank you | 00:52 |
| *** Jeffrey4l has joined #openstack-infra | 00:54 | |
| *** zz_dimtruck is now known as dimtruck | 00:55 | |
| *** ybathia has quit IRC | 00:55 | |
| asselin_ | clarkb, ok so I'm not crazy...something is off. e.g. ovh-bhs1 ran task ListServersTask is all 2.5 s and graphite is showing them as 0.20 s | 00:55 |
| clarkb | asselin_: how are you querying? can you paste your url here? | 00:55 |
| nibalizer | clarkb: sure | 00:57 |
| asselin_ | clarkb, not sure how to get the url....here's a screen shot: http://imgur.com/UHsDerv | 00:57 |
| *** bpokorny has quit IRC | 00:57 | |
| openstackgerrit | Merged openstack/requirements: bump upper constraint for pycadf https://review.openstack.org/281015 | 00:58 |
| *** bpokorny has joined #openstack-infra | 00:58 | |
| clarkb | I think if you go to graph options there is a thing you can do | 00:58 |
| asselin_ | clarkb, graph data is stats.nodepool.task.ovh-bhs1.ListServersTask | 00:58 |
| clarkb | keep in mind that that says nothing about the scale of the data | 00:58 |
| nibalizer | clarkb: 0.0.4? | 00:58 |
| clarkb | nibalizer: ya just the minimal bump because we havne't really done anything | 00:58 |
| *** harlowja_at_home has quit IRC | 00:58 | |
| clarkb | nibalizer: just bookkeeping | 00:59 |
| asselin_ | clarkb, ok...open image in a new tab: http://graphite.openstack.org/render/?width=586&height=308&_salt=1455757161.847&target=stats.nodepool.task.ovh-bhs1.ListServersTask | 01:00 |
| clarkb | ok so it isn't doing any munging should just show you raw data points, now we need to map back to nodepool and see what those are | 01:00 |
| clarkb | dt = last_ts - start ; self.statsd.timing(key, int(dt * 1000)) | 01:01 |
| clarkb | both of those are time.time() output so it is seconds * 1000 to get miliseconds | 01:02 |
| *** weshay has quit IRC | 01:03 | |
| clarkb | next we can grab the json output | 01:03 |
| *** achanda has quit IRC | 01:03 | |
| asselin_ | clarkb, how do you grab the json output? | 01:04 |
| clarkb | asselin_: I would have to look that up but its another query parameter so &something=json | 01:04 |
| clarkb | output maybe? | 01:04 |
| clarkb | format=json | 01:05 |
| asselin_ | oh..the graphite json data | 01:05 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for Heat booting a VM on a network w/o subnet https://review.openstack.org/280824 | 01:05 |
| *** sdake_ is now known as sdake | 01:05 | |
| clarkb | http://graphite.openstack.org/render/?width=586&height=308&_salt=1455757161.847&target=stats.nodepool.task.ovh-bhs1.ListServersTask&format=json | 01:05 |
| clarkb | each of those is a tuple raw data and timestamp | 01:05 |
| *** eil397 has quit IRC | 01:05 | |
| clarkb | I don't know why those appear to be skewed | 01:06 |
| clarkb | oh it calls statsd.timing | 01:06 |
| clarkb | so you need to look at what that does to timestamps | 01:06 |
| clarkb | before shipping them off | 01:06 |
| nibalizer | clarkb: tag pushed | 01:06 |
| clarkb | nibalizer: kk, that should cause some release jobs to run (at least one) that will push to jenkins maven reo | 01:06 |
| *** sridhar_ram1 has quit IRC | 01:06 | |
| *** Swami has quit IRC | 01:07 | |
| asselin_ | clarkb, the data points in the json match the graph .01 0.15 | 01:07 |
| nibalizer | woot | 01:07 |
| *** doug-fish has joined #openstack-infra | 01:08 | |
| clarkb | asselin_: ya but notice that the function call is timing specific, it may do data munging | 01:09 |
| clarkb | asselin_: something to chekc up on at least | 01:09 |
| * asselin_ looks | 01:09 | |
| asselin_ | self._send_stat(stat, '%0.6f|ms' % delta, rate) | 01:10 |
| asselin_ | not sure why its using delta...?? | 01:10 |
| *** doug-fish has quit IRC | 01:11 | |
| *** daemontool__ has joined #openstack-infra | 01:11 | |
| asselin_ | maybe it should just be set self._send_stat(stat, '%s|s' % value, rate) | 01:11 |
| clarkb | delta because it is the difference of two times | 01:12 |
| clarkb | and that is saying ms which is correct so ya I dunno | 01:12 |
| *** ccrouch1 has quit IRC | 01:12 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: yum-minimal: clear our rpm/dnf/yum data in chroot https://review.openstack.org/281596 | 01:12 |
| * asselin_ finds the manual | 01:13 | |
| asselin_ | http://statsd.readthedocs.org/en/v3.1/reference.html#timing | 01:14 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for Heat booting a VM on a network w/o subnet https://review.openstack.org/280824 | 01:14 |
| *** Qiming has joined #openstack-infra | 01:14 | |
| *** ccrouch has joined #openstack-infra | 01:15 | |
| asselin_ | The statsd server collects all timers under the stats.timers prefix, and will calculate the lower bound, mean, 90th percentile, upper bound, and count of each timer for each period (by the time you see it in Graphite, that’s usually per minute)." | 01:15 |
| *** chenli has joined #openstack-infra | 01:15 | |
| asselin_ | so that graph is not timer | 01:16 |
| jeblair | clarkb: please do re+3 https://review.openstack.org/273703 | 01:18 |
| *** mtanino has quit IRC | 01:18 | |
| asselin_ | ok so this here? I guess it's ms? http://graphite.openstack.org/render/?width=822&height=404&_salt=1455758356.077&target=stats.timers.nodepool.task.ovh-bhs1.ListServersTask.mean | 01:19 |
| Kennan | AJaeger: I saw issues like | 01:20 |
| Kennan | fter connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='mirror.iad.rax.openstack.org', port=80): Read timed out | 01:20 |
| *** achanda has joined #openstack-infra | 01:20 | |
| Kennan | which makes jenkins build timeout | 01:20 |
| Kennan | it seems related connection issue in jenkins side | 01:21 |
| jeblair | asselin_: stats.nodepool.task.X is not a timer | 01:22 |
| jeblair | clarkb: ^ | 01:23 |
| *** xarses has joined #openstack-infra | 01:23 | |
| clarkb | ah | 01:23 |
| clarkb | a counter then? | 01:23 |
| asselin_ | jeblair, yeah I noticed that...mentioned it above too... | 01:23 |
| jeblair | asselin_, clarkb: that stat is a frequency -- so basically calls / sec | 01:23 |
| jeblair | clarkb: yeah | 01:23 |
| jeblair | clarkb: the raw version of it would be in stats_counts (which would be the absolute count per time period) | 01:24 |
| jeblair | asselin_: stats.timers. is where you'll fine the durations | 01:24 |
| jeblair | asselin_: ah i see your later graph link, yes | 01:24 |
| asselin_ | jeblair, yeah...and it seems its all in ms right?, e.g. 25K is 25 seconds | 01:24 |
| jeblair | asselin_: correct all times are in ms | 01:24 |
| asselin_ | jeblair, i think I mentioned I'm new to graphite...so I'm more likely the one in err | 01:25 |
| jeblair | asselin_: if you check out the grafana dashboard for ovs, you'll see it convert them back to seconds: http://grafana.openstack.org/dashboard/db/nodepool-ovh | 01:25 |
| jeblair | heh, er, ovh :) | 01:25 |
| *** kzaitsev_mb has quit IRC | 01:26 | |
| jeblair | which seems to match the log lines (but always good to double check unit conversions in graphs) | 01:26 |
| openstackgerrit | Merged openstack-infra/nodepool: Handle dib image deletion during periodic cleanup https://review.openstack.org/273703 | 01:27 |
| openstackgerrit | Merged openstack/requirements: glance-store 0.11.0 https://review.openstack.org/280891 | 01:28 |
| openstackgerrit | Merged openstack/requirements: Updated from generate-constraints https://review.openstack.org/281071 | 01:28 |
| openstackgerrit | Merged openstack/requirements: Upperbound on pysaml2 to avoid pycryptodome https://review.openstack.org/280968 | 01:29 |
| asselin_ | clarkb, jeblair thanks for helping me out...I'm making sense of the data now. | 01:30 |
| jeblair | asselin_: great! you can help us with dashboards now! ;) | 01:32 |
| asselin_ | sure can :) | 01:32 |
| fungi | Kennan: there was an outage for that mirror around 20 hours ago... when were you seeing it? | 01:33 |
| *** sdake has quit IRC | 01:34 | |
| *** pvaneck has quit IRC | 01:38 | |
| Kennan | fungi: let me check other new jenkins jobs. maybe that was the time when mirror hit that issue | 01:38 |
| *** sputnik13 has quit IRC | 01:38 | |
| *** thorst has quit IRC | 01:44 | |
| *** sam_wan has joined #openstack-infra | 01:44 | |
| *** esker has joined #openstack-infra | 01:46 | |
| *** pahuang has quit IRC | 01:48 | |
| *** zhurong has joined #openstack-infra | 01:48 | |
| *** esker has quit IRC | 01:51 | |
| *** jpr has joined #openstack-infra | 01:55 | |
| *** sam_wan has quit IRC | 01:56 | |
| *** sam_wan has joined #openstack-infra | 01:58 | |
| dougwig | fungi: good idea | 01:59 |
| fungi | dougwig: i had a good idea? when/where? | 02:00 |
| dougwig | ML/review page. | 02:00 |
| *** sam_wan has quit IRC | 02:01 | |
| *** sam_wan has joined #openstack-infra | 02:01 | |
| *** pahuang has joined #openstack-infra | 02:02 | |
| *** _sarob has joined #openstack-infra | 02:02 | |
| *** sdake has joined #openstack-infra | 02:03 | |
| *** sdake has quit IRC | 02:03 | |
| *** sdake has joined #openstack-infra | 02:04 | |
| *** ryuusei has joined #openstack-infra | 02:05 | |
| *** rhallisey has joined #openstack-infra | 02:05 | |
| *** sam_wan has quit IRC | 02:05 | |
| *** pahuang has quit IRC | 02:07 | |
| *** doug-fish has joined #openstack-infra | 02:08 | |
| *** jpr has quit IRC | 02:08 | |
| *** doug-fish has quit IRC | 02:08 | |
| jhesketh | jeblair: when you get time, your feedback on my last comment here would be appreciated: https://review.openstack.org/#/c/281166/2 | 02:09 |
| *** tphummel has quit IRC | 02:10 | |
| *** changzhi has joined #openstack-infra | 02:11 | |
| smcginnis | Hmm, possible issue. Build artifacts are missing here: http://logs.openstack.org/25/274825/8/check/gate-cinder-pep8/d0579d6/ | 02:11 |
| smcginnis | pep8 failure here: https://review.openstack.org/#/c/274825/ | 02:12 |
| *** wolverineav has quit IRC | 02:13 | |
| *** wolverineav has joined #openstack-infra | 02:14 | |
| *** mriedem has left #openstack-infra | 02:14 | |
| smcginnis | Same on this one: https://review.openstack.org/#/c/224910/ | 02:15 |
| *** wolverineav has quit IRC | 02:18 | |
| *** yamahata has quit IRC | 02:19 | |
| *** pahuang has joined #openstack-infra | 02:20 | |
| *** wolverineav has joined #openstack-infra | 02:22 | |
| *** dkranz has joined #openstack-infra | 02:23 | |
| *** tzn has joined #openstack-infra | 02:24 | |
| *** angdraug has quit IRC | 02:26 | |
| *** wolverineav has quit IRC | 02:27 | |
| *** tzn has quit IRC | 02:29 | |
| *** Guest22069 is now known as jogo | 02:30 | |
| *** jogo has quit IRC | 02:31 | |
| *** jogo has joined #openstack-infra | 02:31 | |
| *** jogo has joined #openstack-infra | 02:31 | |
| *** ryuusei has quit IRC | 02:33 | |
| *** hichihara has joined #openstack-infra | 02:33 | |
| *** Sukhdev has quit IRC | 02:33 | |
| *** harlowja_at_home has joined #openstack-infra | 02:34 | |
| *** sam_wan has joined #openstack-infra | 02:34 | |
| *** dkranz has quit IRC | 02:36 | |
| *** _sarob has quit IRC | 02:40 | |
| *** thorst has joined #openstack-infra | 02:45 | |
| *** rlandy has quit IRC | 02:46 | |
| *** thorst has quit IRC | 02:53 | |
| *** tiswanso has joined #openstack-infra | 02:53 | |
| *** apoorvad has quit IRC | 02:54 | |
| *** esker has joined #openstack-infra | 02:55 | |
| *** yamahata has joined #openstack-infra | 03:00 | |
| *** esker has quit IRC | 03:00 | |
| *** ganso has quit IRC | 03:00 | |
| *** jpr has joined #openstack-infra | 03:01 | |
| *** doug-fish has joined #openstack-infra | 03:04 | |
| *** wolverineav has joined #openstack-infra | 03:04 | |
| *** salv-orl_ has quit IRC | 03:04 | |
| *** shashank_hegde has joined #openstack-infra | 03:07 | |
| *** doug-fish has quit IRC | 03:09 | |
| *** wolverineav has quit IRC | 03:11 | |
| *** gildub has quit IRC | 03:11 | |
| *** wolverineav has joined #openstack-infra | 03:12 | |
| jhesketh | any cores around to help reivew https://review.openstack.org/#/c/281016/ ? | 03:17 |
| *** jpr has quit IRC | 03:19 | |
| *** sridhar_ram has joined #openstack-infra | 03:19 | |
| *** sdake has quit IRC | 03:20 | |
| *** bpokorny has quit IRC | 03:20 | |
| *** yuanying_ has quit IRC | 03:21 | |
| *** amrith is now known as _amrith_ | 03:22 | |
| *** sridhar_ram has quit IRC | 03:22 | |
| *** tzn has joined #openstack-infra | 03:25 | |
| *** shashank_hegde has quit IRC | 03:28 | |
| *** tzn has quit IRC | 03:29 | |
| *** dims_ has quit IRC | 03:30 | |
| *** dims has joined #openstack-infra | 03:32 | |
| *** shashank_hegde has joined #openstack-infra | 03:35 | |
| *** gildub has joined #openstack-infra | 03:35 | |
| *** woodster_ has quit IRC | 03:36 | |
| *** rguillebert has quit IRC | 03:37 | |
| tonyb | jhesketh: I need to disappear but: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ERROR%3Azuul.Repo%3AUnable%5C%22%20AND%20tags%3A%5C%22console%5C%22 (7days) seem to show an increase to me | 03:39 |
| *** yuanying has joined #openstack-infra | 03:39 | |
| jhesketh | it does seem that way | 03:39 |
| *** jlk has quit IRC | 03:39 | |
| jhesketh | need to get some reviews on https://review.openstack.org/#/c/281130/ | 03:40 |
| *** jlk has joined #openstack-infra | 03:40 | |
| *** jlk has quit IRC | 03:40 | |
| *** jlk has joined #openstack-infra | 03:40 | |
| *** yamamoto has joined #openstack-infra | 03:41 | |
| *** harlowja_at_home has quit IRC | 03:41 | |
| *** shashank_hegde has quit IRC | 03:42 | |
| *** shashank_hegde has joined #openstack-infra | 03:45 | |
| *** shashank_hegde has quit IRC | 03:47 | |
| *** KanagarajM has joined #openstack-infra | 03:48 | |
| *** thorst has joined #openstack-infra | 03:51 | |
| openstackgerrit | Merged openstack-infra/project-config: Add new job template for dsvm plugins with src https://review.openstack.org/281016 | 03:52 |
| *** fedexo has joined #openstack-infra | 03:52 | |
| *** yuanying has quit IRC | 03:55 | |
| *** baoli has joined #openstack-infra | 03:55 | |
| *** thorst has quit IRC | 03:58 | |
| KanagarajM | ianw, hi arround? | 04:01 |
| ianw | KanagarajM: yep | 04:01 |
| KanagarajM | ianw, could you please have a look https://review.openstack.org/280662 | 04:01 |
| *** chenli has quit IRC | 04:02 | |
| openstackgerrit | Merged openstack-infra/project-config: Add Neutron Failure Rate dashboard to Grafana https://review.openstack.org/278832 | 04:03 |
| ianw | KanagarajM: honestly, I usually leave new project stuff to others who have better experience in that area | 04:03 |
| *** rhallisey has quit IRC | 04:04 | |
| openstackgerrit | Henry Gessau proposed openstack/requirements: Constrain eventlet to 0.18.2 for now https://review.openstack.org/281479 | 04:05 |
| *** yuanying has joined #openstack-infra | 04:05 | |
| KanagarajM | ianw, oh, ok. I have requested AJaeger yesterday and would you please suggest someone else as Ajaeger is not online. | 04:06 |
| *** shashank_hegde has joined #openstack-infra | 04:06 | |
| ianw | KanagarajM: the right people will look, it might just take a bit ... AJaeger usually comes online in a few hours from now | 04:06 |
| *** yuanying_ has joined #openstack-infra | 04:07 | |
| KanagarajM | ianw, ok. thanks :) | 04:08 |
| *** yuanying has quit IRC | 04:09 | |
| *** tphummel has joined #openstack-infra | 04:14 | |
| *** dimtruck is now known as zz_dimtruck | 04:14 | |
| *** baoli has quit IRC | 04:17 | |
| *** baoli has joined #openstack-infra | 04:17 | |
| *** sam_wan has quit IRC | 04:20 | |
| openstackgerrit | Merged openstack-infra/project-config: Set ZUUL_REF for devstack plugins list proposal periodic job https://review.openstack.org/281275 | 04:22 |
| *** tzn has joined #openstack-infra | 04:26 | |
| *** tzn has quit IRC | 04:30 | |
| *** shashank_hegde has quit IRC | 04:30 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Use pip install of dib for testing https://review.openstack.org/219949 | 04:37 |
| ianw | greghaynes: just been sitting around in my queue, but i think still relevant ^ | 04:38 |
| *** tiswanso has quit IRC | 04:39 | |
| *** max_lobur has joined #openstack-infra | 04:40 | |
| *** arxcruz has quit IRC | 04:41 | |
| *** chenli has joined #openstack-infra | 04:44 | |
| *** wolverineav has joined #openstack-infra | 04:48 | |
| *** xyang1 has joined #openstack-infra | 04:50 | |
| *** dims_ has joined #openstack-infra | 04:53 | |
| *** wolverineav has quit IRC | 04:53 | |
| *** dims has quit IRC | 04:55 | |
| *** thorst has joined #openstack-infra | 04:56 | |
| *** xyang1 has quit IRC | 04:58 | |
| *** dims_ has quit IRC | 05:00 | |
| *** esker has joined #openstack-infra | 05:01 | |
| *** baoli has quit IRC | 05:01 | |
| *** dims has joined #openstack-infra | 05:02 | |
| *** dims has quit IRC | 05:02 | |
| *** thorst has quit IRC | 05:04 | |
| *** esker has quit IRC | 05:05 | |
| *** zhurong has quit IRC | 05:05 | |
| *** roxanaghe has joined #openstack-infra | 05:09 | |
| *** tphummel has quit IRC | 05:13 | |
| *** tphummel has joined #openstack-infra | 05:15 | |
| *** wolverineav has joined #openstack-infra | 05:15 | |
| *** wolverineav has quit IRC | 05:19 | |
| *** wolverineav has joined #openstack-infra | 05:19 | |
| *** esker has joined #openstack-infra | 05:21 | |
| *** FallenPegasus has joined #openstack-infra | 05:22 | |
| *** kushal has joined #openstack-infra | 05:23 | |
| *** [1]Thelo has joined #openstack-infra | 05:23 | |
| *** Thelo has quit IRC | 05:25 | |
| *** [1]Thelo is now known as Thelo | 05:25 | |
| *** esker has quit IRC | 05:26 | |
| *** tzn has joined #openstack-infra | 05:26 | |
| *** maishsk_ has joined #openstack-infra | 05:27 | |
| openstackgerrit | Kevin Benton proposed openstack-infra/project-config: Use config drive in Neutron jobs https://review.openstack.org/281633 | 05:28 |
| *** maishsk has quit IRC | 05:28 | |
| *** maishsk_ is now known as maishsk | 05:28 | |
| *** zhurong has joined #openstack-infra | 05:29 | |
| *** tzn has quit IRC | 05:31 | |
| *** abregman has joined #openstack-infra | 05:32 | |
| *** roxanaghe has quit IRC | 05:34 | |
| *** gyee has quit IRC | 05:35 | |
| *** aeng has quit IRC | 05:38 | |
| *** maishsk has quit IRC | 05:39 | |
| *** maishsk_ has joined #openstack-infra | 05:39 | |
| *** sam_wan has joined #openstack-infra | 05:42 | |
| *** maishsk_ has quit IRC | 05:44 | |
| *** infra-red has joined #openstack-infra | 05:45 | |
| *** changzhi has quit IRC | 05:53 | |
| *** _nadya_ has joined #openstack-infra | 05:54 | |
| *** thorst has joined #openstack-infra | 06:02 | |
| *** sc68cal has quit IRC | 06:03 | |
| *** tphummel has quit IRC | 06:04 | |
| *** sridhar_ram has joined #openstack-infra | 06:05 | |
| *** thorst has quit IRC | 06:09 | |
| *** FallenPegasus has quit IRC | 06:10 | |
| *** sridhar_ram1 has joined #openstack-infra | 06:12 | |
| *** FallenPegasus has joined #openstack-infra | 06:12 | |
| *** sridhar_ram has quit IRC | 06:13 | |
| *** gildub has quit IRC | 06:13 | |
| openstackgerrit | Maksim Malchuk proposed openstack-infra/project-config: Add fuel-virtualbox project https://review.openstack.org/279074 | 06:14 |
| *** tphummel has joined #openstack-infra | 06:16 | |
| *** infra-red has quit IRC | 06:23 | |
| *** _nadya_ has quit IRC | 06:23 | |
| *** jaosorior has quit IRC | 06:23 | |
| *** esker has joined #openstack-infra | 06:24 | |
| *** jaosorior has joined #openstack-infra | 06:24 | |
| *** tzn has joined #openstack-infra | 06:27 | |
| *** esker has quit IRC | 06:28 | |
| *** sridhar_ram1 has quit IRC | 06:28 | |
| *** tzn has quit IRC | 06:32 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/281661 | 06:34 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/281073 | 06:38 |
| *** sam_wan has quit IRC | 06:41 | |
| *** fedexo has quit IRC | 06:47 | |
| openstackgerrit | Merged openstack/requirements: bump upper constraint for keystoneauth https://review.openstack.org/281022 | 06:48 |
| *** FallenPegasus has quit IRC | 06:53 | |
| *** fedexo has joined #openstack-infra | 06:54 | |
| *** infra-red has joined #openstack-infra | 07:01 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Skip rally runs on doc changes https://review.openstack.org/281665 | 07:05 |
| *** esker has joined #openstack-infra | 07:06 | |
| *** nijaba has quit IRC | 07:07 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Skip rally runs on Neutron's doc changes https://review.openstack.org/281665 | 07:07 |
| *** maishsk has joined #openstack-infra | 07:08 | |
| *** armax has quit IRC | 07:09 | |
| *** thorst has joined #openstack-infra | 07:09 | |
| *** esker has quit IRC | 07:11 | |
| *** thorst has quit IRC | 07:14 | |
| *** achanda has quit IRC | 07:14 | |
| *** achanda has joined #openstack-infra | 07:17 | |
| *** scheuran has joined #openstack-infra | 07:17 | |
| *** maishsk has quit IRC | 07:18 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Upgrade Castellan functional gate https://review.openstack.org/281464 | 07:22 |
| *** esker has joined #openstack-infra | 07:27 | |
| *** tzn has joined #openstack-infra | 07:28 | |
| *** gongysh has joined #openstack-infra | 07:29 | |
| *** camunoz has quit IRC | 07:29 | |
| openstackgerrit | Merged openstack-infra/project-config: Enable gate-tempest-dsvm-neutron-src-oslo.service for liberty https://review.openstack.org/281559 | 07:30 |
| *** ianw has quit IRC | 07:30 | |
| *** ianw has joined #openstack-infra | 07:31 | |
| *** maishsk has joined #openstack-infra | 07:31 | |
| *** esker has quit IRC | 07:31 | |
| openstackgerrit | Sergey Lukjanov proposed openstack/requirements: Update python-saahraclient to the 0.12.0 https://review.openstack.org/281673 | 07:31 |
| *** tzn has quit IRC | 07:32 | |
| openstackgerrit | Sergey Lukjanov proposed openstack/requirements: Update python-saahraclient to the 0.12.0 https://review.openstack.org/281673 | 07:33 |
| *** [HeOS] has joined #openstack-infra | 07:36 | |
| *** HeOS has quit IRC | 07:37 | |
| *** nmagnezi has joined #openstack-infra | 07:40 | |
| *** e0ne has joined #openstack-infra | 07:41 | |
| yolanda | good morning | 07:44 |
| *** salv-orlando has joined #openstack-infra | 07:44 | |
| *** Hal has joined #openstack-infra | 07:46 | |
| *** Hal is now known as Guest20178 | 07:47 | |
| openstackgerrit | Steve Martinelli proposed openstack/requirements: bump upper constraint for keystonemiddleware https://review.openstack.org/281028 | 07:47 |
| *** rcernin has joined #openstack-infra | 07:51 | |
| *** samuelBartel has quit IRC | 07:52 | |
| *** e0ne has quit IRC | 07:58 | |
| *** flepied has quit IRC | 07:58 | |
| *** kushal has quit IRC | 07:59 | |
| *** ildikov has quit IRC | 08:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Upgrade Castellan functional gate https://review.openstack.org/281464 | 08:00 |
| *** pcaruana has joined #openstack-infra | 08:05 | |
| *** arxcruz has joined #openstack-infra | 08:07 | |
| *** esker has joined #openstack-infra | 08:08 | |
| *** gongysh has quit IRC | 08:09 | |
| *** nijaba has joined #openstack-infra | 08:10 | |
| *** nijaba has joined #openstack-infra | 08:10 | |
| *** thorst has joined #openstack-infra | 08:11 | |
| *** ildikov has joined #openstack-infra | 08:12 | |
| *** esker has quit IRC | 08:13 | |
| *** jtomasek|afk is now known as jtomasek | 08:15 | |
| *** shashank_hegde has joined #openstack-infra | 08:16 | |
| *** sflanigan has quit IRC | 08:17 | |
| *** thorst has quit IRC | 08:18 | |
| *** fedexo has quit IRC | 08:20 | |
| *** mikelk has joined #openstack-infra | 08:21 | |
| *** jlanoux has joined #openstack-infra | 08:24 | |
| *** nijaba has quit IRC | 08:25 | |
| *** jlanoux has quit IRC | 08:26 | |
| *** jlanoux has joined #openstack-infra | 08:26 | |
| *** tzn has joined #openstack-infra | 08:29 | |
| *** mikelk has quit IRC | 08:30 | |
| *** mikelk has joined #openstack-infra | 08:31 | |
| *** tzn has quit IRC | 08:33 | |
| *** nijaba has joined #openstack-infra | 08:35 | |
| *** nijaba has joined #openstack-infra | 08:35 | |
| *** matrohon has joined #openstack-infra | 08:35 | |
| *** flepied has joined #openstack-infra | 08:36 | |
| *** jordanP has joined #openstack-infra | 08:39 | |
| *** max_lobur has quit IRC | 08:41 | |
| *** Jeffrey4l has quit IRC | 08:45 | |
| *** matrohon has quit IRC | 08:47 | |
| *** dingyichen has quit IRC | 08:48 | |
| *** vgridnev has joined #openstack-infra | 08:48 | |
| *** [HeOS] has quit IRC | 08:48 | |
| *** [HeOS] has joined #openstack-infra | 08:48 | |
| *** esker has joined #openstack-infra | 08:50 | |
| *** piet has joined #openstack-infra | 08:51 | |
| *** fhubik has joined #openstack-infra | 08:53 | |
| *** fhubik is now known as fhubik_brb | 08:53 | |
| *** hashar has joined #openstack-infra | 08:53 | |
| *** karlamrhein has joined #openstack-infra | 08:53 | |
| *** nijaba has quit IRC | 08:53 | |
| openstackgerrit | Aliaksandr Buhayeu proposed openstack/python-jenkins: Adds possibility to retrieve all jobs from the view specified https://review.openstack.org/279750 | 08:53 |
| *** esker has quit IRC | 08:54 | |
| *** zeih has joined #openstack-infra | 08:55 | |
| *** fhubik_brb is now known as fhubik | 08:56 | |
| *** Jeffrey4l has joined #openstack-infra | 08:58 | |
| *** nijaba has joined #openstack-infra | 08:59 | |
| *** nijaba has joined #openstack-infra | 08:59 | |
| *** karlamrhein has quit IRC | 09:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Adds new project os-namos https://review.openstack.org/280662 | 09:01 |
| *** fhubik is now known as fhubik_brb | 09:01 | |
| openstackgerrit | Merged openstack-infra/project-config: Use pip install of dib for testing https://review.openstack.org/219949 | 09:02 |
| *** AJaeger has joined #openstack-infra | 09:04 | |
| AJaeger | morning | 09:04 |
| AJaeger | yolanda: for https://review.openstack.org/#/c/281465 - kiloeyes is marked as executable, the change removes the x permissions. See also commit message. | 09:04 |
| *** tzn has joined #openstack-infra | 09:04 | |
| yolanda | oh missed that | 09:05 |
| *** rakhmerov has quit IRC | 09:05 | |
| AJaeger | no problem - thanks for reviewing | 09:05 |
| *** rakhmerov has joined #openstack-infra | 09:05 | |
| *** Qiming has quit IRC | 09:06 | |
| *** dizquierdo has joined #openstack-infra | 09:08 | |
| *** _nadya_ has joined #openstack-infra | 09:09 | |
| *** tzn has quit IRC | 09:10 | |
| *** esker has joined #openstack-infra | 09:11 | |
| *** nijaba has quit IRC | 09:13 | |
| *** fhubik_brb is now known as fhubik | 09:14 | |
| *** esker has quit IRC | 09:15 | |
| *** max_lobur has joined #openstack-infra | 09:16 | |
| *** thorst has joined #openstack-infra | 09:17 | |
| *** nijaba has joined #openstack-infra | 09:17 | |
| *** nijaba has joined #openstack-infra | 09:17 | |
| openstackgerrit | Merged openstack-infra/project-config: Cleanup some ACLs https://review.openstack.org/281465 | 09:18 |
| openstackgerrit | Merged openstack-infra/project-config: Fix tag publishing of server projects https://review.openstack.org/279238 | 09:18 |
| openstackgerrit | Merged openstack-infra/project-config: Watcher multinode Tempest job fixup https://review.openstack.org/277923 | 09:18 |
| openstackgerrit | Merged openstack-infra/project-config: Create openstack/ramdisk-func-test repository https://review.openstack.org/279120 | 09:19 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual: Document gerrit ACL for stable team https://review.openstack.org/281712 | 09:19 |
| *** ildikov has quit IRC | 09:20 | |
| AJaeger | fungi, yolanda: https://github.com/openstack/giftwrap/ is still empty. Will it only get content after the next merge? Or should it sync automatically - and fails somehow? | 09:23 |
| yolanda | AJaeger as i got the instructions from fungi, will try to fix | 09:23 |
| *** thorst has quit IRC | 09:24 | |
| *** lucas-dinner is now known as lucasagomes | 09:24 | |
| *** dtantsur|afk is now known as dtantsur | 09:25 | |
| *** achanda has quit IRC | 09:25 | |
| openstackgerrit | Merged openstack-infra/project-config: Switch most of ironic jobs to iPXE and remove a separate iPXE job https://review.openstack.org/281253 | 09:27 |
| dguitarbite | AJaeger: I need your help in figuring out cross-release tarball generation | 09:28 |
| *** achanda has joined #openstack-infra | 09:28 | |
| dguitarbite | AJaeger: https://review.openstack.org/#/c/281713/ | 09:29 |
| *** achanda has quit IRC | 09:29 | |
| dguitarbite | I need to trigger this script for all releases to build and publish tar/zip files. | 09:29 |
| *** ociuhandu has joined #openstack-infra | 09:29 | |
| *** wolverineav has quit IRC | 09:30 | |
| AJaeger | dguitarbite: normally you run a post job - and that one runs on your branch | 09:30 |
| dguitarbite | Do I create a post-job template like training-labs.yaml in jenkins/jobs and then trigger it for different branches from the projects.yaml? | 09:31 |
| yolanda | AJaeger ok , i added gerrit group to github. Going to send documentation about that, before i forget | 09:31 |
| AJaeger | That's what I did with 281712 - document fungi's great change 281563 directly.. | 09:32 |
| AJaeger | thanks, yolanda. | 09:32 |
| AJaeger | dguitarbite: you just merge content for each branch... | 09:32 |
| AJaeger | dguitarbite: I'll grab a coffee and visit your office... | 09:32 |
| dguitarbite | Ok, Ill be free in 10 minutes, in a standup | 09:33 |
| yolanda | AJaeger, yes, we have too many things that are known for some people, and is good to have it shared as a knowledge base | 09:33 |
| *** asettle has joined #openstack-infra | 09:33 | |
| yolanda | so, if i know how to fix gerrit, any infra-root shall have this knowledge and i should share it | 09:33 |
| *** flepied has quit IRC | 09:34 | |
| *** flepied has joined #openstack-infra | 09:34 | |
| *** shashank_hegde has quit IRC | 09:34 | |
| *** e0ne has joined #openstack-infra | 09:35 | |
| *** ildikov has joined #openstack-infra | 09:36 | |
| openstackgerrit | Merged openstack-infra/puppet-refstack: use specific tag version of refstack for installation. https://review.openstack.org/279511 | 09:36 |
| *** chenli has quit IRC | 09:37 | |
| AJaeger | yolanda: I really like that approach - it's the best way forward... | 09:38 |
| *** fhubik is now known as fhubik_brb | 09:38 | |
| AJaeger | yippie, https://github.com/openstack/giftwrap/ has content! | 09:38 |
| AJaeger | thanks, yolanda ! | 09:38 |
| yolanda | anytime :) | 09:38 |
| yolanda | woot! | 09:38 |
| abregman | hi everyone. my name is Arie and I would like to contribute to one (or several) of infra projects. I have experience with Python, Ansible, Jenkins and Linux (shell, internals...). Any suggestion on how to start? | 09:39 |
| openstackgerrit | Paul Van Eck proposed openstack-infra/project-config: Add pypi publish jobs to RefStack https://review.openstack.org/281720 | 09:39 |
| *** doug-fish has joined #openstack-infra | 09:40 | |
| openstackgerrit | Pranav Salunke proposed openstack-infra/project-config: Adds training-labs post jobs to publish tarballs https://review.openstack.org/281721 | 09:41 |
| dguitarbite | AJaeger: https://review.openstack.org/#/c/281721/1 | 09:41 |
| dguitarbite | AJaeger: Heading over in 5 | 09:42 |
| *** asettle has quit IRC | 09:43 | |
| *** doug-fish has quit IRC | 09:44 | |
| openstackgerrit | Merged openstack-infra/project-config: Normalize projects.yaml https://review.openstack.org/281073 | 09:45 |
| *** samuelBartel has joined #openstack-infra | 09:45 | |
| *** achanda has joined #openstack-infra | 09:46 | |
| AJaeger | wellcome, abregman ! | 09:47 |
| abregman | AJaeger: thanks :) | 09:49 |
| *** asettle has joined #openstack-infra | 09:49 | |
| *** asettle has quit IRC | 09:49 | |
| *** jlanoux has quit IRC | 09:49 | |
| *** jlanoux has joined #openstack-infra | 09:50 | |
| AJaeger | abregman: best come back during US times when most of the team is here - there's so much to do, I'm not sure where to point you first and have other tasks to do right now. | 09:50 |
| abregman | AJaeger: np! I'll try again later today | 09:50 |
| *** BobBall_AWOL is now known as BobBall | 09:51 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add troubleshooting section to github https://review.openstack.org/281726 | 09:53 |
| *** zhurong has quit IRC | 09:54 | |
| *** piet has quit IRC | 09:55 | |
| *** piet has joined #openstack-infra | 09:55 | |
| *** derekh has joined #openstack-infra | 09:56 | |
| *** fhubik_brb is now known as fhubik | 09:57 | |
| *** xiexs___ has joined #openstack-infra | 09:59 | |
| *** asettle has joined #openstack-infra | 09:59 | |
| xiexs___ | hi, all, is there a problem with check-citrix-xenserver CI? | 09:59 |
| BobBall | yes | 09:59 |
| BobBall | I'm investigating | 09:59 |
| xiexs___ | BobBall: OK, thanks. | 10:00 |
| BobBall | xiexs___: looks like a network error | 10:01 |
| BobBall | Or maybe it's not | 10:02 |
| *** ganso has joined #openstack-infra | 10:02 | |
| *** openstackgerrit has quit IRC | 10:02 | |
| *** openstackgerrit has joined #openstack-infra | 10:03 | |
| *** salv-orl_ has joined #openstack-infra | 10:03 | |
| *** asettle has quit IRC | 10:05 | |
| *** salv-orlando has quit IRC | 10:06 | |
| eantyshev | yolanda: Hello! Could you please look at this simple patch for system-config: https://review.openstack.org/239479 ? Thanks! | 10:06 |
| yolanda | hi eantyshev | 10:06 |
| yolanda | i'll do | 10:06 |
| *** __oanson has joined #openstack-infra | 10:06 | |
| *** kushal has joined #openstack-infra | 10:06 | |
| openstackgerrit | Paul Van Eck proposed openstack-infra/puppet-refstack: Make puppet pull releases from Pypi https://review.openstack.org/281737 | 10:07 |
| xiexs___ | BobBall:yes, the error msg is "sudo: ovs-vsctl: command not found" in the ci log. | 10:07 |
| *** markmcd1 has quit IRC | 10:09 | |
| *** markmcd has quit IRC | 10:09 | |
| *** _oanson has quit IRC | 10:10 | |
| BobBall | That's a different error to the one I was looking at | 10:10 |
| BobBall | Where are your log files? | 10:10 |
| yolanda | eantyshev, no time to test it now... do you have a testenv for it? can you send me some samples of the results of that tool? or if they are short enough, paste on the change | 10:10 |
| *** markmcd has joined #openstack-infra | 10:11 | |
| *** achanda has quit IRC | 10:12 | |
| *** esker has joined #openstack-infra | 10:13 | |
| *** yamahata has quit IRC | 10:14 | |
| *** achanda has joined #openstack-infra | 10:14 | |
| *** FallenPegasus has joined #openstack-infra | 10:14 | |
| eantyshev | yolanda: simple checks http://paste.openstack.org/show/487386/ | 10:16 |
| *** KanagarajM has left #openstack-infra | 10:16 | |
| *** fhubik is now known as fhubik_brb | 10:16 | |
| *** achanda has quit IRC | 10:16 | |
| *** piet has quit IRC | 10:18 | |
| *** esker has quit IRC | 10:18 | |
| xiexs___ | BobBall:http://dd6b71949550285df7dc-dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/55/281155/4/6575/logs/devstacklog.txt.gz | 10:18 |
| *** piet has joined #openstack-infra | 10:18 | |
| yolanda | eantyshev, can you add that link to the review, and then i +2? | 10:19 |
| *** FallenPegasus has quit IRC | 10:19 | |
| *** _sarob has joined #openstack-infra | 10:19 | |
| openstackgerrit | Evgeny Antyshev proposed openstack-infra/system-config: Make hieraedit.py to not shuffle data items https://review.openstack.org/239479 | 10:20 |
| yolanda | thanks | 10:20 |
| *** thorst has joined #openstack-infra | 10:22 | |
| *** _sarob has quit IRC | 10:24 | |
| *** Jeffrey4l has quit IRC | 10:25 | |
| BobBall | xiexs___: OK - you need to look for the first error - in this case it's Download error on http://pypi.dfw.openstack.org/simple/pbr/: [Errno 101] Network is unreachable -- Some packages may not be found! | 10:29 |
| *** thorst has quit IRC | 10:29 | |
| *** kushal has quit IRC | 10:29 | |
| BobBall | xiexs___: I don't know why my CI ended up using infra's pypi mirror again - I was sure I had set it to use pypi.python.org but clearly it's still trying to access it. Presumably infra's pypi mirror has changed today | 10:29 |
| BobBall | xiexs___: The issues with ovs-ofctl are simply because devstack has not completed running and when it tries to collect logs it tries to use a program that has not yet been installed. | 10:30 |
| *** tzn has joined #openstack-infra | 10:31 | |
| *** wolverineav has joined #openstack-infra | 10:31 | |
| *** fhubik_brb is now known as fhubik | 10:31 | |
| xiexs___ | BobBall: OK, got it. thanks. | 10:32 |
| *** wolverineav has quit IRC | 10:35 | |
| *** ldnunes has joined #openstack-infra | 10:35 | |
| AJaeger | BobBall: rax is doing network updates, so that explains the timeouts. But you're right, your setup is strange and should be fixed... | 10:36 |
| *** tpsilva has joined #openstack-infra | 10:36 | |
| *** kushal has joined #openstack-infra | 10:36 | |
| *** aarefiev has quit IRC | 10:37 | |
| *** piet has quit IRC | 10:38 | |
| *** piet has joined #openstack-infra | 10:39 | |
| *** gnuoy_ has joined #openstack-infra | 10:39 | |
| *** gnuoy_ has quit IRC | 10:41 | |
| *** doug-fish has joined #openstack-infra | 10:41 | |
| *** daemontool__ has quit IRC | 10:41 | |
| *** daemontool has joined #openstack-infra | 10:41 | |
| *** fhubik is now known as fhubik_brb | 10:41 | |
| *** Jeffrey4l has joined #openstack-infra | 10:42 | |
| *** daemontool has quit IRC | 10:43 | |
| *** daemontool_ has joined #openstack-infra | 10:43 | |
| *** hichihara has quit IRC | 10:44 | |
| *** piet has quit IRC | 10:44 | |
| *** piet has joined #openstack-infra | 10:45 | |
| *** doug-fish has quit IRC | 10:46 | |
| *** rfolco has joined #openstack-infra | 10:46 | |
| *** dtantsur is now known as dtantsur|brb | 10:46 | |
| BobBall | I'm confused though... Do you know how this pypi mirror is set up? I assumed through nodepool-scripts/configure_mirror.sh but that only gets called from multinode_setup.sh which isn't called in my setup | 10:48 |
| *** sdague has joined #openstack-infra | 10:49 | |
| *** daemontool has joined #openstack-infra | 10:49 | |
| *** daemontool_ has quit IRC | 10:50 | |
| BobBall | More confused - /etc/pip.conf has index-url set to pypi.python.org (although it does have all of the infra mirrors listed as trusted-host's); so how is it picking up an infra mirror? | 10:51 |
| *** cdent has joined #openstack-infra | 10:52 | |
| *** electrofelix has joined #openstack-infra | 10:52 | |
| openstackgerrit | xiangxinyong proposed openstack-infra/project-config: Add dashboard and client for Smaug https://review.openstack.org/281019 | 10:53 |
| AJaeger | BobBall: did you log into a running system and check it? | 10:53 |
| *** jed56 has joined #openstack-infra | 10:54 | |
| BobBall | Yes | 10:54 |
| AJaeger | BobBall: I hope jeblair or mordred can help you with that | 10:55 |
| BobBall | Well, I say yes, but I mean actually maybe. I made a code change that I thought would work then realised it _really_ wouldn't fix it... But I suppose it's vaguely possible that it did fix it somehow | 10:56 |
| *** rguillebert has joined #openstack-infra | 10:56 | |
| BobBall | OK - I can say yes now :D | 10:56 |
| openstackgerrit | Alexander Evseev proposed openstack-infra/zuul: Add possibility to use SSH instead of HTTP to get ref list https://review.openstack.org/246433 | 10:57 |
| *** rossella_s has quit IRC | 10:57 | |
| *** rossella_s has joined #openstack-infra | 10:58 | |
| *** dims has joined #openstack-infra | 10:59 | |
| *** akuznetsova has left #openstack-infra | 10:59 | |
| *** esker has joined #openstack-infra | 11:00 | |
| *** marcusvrn_ has joined #openstack-infra | 11:00 | |
| xiexs___ | BobBall: so the problem is solved now? | 11:01 |
| *** piet has quit IRC | 11:02 | |
| *** yamamoto has quit IRC | 11:03 | |
| *** daemontool has quit IRC | 11:04 | |
| *** daemontool has joined #openstack-infra | 11:04 | |
| *** piet has joined #openstack-infra | 11:04 | |
| *** esker has quit IRC | 11:05 | |
| BobBall | Nope - really not. It's very confusing to me. | 11:07 |
| *** ihrachys has joined #openstack-infra | 11:08 | |
| mordred | BobBall: configure_mirror.sh also gets called in our nodepool ready scripts - not just for multinode | 11:08 |
| *** aysyd has joined #openstack-infra | 11:08 | |
| mordred | BobBall: nodepool/nodepool.yaml: ready-script: configure_mirror.sh | 11:09 |
| mordred | dunno if you've got that going too | 11:09 |
| BobBall | Nope :( nothing set for ready-script | 11:10 |
| *** vgridnev has quit IRC | 11:11 | |
| *** aarefiev has joined #openstack-infra | 11:11 | |
| BobBall | Also, for giggles, I just ran devstack in a node that failed due to the infra pypi mirror | 11:11 |
| BobBall | Guess what. | 11:11 |
| BobBall | It's got a long way passed the failure point | 11:11 |
| BobBall | So I'm thinking it's environment somewhere rather than a config file | 11:12 |
| *** jcoufal has joined #openstack-infra | 11:15 | |
| mordred | fascinating. | 11:15 |
| *** asettle has joined #openstack-infra | 11:15 | |
| *** esker has joined #openstack-infra | 11:15 | |
| *** jlanoux has quit IRC | 11:15 | |
| *** jlanoux has joined #openstack-infra | 11:15 | |
| *** ihrachys has quit IRC | 11:16 | |
| BobBall | OK - so it got a lot further, but failed with the same error trying to install pbr>=1.8 in the tempest venv (which is good I suppose) | 11:16 |
| *** ihrachys has joined #openstack-infra | 11:16 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Add ability to configure nova quota https://review.openstack.org/281770 | 11:17 |
| *** achanda has joined #openstack-infra | 11:17 | |
| wznoinsk | hi infra | 11:17 |
| *** jkraj has joined #openstack-infra | 11:18 | |
| *** piet has quit IRC | 11:19 | |
| *** Jeffrey4l has quit IRC | 11:19 | |
| BobBall | Does having the -infra mirrors listed as trusted-host's in pip.conf cause this problem? or is there likely to be another reference to the mirror elsewhere? | 11:19 |
| *** esker has quit IRC | 11:20 | |
| wznoinsk | would I need a full blown openstack (controller + compute services) as slaves for nodepool/jenkins when using downstream-puppet package? | 11:21 |
| *** vgridnev has joined #openstack-infra | 11:21 | |
| *** ociuhandu has quit IRC | 11:22 | |
| xiexs___ | BobBall: but i think the mirror is not the root reason, as you can refer to another devstack log which also used the infra mirror but it works fine, although there was still the same problem. | 11:22 |
| xiexs___ | BobBall: you can check this: http://dd6b71949550285df7dc-dda4e480e005aaa13ec303551d2d8155.r49.cf1.rackcdn.com/52/281152/1/6494/logs/devstacklog.txt.gz | 11:23 |
| BobBall | Found the broken config - it's coming from ~jenkins/.pydistutils.cfg copied by devstack-gate/functions | 11:23 |
| *** achanda has quit IRC | 11:24 | |
| *** vgridnev has quit IRC | 11:24 | |
| BobBall | xiexs___: It's definitely the pypi mirror as the root reason (although possibly because the network has broken and/or ifra have moved their pypi mirror) | 11:24 |
| mordred | BobBall: what is it configured to hit? | 11:24 |
| BobBall | http://pypi.dfw.openstack.org/simple | 11:24 |
| mordred | that is not a thing anymore | 11:25 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add the ability to configure quota using hiera https://review.openstack.org/281773 | 11:25 |
| BobBall | Indeed - now I'm trying to figure out _how_ it got configured to hit that in my environment | 11:25 |
| *** kzaitsev_mb has joined #openstack-infra | 11:25 | |
| *** piet has joined #openstack-infra | 11:25 | |
| *** erikm has quit IRC | 11:25 | |
| *** vgridnev has joined #openstack-infra | 11:26 | |
| mordred | BobBall: if you're re-using any of our puppet, modules/openstack_project/manifests/slave_common.pp installs that file | 11:26 |
| mordred | and it has modules/openstack_project/files/pydistutils.cfg:index_url = http://pypi.dfw.openstack.org/simple | 11:26 |
| mordred | (or, at least it did - I'm pulling to doublecheck) | 11:26 |
| *** thorst has joined #openstack-infra | 11:27 | |
| BobBall | ahhhhhh! Yay! That'll be it! | 11:27 |
| BobBall | I think I'm re-using your puppet yes | 11:27 |
| *** vgridnev has quit IRC | 11:28 | |
| *** vgridnev has joined #openstack-infra | 11:28 | |
| mordred | BobBall: current file has https://github.com/openstack-infra/system-config/blob/master/modules/openstack_project/files/pydistutils.cfg | 11:29 |
| mordred | BobBall: so an update should get you fixed | 11:29 |
| *** Jeffrey4l has joined #openstack-infra | 11:29 | |
| BobBall | awesome - found that | 11:29 |
| BobBall | many thanks mordred | 11:29 |
| mordred | sure nuff! | 11:29 |
| BobBall | but I don't want to be using your pypi mirror for that either - I should be using upstream pypi for my ci | 11:29 |
| BobBall | I'll modify my system-config to fix the file to use the primary pypi source | 11:30 |
| * BobBall gives mordred 10 house points for the help | 11:30 | |
| *** dims_ has joined #openstack-infra | 11:30 | |
| *** dims has quit IRC | 11:31 | |
| *** ihrachys has quit IRC | 11:32 | |
| *** |-paul-| has joined #openstack-infra | 11:32 | |
| BobBall | Awesome :( Now I've got to fight against the image regeneration having not worked for over a month without me noticing! :) | 11:34 |
| *** thorst has quit IRC | 11:34 | |
| mordred | BobBall: that sounds like fun! | 11:34 |
| BobBall | You like fun, don't you mordred. I've got 10 more house points up for grabs if you want to have some fun. | 11:35 |
| mordred | hehehe | 11:35 |
| mordred | _I_ am trying to figure out why the image I built of xenial is not pingable when I boot it in a cloud | 11:36 |
| mordred | as a sort of "start getting ready for the new LTS" kind of fun | 11:36 |
| mordred | and to make it better - I'm doing it over conference wifi | 11:36 |
| BobBall | Maybe you can blame RAX's network updates? | 11:36 |
| mordred | I'd like to | 11:39 |
| mordred | but it's running in vexxhost and my other machine is up ... | 11:39 |
| mordred | b | 11:39 |
| mordred | so I think I ahve to blame me | 11:39 |
| mordred | greghaynes: [[0;1;31mFAILED[0m] Failed to start DHCP interface ens3. | 11:40 |
| mordred | See 'systemctl status glean@ens3.service' for details. | 11:40 |
| *** Wei_Liu has joined #openstack-infra | 11:41 | |
| BobBall | Uh oh | 11:41 |
| *** dtantsur|brb is now known as dtantsur | 11:41 | |
| mordred | greghaynes, yolanda: ^^ I get that when I try to boot an image of xenial I made with DIB_RELEASE=xenial disk-image-create -n -x --no-tmpfs -o ubuntu-xenial ubuntu-minimal vm simple-init | 11:41 |
| yolanda | mordred, do you have more details when you run systemctl? | 11:42 |
| mordred | yolanda: well - I can't log in because dhcp isn't working :) | 11:42 |
| *** cdent_ has joined #openstack-infra | 11:42 | |
| yolanda | that should be on a chroot? | 11:43 |
| mordred | yolanda: I got that info from the nova console-log | 11:43 |
| yolanda | mm, can you try adding devuser? | 11:43 |
| mordred | yolanda: I think I need to make a new image with a user that has a password so that I can log in via web console | 11:43 |
| mordred | yeah | 11:43 |
| yolanda | so you can enter from console? | 11:43 |
| yolanda | you typed faster :) | 11:43 |
| mordred | well, I could if I had a user with a password | 11:43 |
| mordred | I type fast :) | 11:44 |
| *** asettle has quit IRC | 11:45 | |
| *** cdent has quit IRC | 11:46 | |
| *** cdent_ is now known as cdent | 11:46 | |
| *** binbincong has quit IRC | 11:46 | |
| *** yamamoto has joined #openstack-infra | 11:48 | |
| mordred | yolanda: btw - if you ever want to play with building xenial images before they're released, you can do: | 11:49 |
| mordred | sudo ln -s /usr/share/debootstrap/scripts/gutsy /usr/share/debootstrap/scripts/xenial | 11:49 |
| mordred | yolanda: which will let debootstrap know how to build them | 11:49 |
| *** fhubik_brb is now known as fhubik | 11:49 | |
| *** yamamoto_ has joined #openstack-infra | 11:49 | |
| *** Jeffrey4l__ has joined #openstack-infra | 11:50 | |
| yolanda | mordred, it might be interesting. Why are you using xenial images for? are we going to run tests in our ci? | 11:50 |
| mordred | yolanda: it's the next LTS - so I've been looking to see if there are any changes we need to make to be able to build xenial images | 11:51 |
| mordred | yolanda: once it's out I expect we'll want to move the gate jobs to it from trusty | 11:51 |
| mordred | which will, of course, be fun | 11:51 |
| yolanda | i'm sure there will be some failures. Nothing is always as easy | 11:51 |
| mordred | nope | 11:51 |
| mordred | I have decided that instead of building the image on my laptop and uploading it to the cloud over conference wifi | 11:52 |
| mordred | I will build the image on a server in the cloud and upload it to the cloud from there :) | 11:52 |
| *** Jeffrey4l has quit IRC | 11:52 | |
| yolanda | it wll be faster, conference wifis are never good | 11:53 |
| Wei_Liu | hi, I want to build mu own puppetmaster, but got error "Error: Must pass jenkins_api_key to Class[Openstack_project::Puppetmaster] at line 1 on node puppet-master-lw" when I run "puppet apply --modulepath='/opt/system-config/production/modules:/etc/puppet/modules' -e 'include openstack_project::puppetmaster'", what should I do? | 11:53 |
| yolanda | how is ansible conf going? | 11:53 |
| *** yamamoto has quit IRC | 11:53 | |
| mordred | yolanda: it's going well - the contributor summit thing yesterday was very nice | 11:54 |
| yolanda | i would have liked to go, but as mid-cycle is next week, is too much time outside home | 11:55 |
| *** __oanson is now known as oanson | 11:55 | |
| mordred | yah. I wish there were more time in between | 11:55 |
| *** Qiming has joined #openstack-infra | 11:56 | |
| *** yamamoto_ has quit IRC | 11:57 | |
| yolanda | mordred so xenial will be on systemd right? how is glean support for that, is fully functional? | 11:59 |
| mordred | yah - we support it already because centos7 and fedora | 12:00 |
| yolanda | we had several problems in the way networks are configured in upstart, and we changed the event. Maybe you are hitting something similar on systemd? | 12:00 |
| *** piet has quit IRC | 12:00 | |
| mordred | possibly | 12:00 |
| openstackgerrit | Andrey Pavlov proposed openstack-infra/project-config: Add pypi jobs to refstack. https://review.openstack.org/281793 | 12:01 |
| yolanda | remember that on upstart glean was spinning "on device-up" and we changed to "on networking start" | 12:01 |
| *** piet has joined #openstack-infra | 12:02 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Add ability to configure nova quota https://review.openstack.org/281770 | 12:03 |
| *** salv-orl_ has quit IRC | 12:03 | |
| *** BobBall has quit IRC | 12:05 | |
| *** EricGonczer_ has joined #openstack-infra | 12:06 | |
| sdague | can someone with uber gerrit powers add me to - https://review.openstack.org/#/admin/groups/1289,members | 12:07 |
| *** rhallisey has joined #openstack-infra | 12:10 | |
| *** yamamoto has joined #openstack-infra | 12:11 | |
| *** erlon has joined #openstack-infra | 12:11 | |
| igorbelikov | mordred: I had a similar issue trying to get debian jessie running, the cause was missing ifupdown package, glean tried to use /sbin/ifup and failed | 12:12 |
| igorbelikov | mordred: checked xenial - looks like ifupdown is not included in minimal image, so may be the same | 12:13 |
| *** piet has quit IRC | 12:15 | |
| *** piet has joined #openstack-infra | 12:15 | |
| openstackgerrit | Dmitriy Ukhlov proposed openstack-infra/project-config: Fixes neutron template for oslo.messaging drivers test https://review.openstack.org/281375 | 12:16 |
| *** oanson has quit IRC | 12:16 | |
| *** piet has quit IRC | 12:17 | |
| *** piet has joined #openstack-infra | 12:18 | |
| *** tinyhippo has joined #openstack-infra | 12:19 | |
| mordred | pro-tip - when testing glean on xenial - make sure you add the simple-init element so that glean is installed at all | 12:20 |
| mordred | THEN it will be easier to test good suggestions from people | 12:20 |
| *** dizquierdo is now known as dizquierdo_afk | 12:21 | |
| *** yamamoto has quit IRC | 12:22 | |
| mordred | sdague: done | 12:22 |
| *** yamamoto has joined #openstack-infra | 12:23 | |
| *** BobBall has joined #openstack-infra | 12:23 | |
| *** ociuhandu has joined #openstack-infra | 12:23 | |
| sdague | mordred: thanks | 12:23 |
| yolanda | mordred, but you showed me simple-init in the list of elements | 12:26 |
| *** salv-orlando has joined #openstack-infra | 12:26 | |
| yolanda | what were you missing? | 12:26 |
| *** fhubik is now known as fhubik_brb | 12:27 | |
| mordred | yolanda: I somehow pasted it poorly :) | 12:27 |
| *** yamamoto has quit IRC | 12:27 | |
| *** fhubik_brb is now known as fhubik | 12:28 | |
| *** oanson has joined #openstack-infra | 12:28 | |
| yolanda | mordred so it works fine now? | 12:32 |
| mordred | well - rebuilding the image now | 12:32 |
| *** _oanson has joined #openstack-infra | 12:33 | |
| *** _amrith_ is now known as amrith | 12:33 | |
| openstackgerrit | Victor Stinner proposed openstack/requirements: Bump oslo.utils to >= 3.5 to get to_utf8() https://review.openstack.org/281807 | 12:34 |
| *** piet has quit IRC | 12:35 | |
| *** Daisy has joined #openstack-infra | 12:36 | |
| mordred | oh good | 12:36 |
| mordred | yolanda: /sys/class/net changed | 12:36 |
| yolanda | is no longer sys/class/net? | 12:37 |
| *** oanson has quit IRC | 12:37 | |
| *** daemontool has quit IRC | 12:37 | |
| *** __oanson has joined #openstack-infra | 12:39 | |
| mordred | yolanda: no - I have a thing trying to access /sys/class/net/en3 - but the thing that's there is /sys/class/net/ens3 | 12:42 |
| mordred | so something is getting the wrong list of interface names | 12:42 |
| *** _oanson has quit IRC | 12:42 | |
| yolanda | mordred, what event triggers glean for systemd? | 12:42 |
| yolanda | is that at some point where all the interfaces are up? | 12:43 |
| mordred | After=network.service network.target | 12:43 |
| mordred | ugh | 12:43 |
| mordred | we have assumptions in our stuff that if the system is systemd it's redhat based | 12:44 |
| mordred | this is going to be more work than I'd hoped | 12:44 |
| *** thorst has joined #openstack-infra | 12:45 | |
| *** yamamoto has joined #openstack-infra | 12:46 | |
| *** _sarob has joined #openstack-infra | 12:46 | |
| *** esikachev_afk has joined #openstack-infra | 12:50 | |
| *** _sarob has quit IRC | 12:51 | |
| *** doug-fish has joined #openstack-infra | 12:54 | |
| *** |-paul-| has quit IRC | 12:56 | |
| *** rfolco has quit IRC | 12:57 | |
| *** julim has joined #openstack-infra | 12:57 | |
| *** rlandy has joined #openstack-infra | 12:57 | |
| *** rfolco has joined #openstack-infra | 12:57 | |
| *** esikachev_afk is now known as esikachev | 12:58 | |
| mordred | igorbelikov: yes - I have also found that ifupdown is missing (thanks) | 12:58 |
| mordred | and also openssh-server | 12:58 |
| *** asettle has joined #openstack-infra | 12:58 | |
| *** bogdando has joined #openstack-infra | 12:59 | |
| yolanda | once that we have working with upstart, it changes.. so what is the issue that you are seeing? | 13:00 |
| *** matrohon has joined #openstack-infra | 13:00 | |
| yolanda | i ended adding several debug messages so i could get a complete idea of the execution flow, in the case of upstart, the problems were the races | 13:00 |
| bogdando | which is the place I can submit a patch to the mailman matching patterns? For Fuel it seems matching only ones with a capitalized F, [Fuel]. And tags like [fuel] go to nowhere, even tho I enabled the mailman for Fuel... | 13:01 |
| *** Wei_Liu has quit IRC | 13:01 | |
| *** dizquierdo_afk is now known as dizquierdo | 13:02 | |
| *** fhubik is now known as fhubik_brb | 13:02 | |
| * esikachev is now away: Away from keyboard | 13:02 | |
| *** esikachev is now known as esikachev_afk | 13:02 | |
| *** _oanson has joined #openstack-infra | 13:03 | |
| *** asettle has quit IRC | 13:03 | |
| igorbelikov | mordred: actually had similar stuff with /sys/class/net also, glean was trying to access eth.0 instead of eth0, but this somehow vanished with ifupdown and dhcp client installation, not sure where's the connection here though | 13:04 |
| mordred | igorbelikov: good to know | 13:04 |
| mordred | igorbelikov: I'm trying again with ifupdown installed | 13:04 |
| mordred | yolanda: yah - I'm going to send up apatch that doesn't hard-code redhat thing sin the systemd service file | 13:05 |
| *** fhubik_brb is now known as fhubik | 13:05 | |
| mordred | yolanda: I believe the timing is correct - it's just trying to access non-existent things right now | 13:05 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Changed display type from touch to native https://review.openstack.org/281828 | 13:06 |
| *** __oanson has quit IRC | 13:06 | |
| *** gordc has joined #openstack-infra | 13:12 | |
| *** e0ne has quit IRC | 13:13 | |
| *** esikachev_afk is now known as esikachev | 13:14 | |
| *** e0ne has joined #openstack-infra | 13:14 | |
| mordred | igorbelikov, yolanda: WOOT. adding ifupdown fixed it | 13:14 |
| igorbelikov | mordred: yep, strange | 13:14 |
| tinyhippo | mordred: am I right in thinking that you'd be the person who is looking after the community side of openstack? as far as the relationship with freenode is concerned? | 13:14 |
| igorbelikov | but glad it worked:) | 13:14 |
| mordred | me too | 13:14 |
| openstackgerrit | Thomas Bechtold proposed openstack-infra/irc-meetings: Add rpm-packaging team meeting https://review.openstack.org/279950 | 13:14 |
| mordred | tinyhippo: jeblair and fungi are actually the people who hold on to that | 13:15 |
| mordred | tinyhippo: but I can attempt to help? | 13:15 |
| mordred | tinyhippo: we aren't doing evil things to you at the moment, are we? | 13:15 |
| tinyhippo | mordred: I just wanted to pop in and say hi, and ask whether there's anything I could help with at all, but perhaps its worth a conversation another time, if you're busy? | 13:16 |
| *** kgiusti has joined #openstack-infra | 13:16 | |
| mordred | tinyhippo: I'm totally not busy - but I'm pretty sure fungi and jeblair would also love to chat with you - fungi usually wakes up close to now (US East) - jeblair should be up in about 2.75 hours | 13:17 |
| mordred | tinyhippo: and hi! | 13:17 |
| *** baoli has joined #openstack-infra | 13:17 | |
| tinyhippo | \o/ | 13:17 |
| *** kgiusti has left #openstack-infra | 13:17 | |
| *** zhurong has joined #openstack-infra | 13:18 | |
| *** baoli_ has joined #openstack-infra | 13:18 | |
| openstackgerrit | Merged openstack-infra/openstackid: Changed display type from touch to native https://review.openstack.org/281828 | 13:18 |
| mordred | wow. that's a fascinating commit message for an openid server | 13:19 |
| Shrews | tinyhippo ... that's a great nick. it's also a worst cat, which is mordred's most favorite thing | 13:19 |
| openstackgerrit | Monty Taylor proposed openstack/diskimage-builder: Depend on ifupdown in simple-init https://review.openstack.org/281843 | 13:21 |
| mordred | Shrews: this is a true statement you make | 13:21 |
| mordred | greghaynes, SpamapS: ^^ that makes simple-init work on xenial for me | 13:21 |
| *** EricGonczer_ has quit IRC | 13:21 | |
| *** baoli has quit IRC | 13:22 | |
| tinyhippo | Shrews: yours isn't too bad either! | 13:22 |
| *** achanda has joined #openstack-infra | 13:22 | |
| *** lucasagomes is now known as lucas-hungry | 13:22 | |
| *** tiswanso has joined #openstack-infra | 13:26 | |
| *** tiswanso has quit IRC | 13:26 | |
| *** andymaier has joined #openstack-infra | 13:26 | |
| *** tiswanso has joined #openstack-infra | 13:27 | |
| *** achanda has quit IRC | 13:27 | |
| yolanda | mordred, nice | 13:28 |
| *** kgiusti has joined #openstack-infra | 13:28 | |
| kragniz | mordred: tinyhippo: there was talk a while back (maybe at the tokyo summit?) of giving out freenode openstack/ cloaks again | 13:29 |
| kragniz | is that something worth pursuing? | 13:29 |
| *** Qiming has quit IRC | 13:30 | |
| *** amrith is now known as _amrith_ | 13:31 | |
| *** Qiming has joined #openstack-infra | 13:32 | |
| *** Qiming has quit IRC | 13:32 | |
| openstackgerrit | Monty Taylor proposed openstack/requirements: Update constraint for os-client-config https://review.openstack.org/281852 | 13:33 |
| *** Qiming has joined #openstack-infra | 13:34 | |
| tinyhippo | kragniz: mordred: I/staff would be happy to assign them if openstack GCs wanted us to set them :) | 13:36 |
| *** Liuqing has joined #openstack-infra | 13:38 | |
| *** edmondsw has joined #openstack-infra | 13:39 | |
| *** thiagop has joined #openstack-infra | 13:40 | |
| *** sdake has joined #openstack-infra | 13:41 | |
| *** EricGonczer_ has joined #openstack-infra | 13:41 | |
| *** edmondsw has quit IRC | 13:41 | |
| *** edmondsw has joined #openstack-infra | 13:42 | |
| thiagop | Hey ya, good day! I have a very quick question: is there any change I can get nodepool configuring physical machines (or KVM vm's) as nodes? | 13:42 |
| *** yamamoto has quit IRC | 13:42 | |
| mordred | thiagop: at the moment the answer for that would be to have a nova with an ironic driver | 13:44 |
| mordred | thiagop: we have plans to support non-dynamic resources with zuul v3 - but those are a few months out from being real | 13:44 |
| *** yamamoto has joined #openstack-infra | 13:45 | |
| thiagop | Hey mordred, just to put some context into it, we're trying to make a sane deployment for our 3rd party CI for Ironic. What I'm trying to do is to use the scripts of the example to deploy our CI, but it need to be in the same network as the baremetal nodes (it can't be on our cloud then) | 13:46 |
| *** yamamoto has quit IRC | 13:47 | |
| *** yamamoto has joined #openstack-infra | 13:47 | |
| *** yamamoto has quit IRC | 13:47 | |
| mordred | thiagop: yah - that makes sense. So I think what you really want is the stuff we're working on for zuul v3 - which is good, because there is a plan and we're working on it - and bad because it's not ready yet | 13:47 |
| *** krtaylor has joined #openstack-infra | 13:47 | |
| *** xyang1 has joined #openstack-infra | 13:47 | |
| thiagop | mordred: makes sense. How can I get acquainted with what you're doing? | 13:48 |
| *** piet has joined #openstack-infra | 13:48 | |
| mordred | thiagop: I'll point you at some things in just a bit - I've gotta run real quick, back in a few | 13:49 |
| *** yamamoto has joined #openstack-infra | 13:49 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Create ansible-role-ssh https://review.openstack.org/281867 | 13:49 |
| thiagop | mordred: np, I can wait. TY. | 13:49 |
| fungi | bogdando: those are configured manually by the admins of each individual list. if you're talking about the openstack-dev ml you need ttx to adjust that (though i believe he's travelling at the moment so you may want to check back next week) | 13:53 |
| fungi | abregman: welcome and thanks for your interest! probably the best place to start is to begin reviewing proposed changes to the various openstack-infra repositories in our gerrit. it's the best way to get a feel for what we're running and how it's maintained. once you're comfortable with that, the next easiest step is usually to try submitting a patch for some small bug, then work your way up to more | 13:55 |
| fungi | complex tasks from there | 13:55 |
| bogdando | fungi, thank you! | 13:56 |
| abregman | fungi: thanks fungi for the answer! is there a good way to query small bugs? | 13:56 |
| *** reed_ has joined #openstack-infra | 13:57 | |
| fungi | abregman: usually easier to spot them yourself. see something which needs fixing and tackle it. as you review you'll start to get a feel for things that are left undone for later cleanup, based on comments and the like | 13:57 |
| *** dkranz has joined #openstack-infra | 13:57 | |
| *** amitgandhinz has joined #openstack-infra | 13:58 | |
| *** piet has quit IRC | 13:58 | |
| *** dizquierdo has quit IRC | 13:58 | |
| yolanda | hi fungi, i sent a change to document how to troubleshoot github https://review.openstack.org/281726 | 13:58 |
| fungi | tinyhippo: so far jeblair has been our primary liaison to freenode staff, but i'm happy to start whatever discussions you'd like. i don't know that we have any significant challenges with freenode at the moment. weathering the usual netsplits/outages/ddos impact, random spammers wandering into channels, trolls posting offensive message floods, et cetera but nothing out of the ordinary for irc ;) | 13:58 |
| yolanda | i abandoned the one from git, to properly rework , and talk about how to troubleshoot ssh problems on mirrors generally | 13:59 |
| fungi | yolanda: thanks! taking a look now | 13:59 |
| *** zhurong has quit IRC | 13:59 | |
| yolanda | issue with giftwrap project is solved following your guidelines | 13:59 |
| *** yamamoto has quit IRC | 13:59 | |
| fungi | yolanda: yep, saw that in scrollback. thanks for fixing! | 13:59 |
| abregman | fungi: k I'll try that. thanks again :) | 14:01 |
| *** Liuqing has quit IRC | 14:02 | |
| fungi | yolanda: looks great. i left a suggestion there to also describe the outward-facing symptom if you want to improve it further | 14:02 |
| yolanda | sure | 14:02 |
| *** dtantsur is now known as dtantsur|brb | 14:02 | |
| *** ddieterly has joined #openstack-infra | 14:04 | |
| abregman | k, I've started with something very small: https://review.openstack.org/#/c/281806 | 14:07 |
| *** [HeOS] has quit IRC | 14:08 | |
| *** gsagie has joined #openstack-infra | 14:09 | |
| *** EricGonczer_ has quit IRC | 14:09 | |
| fungi | abregman: wow, the original wording there was pretty unparseable | 14:10 |
| abregman | fungi: yeah :) | 14:11 |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add troubleshooting section to github https://review.openstack.org/281726 | 14:11 |
| *** HeOS has joined #openstack-infra | 14:11 | |
| *** vgridnev has quit IRC | 14:11 | |
| *** vgridnev has joined #openstack-infra | 14:12 | |
| *** ddieterly has quit IRC | 14:12 | |
| *** ddieterly has joined #openstack-infra | 14:13 | |
| openstackgerrit | Victor Morales proposed openstack/requirements: Add schemadisplay-sphinx to test requirements https://review.openstack.org/281880 | 14:14 |
| *** ddieterl_ has joined #openstack-infra | 14:15 | |
| *** ddieterl_ has quit IRC | 14:17 | |
| *** ddieterl_ has joined #openstack-infra | 14:17 | |
| yolanda | fungi, also in terms of documentation , i have this patch waiting for a long time https://review.openstack.org/265359 | 14:18 |
| *** ddieterly has quit IRC | 14:18 | |
| openstackgerrit | Pranav Salunke proposed openstack-infra/project-config: Adds training-labs post jobs to publish tarballs https://review.openstack.org/281721 | 14:19 |
| *** _nadya_ has quit IRC | 14:19 | |
| openstackgerrit | Pranav Salunke proposed openstack-infra/project-config: Adds training-labs post jobs to publish tarballs https://review.openstack.org/281721 | 14:21 |
| fungi | yolanda: oh, right, i remember now when we discussed that--will give it a readthrough in a bit | 14:21 |
| *** weshay has joined #openstack-infra | 14:23 | |
| *** jaosorior has quit IRC | 14:24 | |
| *** jaosorior has joined #openstack-infra | 14:25 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/281661 | 14:26 |
| *** mriedem has joined #openstack-infra | 14:26 | |
| *** lucas-hungry is now known as lucasagomes | 14:28 | |
| *** jsavak has joined #openstack-infra | 14:31 | |
| *** zeih has quit IRC | 14:32 | |
| *** _nadya_ has joined #openstack-infra | 14:33 | |
| *** eharney has quit IRC | 14:33 | |
| *** mtanino has joined #openstack-infra | 14:35 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Add infracloud east servers to puppetmaster https://review.openstack.org/281889 | 14:35 |
| *** zeih has joined #openstack-infra | 14:36 | |
| *** jkraj has quit IRC | 14:36 | |
| *** jkraj has joined #openstack-infra | 14:37 | |
| *** vgridnev has quit IRC | 14:38 | |
| *** NobodyCa1 has joined #openstack-infra | 14:39 | |
| *** jtomasek has quit IRC | 14:39 | |
| *** esp has joined #openstack-infra | 14:39 | |
| *** deva_ has joined #openstack-infra | 14:40 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard: Change MySQL search mode to 'boolean' https://review.openstack.org/281890 | 14:40 |
| *** sabeen1 has joined #openstack-infra | 14:40 | |
| *** zeih has quit IRC | 14:41 | |
| *** jsavak has quit IRC | 14:42 | |
| *** fhubik is now known as fhubik_brb | 14:43 | |
| *** _amrith_ is now known as amrith | 14:43 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Instruct puppet to configure settings before applying https://review.openstack.org/281892 | 14:43 |
| *** vgridnev has joined #openstack-infra | 14:43 | |
| *** jsavak has joined #openstack-infra | 14:44 | |
| *** vgridnev has quit IRC | 14:44 | |
| *** jtomasek has joined #openstack-infra | 14:44 | |
| *** pradk has joined #openstack-infra | 14:44 | |
| *** gongysh has joined #openstack-infra | 14:45 | |
| *** deva_ has quit IRC | 14:46 | |
| *** fhubik_brb is now known as fhubik | 14:46 | |
| *** NobodyCa1 has quit IRC | 14:46 | |
| *** esp has quit IRC | 14:46 | |
| *** ddieterl_ is now known as ddieterly | 14:47 | |
| *** piet has joined #openstack-infra | 14:47 | |
| *** rajinir has joined #openstack-infra | 14:50 | |
| *** eharney has joined #openstack-infra | 14:50 | |
| *** vgridnev has joined #openstack-infra | 14:50 | |
| *** piet has quit IRC | 14:52 | |
| *** dtantsur|brb is now known as dtantsur | 14:53 | |
| pabelanger | fungi: I have some time to test some of the new bindep experimental jobs. Starting with cinder | 14:54 |
| pabelanger | review 281893 | 14:54 |
| smcginnis | pabelanger: Jeremy beat you to it: https://review.openstack.org/#/c/281388/ | 14:55 |
| trash | SpamapS: Can you have another look at my https://review.openstack.org/#/c/280178/7 change? | 14:55 |
| pabelanger | smcginnis: doh! I did a quick search and didn't see anything | 14:56 |
| pabelanger | thanks for the heads up | 14:56 |
| smcginnis | pabelanger: np | 14:56 |
| *** jaosorior has quit IRC | 14:58 | |
| *** jaosorior has joined #openstack-infra | 14:59 | |
| *** yamamoto has joined #openstack-infra | 14:59 | |
| dansmith | several of us have had various failing pep8 and docs jobs, which show up as failed and yield no logs, like this: http://logs.openstack.org/42/279242/3/check/gate-nova-pep8/b804854/ | 15:01 |
| *** larainema has quit IRC | 15:01 | |
| *** rbrndt_ has joined #openstack-infra | 15:01 | |
| *** tzn has quit IRC | 15:01 | |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Adding integration test support for app catalog horizon plugin https://review.openstack.org/276440 | 15:01 |
| fungi | pabelanger: i uploaded more than 40 changes to the projects/branches where i added those jobs and left check experimental comments on all of them. look for topic:bindep-testing | 15:02 |
| pabelanger | fungi: Ya, see them all now | 15:02 |
| *** larainema has joined #openstack-infra | 15:03 | |
| fungi | i haven't had time to start looking at the job logs for them yet, was on my agenda for today, if you want to tackle looking through some that would be a huge help | 15:03 |
| *** tzn has joined #openstack-infra | 15:03 | |
| SpamapS | trash: I'll try to look at that very soon. | 15:03 |
| *** xarses has quit IRC | 15:04 | |
| pabelanger | fungi: so, randomly looking at a job, I see a 'cc' failure. A quick look into bindep-fallback.txt doesn't show gcc package, is that something we should be including there or done some other place? | 15:04 |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Add heat lbaas specific test for v1 and v2 https://review.openstack.org/281354 | 15:04 |
| *** gsagie has left #openstack-infra | 15:04 | |
| *** amotoki has joined #openstack-infra | 15:05 | |
| fungi | pabelanger: quite possible that we've missed preinstalling gcc on ubuntu-trusty images? | 15:05 |
| *** sc68cal has joined #openstack-infra | 15:05 | |
| fungi | pabelanger: i would have expected it to get pulled in as a dependency of one of the other packages in the fallback list | 15:05 |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Add heat lbaas specific test for v1 and v2 https://review.openstack.org/281354 | 15:05 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:05 | |
| *** sdake has quit IRC | 15:05 | |
| *** alivigni has joined #openstack-infra | 15:06 | |
| trash | SpamapS: cool, thanks | 15:06 |
| *** zeih has joined #openstack-infra | 15:06 | |
| pabelanger | fungi: maybe, that is what I am going to look into. But mostly trying to find out what should be in bindep-fallback.txt and what not. So, things like compliers, I don't think belong in bindep? | 15:06 |
| fungi | when i put that list together i manually compared the packages present on a bare-trusty node with those present on an ubuntu-trusty, then again after installing the fallback list | 15:06 |
| fungi | made sure every package on bare-trusty was also present on ubuntu-trusty after the fallback list was applied | 15:07 |
| *** esker has joined #openstack-infra | 15:07 | |
| *** esker has quit IRC | 15:07 | |
| fungi | but that was a couple months ago, so it's possible there has been a digression somewhere | 15:07 |
| *** esker has joined #openstack-infra | 15:07 | |
| pabelanger | a quick grep for system-config only shows gcc installed onto wheel_mirror_slave.pp | 15:07 |
| *** EricGonczer_ has joined #openstack-infra | 15:08 | |
| *** sdake has joined #openstack-infra | 15:08 | |
| *** tzn has quit IRC | 15:08 | |
| fungi | pabelanger: link one of the failure logs where you saw this error? | 15:08 |
| *** asselin__ has joined #openstack-infra | 15:08 | |
| pabelanger | Ya, let me find it again | 15:08 |
| pabelanger | I was randomly clicking | 15:08 |
| *** EricGonc_ has joined #openstack-infra | 15:09 | |
| *** asselin_ has quit IRC | 15:10 | |
| pabelanger | fungi: okay, not the same job, but found something related to cc: http://logs.openstack.org/88/281388/1/experimental/gate-cinder-pypy-bindep/f13bf87/console.html#_2016-02-17_22_01_29_047 | 15:12 |
| fungi | thanks, i'll see if i can come up with alternative explanations for that one | 15:12 |
| *** abregman has quit IRC | 15:12 | |
| pabelanger | to me, looks like cc is installed, but just an excpetion | 15:12 |
| *** EricGonczer_ has quit IRC | 15:12 | |
| clarkb | mordred thiagop pretty sure nothing prevents you running a bare metal cloud and nodepool on same network | 15:12 |
| *** piet has joined #openstack-infra | 15:13 | |
| fungi | pabelanger: did the python27 job for that same change fail? does that project normally run a pypy job and did it succeed on bare-trusty for the same change? | 15:13 |
| fungi | pabelanger: i'd be willing to chalk this up to psycopg not being usable on pypy (or at least that version of pypy) | 15:14 |
| smcginnis | dansmith: I've been seeing a lot of those too. | 15:14 |
| *** abregman has joined #openstack-infra | 15:14 | |
| smcginnis | dansmith: Rechecking cleared a couple for me, but something definitely up. | 15:14 |
| pabelanger | fungi: Ah, right you are. Looks like pypy is not running for that project | 15:14 |
| pabelanger | will move along to a bindep job that fails, when bare-trusty passes | 15:15 |
| *** piet has quit IRC | 15:15 | |
| dansmith | smcginnis: yeah, I've rechecked several times but my hitrate is poor | 15:17 |
| fungi | smcginnis: dansmith: i'll see if i can find the jenkins console from that job | 15:17 |
| clarkb | that means no log was copied, best way to debug that is find the jenkins run in jenkins | 15:18 |
| clarkb | looks like fungi will go hunting for that | 15:18 |
| pabelanger | fungi: could you confirm if this JJB is running on the correct host? https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/experimental-workers.yaml#L156 | 15:19 |
| pabelanger | I would expect that to be ubuntu-trusty | 15:20 |
| fungi | pabelanger: look at the top of the log at the hostname. it will begin with the label | 15:20 |
| pabelanger | right, this is the job I am looking at: http://logs.openstack.org/91/281391/1/experimental/gate-glance-requirements-bindep/9b262d6/console.html | 15:21 |
| pabelanger | which is failing because bindep is missing | 15:21 |
| pabelanger | i believe, it is because we are using the wrong node | 15:21 |
| fungi | confirmed | 15:21 |
| pabelanger | okay, let me get a patch up to fix that | 15:22 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API https://review.openstack.org/221964 | 15:22 |
| *** gongysh has quit IRC | 15:22 | |
| fungi | pabelanger: double-check i didn't fail to change that on any of the other templates i copied into that file | 15:22 |
| *** piet has joined #openstack-infra | 15:22 | |
| pabelanger | fungi: Yup, just confirming now | 15:23 |
| fungi | however, for now we can just ignore the gate-{name}-requirements-bindep check job results and confirm them later once that's fixed and propagated | 15:23 |
| fungi | good eye! | 15:23 |
| *** [1]Thelo has joined #openstack-infra | 15:23 | |
| pabelanger | agreed | 15:23 |
| *** dizquierdo has joined #openstack-infra | 15:23 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Switch gate-{name}-requirements-bindep to ubuntu-trusty https://review.openstack.org/281907 | 15:25 |
| *** achanda has joined #openstack-infra | 15:25 | |
| *** Thelo has quit IRC | 15:26 | |
| *** [1]Thelo is now known as Thelo | 15:26 | |
| *** yamahata has joined #openstack-infra | 15:27 | |
| *** zz_dimtruck is now known as dimtruck | 15:28 | |
| fungi | smcginnis: dansmith: clarkb: https://jenkins03.openstack.org/job/gate-nova-pep8/5558/console | 15:29 |
| *** ccrouch has quit IRC | 15:29 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/281661 | 15:29 |
| fungi | looks like the job failed to clone nova, and since that was what would have created the workspace for the job the publisher couldn't fire | 15:30 |
| fungi | or i may be misreading that | 15:30 |
| *** achanda has quit IRC | 15:30 | |
| fungi | is it saying /bin/bash doesn't exist? | 15:30 |
| fungi | maybe the rootfs disappeared out from under it? | 15:31 |
| clarkb | but it dodnt log brcause no bash and that crashed the job | 15:31 |
| *** tzn has joined #openstack-infra | 15:31 | |
| fungi | yeah, this is really pathological | 15:31 |
| clarkb | that is a prpblem with the post run publisher | 15:31 |
| clarkb | but also why no bash | 15:32 |
| fungi | the git failure was ValueError: I/O operation on closed file | 15:32 |
| fungi | so i think the filesystem really did just go out to lunch | 15:32 |
| fungi | and, once again, in rax-iad | 15:33 |
| dansmith | is that like one worker has a corrupt filesystem or something? | 15:33 |
| fungi | it's like the instance was working but then very early in the job the instance it lost access to its filesystem | 15:34 |
| fungi | and then *kaboom* | 15:35 |
| *** maishsk has quit IRC | 15:35 | |
| openstackgerrit | Julien Danjou proposed openstack/requirements: Update overtest to 0.11.0 https://review.openstack.org/281912 | 15:36 |
| *** wolverineav has joined #openstack-infra | 15:36 | |
| fungi | rackspace is doing more maintenance which could impact block device connectivity in iad today, but it's not supposed to be starting until 10pm cst according to their note | 15:36 |
| jeblair | and / isn't supposed to be on iscsi afaik | 15:36 |
| fungi | ahh, hopefully. i honestly don't know how rackspace has their environment set up | 15:37 |
| *** weshay has quit IRC | 15:38 | |
| smcginnis | fungi: I was hitting this yesterday as well, so I don't think it's unschedule rax maintenance. | 15:38 |
| fungi | but yeah, if we're seeing lots of these and can get a list, i should be able to grep worker names out of the zuul debug log to correlate back to a specific provider/region | 15:38 |
| *** xarses has joined #openstack-infra | 15:38 | |
| fungi | the trick is getting a representative list of jobs with no uploaded logs assembled | 15:39 |
| *** woodster_ has joined #openstack-infra | 15:39 | |
| fungi | so far my sample size of 1 says this is 100% in rackspace iad, but a larger sample size would be helpful ;) | 15:39 |
| *** jlanoux has quit IRC | 15:39 | |
| clarkb | and logstash wont know about it | 15:40 |
| fungi | right, which is why i have to play logstash on tv | 15:40 |
| clarkb | that publisher failing and crashing the job is :( | 15:40 |
| *** wolverineav has quit IRC | 15:41 | |
| thiagop | clarkb: hey, but the problem turns to be that I'll need a baremetal cloud to run a baremetal devstack testing environment | 15:41 |
| *** jpr has joined #openstack-infra | 15:41 | |
| thiagop | clarkb: seems too "Inception" for me... | 15:41 |
| fungi | clarkb: well, the console uploads happen by the worker copying the file down into the workspace and then uploading it... i wonder if we could use a tmpfs as the download location for the console log to work around situations like this? | 15:41 |
| mtreinish | clarkb, fungi: we do set the node_provider in the metadata now, so you should be able to filter on that in logstash | 15:42 |
| fungi | mtreinish: except when there's no log ever processed into logstash because of this specific failure mode | 15:42 |
| mtreinish | heh, fair point :) | 15:43 |
| clarkb | fungi will still fail on no bash | 15:43 |
| clarkb | I wonder if we can fli the order so co sole log starts before job crash | 15:44 |
| fungi | oh, yep. we're already doing whay i just said, forget it | 15:44 |
| clarkb | the scp co sole log | 15:44 |
| pabelanger | fungi: so far, python-swiftclient seems to be the best example of green across the board for bindep: https://review.openstack.org/#/c/281426/ | 15:44 |
| pabelanger | even pypy job is passing | 15:44 |
| clarkb | thiagop: there are various ways to do it | 15:45 |
| clarkb | thiagop: I just dont want people to think there is some hard requirement for separatong those in the tooling | 15:45 |
| *** zeih has quit IRC | 15:46 | |
| fungi | pabelanger: i basically just added a blanket set of experimental jobs to all those repos regardless of whether the equivalents on bare-trusty were voting, or even being run at all | 15:47 |
| pabelanger | fungi: Yup, but we can confirm the JJB template is correct. | 15:47 |
| pabelanger | fungi: so far, I don't see any failues between original jobs and bindep jobs | 15:47 |
| fungi | pabelanger: the clients are likely to be running a lot more of those jobs already on bare-trusty | 15:47 |
| fungi | pabelanger: that's what i was hoping we'd find, thanks! | 15:48 |
| *** ddieterly has quit IRC | 15:48 | |
| * esikachev is now away: Away from keyboard | 15:49 | |
| *** esikachev is now known as esikachev_afk | 15:49 | |
| *** ddieterly has joined #openstack-infra | 15:49 | |
| pabelanger | fungi: Ya, I'll keep looking but so far working as expected! | 15:49 |
| pabelanger | fungi: nice work | 15:49 |
| *** Qiming has quit IRC | 15:49 | |
| *** kozhukalov has joined #openstack-infra | 15:50 | |
| *** j_king has joined #openstack-infra | 15:50 | |
| *** thiagop has quit IRC | 15:50 | |
| *** infra-red has quit IRC | 15:50 | |
| *** sfinucan has joined #openstack-infra | 15:51 | |
| *** sc68cal has quit IRC | 15:51 | |
| *** infra-red has joined #openstack-infra | 15:51 | |
| *** chem has joined #openstack-infra | 15:51 | |
| fungi | pabelanger: thanks--i'm not especially proud of how long its taken. given the duration i've had this in progress it really had better be mostly mistake-free by now | 15:51 |
| *** vgridnev has quit IRC | 15:52 | |
| openstackgerrit | Pranav Salunke proposed openstack-infra/project-config: Adds training-labs post jobs to publish tarballs https://review.openstack.org/281721 | 15:52 |
| fungi | i'm mostly just paranoid because it's been so long now i expect it to be bitrottinf | 15:52 |
| *** yamamoto has quit IRC | 15:52 | |
| fungi | bitrotting | 15:52 |
| *** zul has quit IRC | 15:53 | |
| jeblair | fungi: like that g you just used | 15:53 |
| *** zul has joined #openstack-infra | 15:53 | |
| fungi | yep. it decayed by several bits | 15:54 |
| *** angdraug has joined #openstack-infra | 15:55 | |
| *** piet has quit IRC | 15:55 | |
| pabelanger | fungi: Ya, np. Anything to help migrate away from bare-* nodes :) | 15:55 |
| *** infra-red has quit IRC | 15:56 | |
| chem | hi, is this the correct place to ask for core reviewer to have a quick look to this one https://review.openstack.org/#/c/272156/ ? | 15:56 |
| openstackgerrit | Merged openstack/gertty: Fix diff crash on perm-only changes https://review.openstack.org/281476 | 15:56 |
| AJaeger | fungi, ianw had a suggestion on https://review.openstack.org/#/c/281562/ - but didn't vote. So, just pointing it out to you so that you - Thanks for the patch series! | 15:57 |
| AJaeger | chem: tell us a bit more about the change so that we can answer your question without opening our browser, please | 15:58 |
| *** jsavak has quit IRC | 15:58 | |
| *** jsavak has joined #openstack-infra | 15:59 | |
| chem | AJaeger: yep, sorry. It's about adding a non-voting jobs to puppet-openstack-cookicutter | 15:59 |
| *** piet has joined #openstack-infra | 15:59 | |
| fungi | clarkb: so without a pretty major refactoring i'm not sure how to bolster the console upload. basically grab_console_log.sh fired (why btw? was /bin/bash only missing some of the time?) but the ephemeral script that would have called zuul_swift_upload.py is separate | 15:59 |
| *** kushal has quit IRC | 15:59 | |
| *** kushal has joined #openstack-infra | 15:59 | |
| *** krotscheck_dcm is now known as krotscheck | 16:00 | |
| fungi | clarkb: i suppose we could collapse them into fewer scripts somehow | 16:00 |
| fungi | AJaeger: awesome, thanks for the heads up | 16:00 |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Adding integration test support for app catalog horizon plugin https://review.openstack.org/276440 | 16:00 |
| jeblair | fungi: are you sure bash was missing? maybe the cwd was missing | 16:00 |
| clarkb | fungi jenkins runs bash to run the other stuff | 16:00 |
| fungi | clarkb: yep, but... it successfully ran grab_console_log.sh which needs bash | 16:01 |
| clarkb | if we flip tge order of the publishers and make scp go first that will get us scp'd console log | 16:01 |
| jeblair | then our console logs will be missing information on uploads | 16:02 |
| fungi | clarkb: i don't see an scp that would have run for this, but maybe i'm misreading the job config. the only publisher for gate-{name}-pep8 is zuul-swift-upload-console-log looks like | 16:02 |
| AJaeger | chem: commented - as much as we love templates, let's not do it in this case ;) | 16:03 |
| clarkb | oh no scp for this at all | 16:03 |
| chem | AJaeger: thanks, looking. | 16:03 |
| *** salv-orl_ has joined #openstack-infra | 16:03 | |
| clarkb | and ya collapsing to a single publisher should mean one bash exec | 16:03 |
| *** FallenPegasus has joined #openstack-infra | 16:03 | |
| *** zigo has quit IRC | 16:03 | |
| fungi | basically we'd end up duplicating the contents of grab_console_log.sh into multiple uploader scripts i guess | 16:04 |
| jeblair | i'm not following why we would want to do that | 16:04 |
| fungi | though i guess multiple is really just two? | 16:04 |
| *** Daisy has quit IRC | 16:04 | |
| *** zigo has joined #openstack-infra | 16:05 | |
| jeblair | it was the first post-build script that failed | 16:05 |
| fungi | yeah, again this is a pathological case. i'm finding it odd that /bin/bash worked to run grab_console_log.sh but then a moment later seems not to exist at all | 16:05 |
| jeblair | oh, weird | 16:05 |
| jeblair | it wasn't was it? | 16:05 |
| *** salv-orlando has quit IRC | 16:06 | |
| *** coolsvap|away has quit IRC | 16:06 | |
| fungi | right, we downloaded the console log successfully. that required a working /bin/bash | 16:06 |
| fungi | then we go to run the shell script to upload it and suddenly /bin/bash is nowhere to be found | 16:06 |
| jeblair | it was the second that failed. i thought it was the first because we were still seeing error output from the previously terminated main build script | 16:06 |
| *** cschwede has quit IRC | 16:06 | |
| jeblair | fungi: i'm still contesting we don't know it's bash that's missing :) | 16:06 |
| fungi | yeah, one possible interpretation is that "No such file or directory" is referring to the cwd? | 16:07 |
| jeblair | yep | 16:07 |
| jeblair | i don't know which, i'm just keeping an open mind :) | 16:08 |
| fungi | same | 16:08 |
| fungi | regardless, i would think grab_console_log.sh was also run with the same cwd | 16:08 |
| jeblair | yes (though also the same bash) | 16:08 |
| fungi | so the mystery is for a similar series of possible events either way | 16:09 |
| fungi | i'm going to have a look in nodepool for another possible cause | 16:09 |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Summit Application API https://review.openstack.org/221964 | 16:10 |
| *** armax has joined #openstack-infra | 16:10 | |
| fungi | no dice. nodepool's log lines up with the expected timeline for this node | 16:12 |
| fungi | so it definitely didn't delete it until well after the job failed | 16:12 |
| openstackgerrit | Lenny Verkhovsky proposed openstack/diskimage-builder: adding ib_ipoib module https://review.openstack.org/281927 | 16:12 |
| *** harlowja_at_home has joined #openstack-infra | 16:13 | |
| *** nmagnezi has quit IRC | 16:13 | |
| *** armax has quit IRC | 16:14 | |
| fungi | here's another failing in the gate pipeline right now, but it's got connection errors to the wheel mirror (also rax-iad): https://jenkins04.openstack.org/job/gate-networking-cisco-python27/157/console | 16:15 |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Add heat lbaas specific test for v1 and v2 https://review.openstack.org/281354 | 16:15 |
| *** pcaruana has quit IRC | 16:15 | |
| openstackgerrit | Merged openstack-infra/project-config: Cleanup of constraint based jobs for neutron-vpnaas https://review.openstack.org/280922 | 16:15 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-cookicutter https://review.openstack.org/272156 | 16:15 |
| fungi | ReadTimeoutError: HTTPConnectionPool(host='mirror.iad.rax.openstack.org', port=80): Read timed out. | 16:16 |
| *** regXboi has joined #openstack-infra | 16:16 | |
| fungi | my gut is telling me that something has been pretty broken in iad since the maintenance window on tuesday | 16:16 |
| jeblair | fungi: the cache fs is still there and readwrite, but seems slow. | 16:17 |
| *** rcernin has quit IRC | 16:17 | |
| jeblair | but it could also just be node<->mirror connection problems | 16:18 |
| elmiko | hi, are draft reviews disabled in gerrit? | 16:18 |
| jeblair | elmiko: yes | 16:18 |
| jeblair | elmiko: they almost never work the way people expect them too | 16:18 |
| elmiko | jeblair: ack, thank you | 16:18 |
| fungi | elmiko: yes. they never worked well, so we completely broke their use to prevent anyone from trying to rely on them | 16:18 |
| jeblair | elmiko: which ended up causing lots of problems, so we disabled them | 16:18 |
| elmiko | fair, thanks for the clarification jeblair, fungi =) | 16:19 |
| fungi | though a good upstream gerrit feature request would be a configuration option to gracefully disable them | 16:19 |
| *** coolsvap|away has joined #openstack-infra | 16:20 | |
| *** piet has quit IRC | 16:20 | |
| *** piet has joined #openstack-infra | 16:20 | |
| *** yamamoto has joined #openstack-infra | 16:21 | |
| *** harlowja_at_home has quit IRC | 16:21 | |
| *** armax has joined #openstack-infra | 16:22 | |
| fungi | johnthetubaguy: do you happen to know whether rackspace is using local disk or network-attached disk for nova instance root filesystems in iad? | 16:22 |
| wznoinsk | hi infra | 16:22 |
| *** roxanaghe has joined #openstack-infra | 16:23 | |
| *** cschwede has joined #openstack-infra | 16:23 | |
| pabelanger | fungi: so far the only issue I can see is with subunit2html: http://logs.openstack.org/27/281427/1/experimental/gate-python-swiftclient-python34-bindep/0590ce7/console.html#_2016-02-17_21_37_20_580 | 16:25 |
| pabelanger | fungi: running the job again to see if it was not a blip | 16:25 |
| wznoinsk | is there a known way how to connect zuul/merger from downstream-puppets deployment via proxy? (wanted to check with you first before messing the code), it looks like gerrit's code doesn't support proxying commands to paramiko | 16:26 |
| alaski | fungi: that will depend on how you're booting the instances. unless you specifically request to boot from a volume it will use local disk | 16:26 |
| *** ccrouch has joined #openstack-infra | 16:26 | |
| *** ildikov has quit IRC | 16:26 | |
| fungi | alaski: booting from snapshot | 16:26 |
| fungi | alaski: so i guess that's local | 16:26 |
| fungi | thanks! | 16:26 |
| alaski | yep. | 16:26 |
| fungi | just trying to figure out if the apparent network connectivity issues and disappearing root fs in iad are related at all | 16:27 |
| clarkb | wznoinsk: I think it can use http(s)? | 16:27 |
| *** ildikov has joined #openstack-infra | 16:27 | |
| wznoinsk | sorry, the gerrit that's in zuul package doesn't seem to have any notion about proxy | 16:28 |
| wznoinsk | clarkb, I guess you meant https for a regular gerrit? | 16:29 |
| alaski | fungi: I no longer have insight into rax issues, but I would be surprised if that were the case. As long as the image is downloaded from Glance correctly, I can't recall if Nova verifies the checksum, the network shouldn't be involved. | 16:29 |
| clarkb | wznoinsk: zuul needs stream events for the event streamwhich is ssh only, the git stuff can http(s) | 16:30 |
| clarkb | wznoinsk: you can also set up a local port forward through corkscrew via http(s) | 16:30 |
| wznoinsk | clarkb, i've put proxying command to /etc/ssh/ssh_config on the machine but it looks like paramiko doesn't take that ssh config into account, still getting clarkb, | 16:31 |
| clarkb | correct | 16:31 |
| *** baoli_ has quit IRC | 16:31 | |
| wznoinsk | (sry, clipboard problem) | 16:31 |
| clarkb | instead you would set up a port forward through your proxy then connect to local side | 16:32 |
| *** bpokorny has joined #openstack-infra | 16:32 | |
| jordanP | I am investigatin why the multinode jobs fail so ofter. It seems that neither OVH nor bluebox pass the Tempest test "test_volume_backed_live_migration" | 16:32 |
| *** sc68cal has joined #openstack-infra | 16:32 | |
| wznoinsk | paste.openstack.org/show/487438 | 16:32 |
| clarkb | that way zuul doesnt need to learn and test a half dozen different proxy technologies | 16:32 |
| jordanP | the logstach query for this is message:"Live Migration failure: operation failed: migration job: unexpectedly failed" | 16:32 |
| *** harlowja_at_home has joined #openstack-infra | 16:33 | |
| jordanP | I think we should not run multinode jobs on those providers. What do you think ? | 16:33 |
| jeblair | jordanP: that's not really a choice | 16:33 |
| *** kozhukal` has joined #openstack-infra | 16:33 | |
| clarkb | wznoinsk: correct psramiko uses the ssh protocol but doesnt use the ssh command's config | 16:34 |
| jeblair | jordanP: we treat all of our providers as interchangable components of an ha system | 16:34 |
| wznoinsk | clarkb, I was thinking of creating a socket that's connecting to the proxy and pasing it to paramiko, it's using httplib.HTTPConnection | 16:34 |
| jeblair | jordanP: we may have to stop using any given provider at short notice, so we need all jobs to be able to run everywhere | 16:34 |
| clarkb | wznoinsk: the issue is there are many implementqtion dependent ways to proxy | 16:35 |
| jordanP | jeblair, I understand. But then we're stuck with those jobs. We can"t make them reliable | 16:35 |
| jordanP | (unless we skip that specific test of course) | 16:35 |
| *** kozhukalov has quit IRC | 16:35 | |
| fungi | alaski: indeed, i saw your e-mail address changed. congrats on the new job! | 16:36 |
| clarkb | wznoinsk: so it is often easiest to setup what your local needs demand and just point zuul to a port forward through that | 16:36 |
| jeblair | jordanP: there must be an underlying cause for why that test is failing; perhaps some resource, timing, or configuration issue exposed by those providers | 16:36 |
| wznoinsk | clarkb, never worked on a client to talk to proxy, I thought HTTP proxying would be fairly same everywhere | 16:36 |
| jeblair | jordanP: when clarkb has a moment, he may have more to add | 16:36 |
| clarkb | wznoinsk: sometines it is socks and sometimes only socksv4 etc etc | 16:36 |
| jordanP | my libvirt knowledge is lacking here. I'll dive more into it | 16:36 |
| wznoinsk | clarkb, except some of them use authentication and some not | 16:36 |
| alaski | fungi: thanks! | 16:36 |
| clarkb | wznoinsk: at hp we used a port forward effectively | 16:37 |
| clarkb | wznoinsk: yes you set it up outside of zuul and have it listen on say port 2222 | 16:37 |
| clarkb | wznoinsk: then you tell zuul to connect to port 2222 | 16:37 |
| wznoinsk | clarkb, sure, can do iptables rules as a last resort, just didn't want to rely on a 2nd thing to be done only to get the connection working | 16:37 |
| clarkb | jordanP: why cant we make them reliable? | 16:37 |
| jordanP | because they fail consistently with OVH and bluebox | 16:38 |
| jordanP | but not with the other providers | 16:38 |
| fungi | jordanP: that does not seem like a permament issue | 16:38 |
| clarkb | consistent fails do not imply unfixable | 16:39 |
| fungi | in theory either the test/job or provider implementation could be adjusted to fix or at least work around whatever is failing | 16:39 |
| jordanP | yes, maybe. I need to deep more into it | 16:39 |
| clarkb | wznoinsk: some people have proposed socks support others have proposed http things also gearman but so much of it is location dependent that not trying to s9lve it all in zuul has made sense | 16:40 |
| fungi | jordanP: yes, step 1 would be to collect some details on what is failing and why (a cloud provider is not a cause, merely a correlation) | 16:40 |
| *** jlanoux has joined #openstack-infra | 16:41 | |
| *** piet has quit IRC | 16:41 | |
| clarkb | wznoinsk: depending on your https proxy just tunneling through may not work etc | 16:41 |
| zaro | morning | 16:41 |
| *** sputnik13 has joined #openstack-infra | 16:42 | |
| fungi | clarkb: jeblair: so there is a nova change in the gate right this moment with four jobs _very_ recently failed with the same "No such file or directory" pattern in their jenkins console logs | 16:43 |
| fungi | three are in rax-iad, one is in rax-dfw | 16:43 |
| *** ctlaugh has quit IRC | 16:44 | |
| fungi | all bare-trusty | 16:44 |
| fungi | it looks like zuul-cloner is in use for these | 16:45 |
| clarkb | wznoinsk: anyways I can say we did port forwarding for gesrman with success at hp | 16:45 |
| wznoinsk | clarkb, I get the rationale why not to fix all these different proxying inside zuul, on the other hand if it;s not in zuul itself it's proxy-technology-specific connection chaining on firewall | 16:45 |
| wznoinsk | clarkb, could you share what you have setup for this port forwarding? | 16:46 |
| clarkb | wznoinsk: yup but everyone has a unique firewall and proxy setup | 16:46 |
| wznoinsk | clarkb, I guess it's not just FORWARD rule in iptables | 16:46 |
| clarkb | wznoinsk: no it was using ssh iirc | 16:46 |
| *** infra-red has joined #openstack-infra | 16:46 | |
| fungi | clarkb: jeblair: however it's not a 100% consistent failure rate because their python27 job on the same change also ran on bare-trusty in rax-iad and completed successfully | 16:46 |
| clarkb | so setup ssh with your proxy details, then make a portforward with ssh | 16:46 |
| wznoinsk | clarkb, makes sense | 16:46 |
| clarkb | then connect to the local side of that | 16:47 |
| *** harlowja_at_home has quit IRC | 16:47 | |
| clarkb | fungi that makes it more interesting | 16:47 |
| pabelanger | clarkb: gearman proxy for zuul? Does that actually exist or something just saying they would like it? | 16:47 |
| wznoinsk | you mean ssh -L 2222 review.openstack.org (via proxy) and connect zuul to local port 2222 ? | 16:47 |
| wznoinsk | clarkb, ^ | 16:48 |
| clarkb | wznoinsk: yup | 16:48 |
| clarkb | pabelanger: no zuul doesnt | 16:48 |
| wznoinsk | clarkb, cheers | 16:48 |
| *** tphummel has joined #openstack-infra | 16:48 | |
| openstackgerrit | Paul Michali proposed openstack-infra/project-config: Enable coverage non-voting for networking-cisco https://review.openstack.org/281956 | 16:49 |
| *** mikelk has quit IRC | 16:49 | |
| clarkb | its not ideal but unlike http protocols like gearman and ssh dont really have standard proxy methods so its all over the map | 16:49 |
| openstackgerrit | Khai Do proposed openstack-infra/project-config: Remove redundant access configuration https://review.openstack.org/281461 | 16:49 |
| *** dims has joined #openstack-infra | 16:50 | |
| fungi | okay, i still have links to those failures, though the change ahead of that nova change just failed a job so the zuul status is no longer displaying them due to a gate reset | 16:51 |
| *** bpokorny has quit IRC | 16:51 | |
| *** dims_ has quit IRC | 16:51 | |
| jeblair | fungi: can you etherpad links to them? | 16:51 |
| fungi | THOUGH! the gate reset was caused by a gate-keystone-python27 job failing in rax-iad with exactly the same failure mode | 16:51 |
| fungi | will do | 16:51 |
| *** sputnik13 has quit IRC | 16:53 | |
| *** scheuran has quit IRC | 16:53 | |
| fungi | jeblair: clarkb: example console logs linked from https://etherpad.openstack.org/p/VywDNLAyvI | 16:53 |
| *** tphummel has quit IRC | 16:53 | |
| *** jaypipes has quit IRC | 16:54 | |
| jeblair | they involve both zm05 and zm08 | 16:55 |
| *** tphummel has joined #openstack-infra | 16:55 | |
| *** exploreshaifali has joined #openstack-infra | 16:55 | |
| *** EricGonc_ has quit IRC | 16:56 | |
| fungi | i don't see any filesystem errors in dmesg on those mergers | 16:56 |
| *** sabeen1 has quit IRC | 16:56 | |
| *** igorbelikov has quit IRC | 16:56 | |
| *** zeih has joined #openstack-infra | 16:57 | |
| *** igorbelikov has joined #openstack-infra | 16:57 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: Add Gentoo to the list of supported distributions https://review.openstack.org/281960 | 16:58 |
| jeblair | 2016-02-18 16:40:25.791 | ERROR: Publisher be.certipost.hudson.plugin.SCPRepositoryPublisher aborted due to exception | 16:58 |
| jeblair | 2016-02-18 16:40:25.791 | /home/jenkins/workspace/gate-nova-tox-functional does not exist. | 16:58 |
| jeblair | fungi, clarkb: ^ i think i'm leaning more toward cwd rather than bash does not exist. | 16:58 |
| prometheanfire | hopefully that one is much simpler :P | 16:58 |
| fungi | oh, yep | 16:59 |
| *** sabeen1 has joined #openstack-infra | 16:59 | |
| *** bpokorny has joined #openstack-infra | 16:59 | |
| *** e0ne has quit IRC | 16:59 | |
| *** max_lobur has quit IRC | 16:59 | |
| *** piet has joined #openstack-infra | 16:59 | |
| fungi | checking to see if the macro is removing the workspace and expecting z-c to recreate it | 16:59 |
| *** piet has quit IRC | 17:00 | |
| *** sabeen3 has joined #openstack-infra | 17:00 | |
| *** _nadya_ has quit IRC | 17:01 | |
| *** gyee has joined #openstack-infra | 17:01 | |
| jeblair | or something about zuul-cloner/GitPython/git is munging that directory? | 17:02 |
| fungi | nope, seems to just specify a clonemap including the ZUUL_PROJECT into . | 17:02 |
| openstackgerrit | Monty Taylor proposed openstack/diskimage-builder: Depend on ifupdown in simple-init https://review.openstack.org/281843 | 17:02 |
| fungi | so the macro is at least not explicitly deleting it | 17:02 |
| clarkb | ya clonemapshould use . and jenkins creates the workspace | 17:02 |
| *** Daisy has joined #openstack-infra | 17:02 | |
| jeblair | gitpython recently released | 17:03 |
| fungi | but still, yes, something about z-c/gitpython could be replacing that directory and a failure partway through is leaving it missing | 17:03 |
| clarkb | dhellmann: on the os-client-config release what requirements change are you looking for? | 17:03 |
| *** esker has quit IRC | 17:03 | |
| *** sabeen1 has quit IRC | 17:03 | |
| *** esker has joined #openstack-infra | 17:03 | |
| *** _nadya_ has joined #openstack-infra | 17:03 | |
| jeblair | it's possible we're seeing this because of a behavior change there; or it's possible ther is no change and this is just bad luck because we're seeing a bunch of timeouts | 17:04 |
| mordred | clarkb: oh - I made one for you | 17:04 |
| *** fedexo has joined #openstack-infra | 17:04 | |
| clarkb | mordred: but no requirements change is needed | 17:04 |
| clarkb | so I am really confused | 17:04 |
| *** jsavak has quit IRC | 17:04 | |
| fungi | jeblair: yep, that could certainly still point back to it all being related to network issues in rackspace | 17:04 |
| mordred | clarkb: it's part of the releases repo process | 17:04 |
| pleia2 | good morning | 17:05 |
| clarkb | mordred: to do what though? literally nothing is needed | 17:05 |
| *** jsavak has joined #openstack-infra | 17:05 | |
| *** _nadya_ has quit IRC | 17:05 | |
| mordred | clarkb: you make a change that bumps the upper constraint that depends on the release patch | 17:05 |
| clarkb | no the automagic job does it... | 17:05 |
| mordred | clarkb: that then allows a job to run that checks the result of having done the thing | 17:05 |
| mordred | yes | 17:05 |
| mordred | you do not actually merge the constraints patch | 17:05 |
| *** tzn has quit IRC | 17:05 | |
| *** fhubik has quit IRC | 17:05 | |
| clarkb | mordred: and it will fail until release happens | 17:05 |
| fungi | jeblair: i wish we had a good way to figure out when this started happening, because my recollection is that people didn't start complaining until the outage on tuesday (which could still be complete coincidence) | 17:06 |
| *** Daisy has quit IRC | 17:07 | |
| jeblair | fungi: we could look for second-order effects -- consult zuul or nodepool logs to find pep8 jobs that ran >= 30 mins | 17:07 |
| *** apoorvad has joined #openstack-infra | 17:07 | |
| fungi | oh! i missed that these were running long | 17:07 |
| jeblair | fungi: yes, they are timing out | 17:07 |
| *** jaypipes has joined #openstack-infra | 17:08 | |
| jeblair | (i'm sure that's related, though unsure as to cause / effect) | 17:08 |
| fungi | gah, so they are | 17:08 |
| fungi | the tracebacks/errors all appear after the jenkins timeout | 17:08 |
| fungi | so i think speculation on any of the errors after the timeout comment are just the world falling apart around them | 17:09 |
| jeblair | fungi: in zuul-cloner, yes | 17:10 |
| armax | AJaeger: ping | 17:10 |
| *** tzn has joined #openstack-infra | 17:11 | |
| *** ericksonsantos has joined #openstack-infra | 17:11 | |
| *** dmsimard is now known as rdobot | 17:11 | |
| *** vgridnev has joined #openstack-infra | 17:11 | |
| fungi | traffic on the git servers doesn't look abnormally high, so i doubt this is a problem with the git cache on the images | 17:11 |
| AJaeger | armax: pong | 17:11 |
| *** esker has quit IRC | 17:11 | |
| fungi | the zuul mergers look pretty healthy too | 17:12 |
| fungi | at least from cacti's view | 17:12 |
| *** thiagop has joined #openstack-infra | 17:12 | |
| clarkb | fungi image buods have been pretty reliable other than ubuntu-trusty to rax | 17:12 |
| clarkb | *image uplaods | 17:12 |
| armax | AJaeger: I was looking at https://review.openstack.org/#/c/281665/ more closely and I am baffled as to why https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L754 didn’t catch it | 17:12 |
| *** rdobot is now known as dmsimard | 17:13 | |
| *** sridhar_ram has joined #openstack-infra | 17:13 | |
| armax | the match is vaild, and yet the job is executed | 17:13 |
| clarkb | jordanP: for migration issues I would start with nova compute and libvirt logs, I would also check to see if that specific test has a 100% fail rate or not | 17:13 |
| jordanP | yeah, yeah I know | 17:14 |
| jordanP | libvirt log are super hard to parse | 17:14 |
| clarkb | the health dashboard is tracking those tests now too so may have more info | 17:14 |
| jordanP | very verbose, and I know the exact timing when it failed but I can"t link to a specific line in the logs | 17:14 |
| fungi | clarkb: well, these are all bare-trusty, so no upload just snapshot | 17:15 |
| *** vgridnev has quit IRC | 17:15 | |
| therve | Could I get a review on a kilo requirements patch: https://review.openstack.org/278984 ? | 17:15 |
| armax | AJaeger: there’s a match on both on job and file affected | 17:15 |
| therve | It's blocking a heat backport | 17:15 |
| AJaeger | armax: I'll be back in an hour or so. Looking at it briefly, I'm confused as well... | 17:15 |
| clarkb | fungi oh right well in general git cache should be good | 17:15 |
| armax | AJaeger: no worries | 17:15 |
| AJaeger | armax: did you check the log output from gate-project-config-layout ? | 17:16 |
| armax | AJaeger: thanks for catching it though | 17:16 |
| * armax does now | 17:16 | |
| armax | I wonder if it’s because of the non-voting flag? | 17:17 |
| armax | it can’t be can it? | 17:17 |
| *** max_lobur has joined #openstack-infra | 17:18 | |
| *** vgridnev has joined #openstack-infra | 17:18 | |
| *** vgridnev has quit IRC | 17:19 | |
| fungi | clarkb: jeblair: perhaps completely unrelated, but my shell server in rax-iad where i run mutt keeps losing imap connectivity to rax's e-mail service hosting openstack.org mailboxes | 17:19 |
| fungi | just as of the past few days | 17:19 |
| armax | AJaeger: if I read this correctly it seems that the conditional is not applied to neutron changes? http://logs.openstack.org/65/281665/2/check/gate-project-config-layout/c725aa8/console.html#_2016-02-18_07_18_20_911 | 17:20 |
| *** claudiub has joined #openstack-infra | 17:22 | |
| pc_m | Hi All! Does anyone know if there is a project that is running coverage testing AND comparing results to a baseline? | 17:22 |
| clarkb | pc_m: rally might | 17:23 |
| boris-42 | pc_m: so we have job | 17:23 |
| boris-42 | pc_m: that runs 2 times coverage with patch and without | 17:23 |
| boris-42 | pc_m: and compares results | 17:23 |
| pc_m | boris-42: cool! Which project? | 17:23 |
| jeblair | fungi, clarkb: there are changes in gitpython 1.0.2 related to fixing deadlocks on cloning | 17:24 |
| jeblair | i wonder if they have had the opposite of the intended effect | 17:24 |
| *** reed_ has quit IRC | 17:25 | |
| fungi | jeblair: yeah, this definitely sounds a lot like the opposite of fixing a deadlock | 17:25 |
| pc_m | Looking to see if there is a mechanism in place that I could use on a project (at least for a period of time, while we try to improve coverage). | 17:25 |
| jeblair | fungi: if we performed the log analysis i suggested and found instances between feb 10 and feb 16 that would point towards gitpython | 17:25 |
| pc_m | boris-42: are you talking about rally? | 17:25 |
| fungi | jeblair: modulo nodepool image update latency | 17:26 |
| boris-42 | pc_m: inside rally we have CI job | 17:26 |
| jeblair | fungi: (rax networking may not be off the hook -- it's possible that we're looking at a network issue; a gitpython issue; or a gitpython issue triggered by a network issue...or many other things still) | 17:26 |
| jeblair | fungi: and that theory doesn't explain the missing dir as much as the timeout | 17:26 |
| fungi | jeblair: also curious we aren't seeing it outside rackspace afaict. zuul-cloner's virtualenv on all our images should in theory be using the latest gitpython | 17:26 |
| boris-42 | pc_m: that runs unit tests against rally and calculates coverage | 17:26 |
| wznoinsk | clarkb, is there a command on the gerrit review.openstack.org server I can run when established ssh connection to (for port forwarding) to keep the ssh connection open? I'm trying to avoid running gerrit stream-events unneccesarily | 17:27 |
| pc_m | boris-42: Nice. I'd be interested in learning more about it. | 17:27 |
| clarkb | fungi: and dsvm jobs use it | 17:27 |
| boris-42 | pc_m: https://github.com/openstack/rally/blob/master/tests/ci/cover.sh | 17:27 |
| boris-42 | pc_m: here is the code | 17:27 |
| pc_m | boris-42: THANKS! | 17:27 |
| clarkb | wznoinsk: stream events is the persistent one and zuul will use it | 17:27 |
| fungi | clarkb: though do the dsvm jobs use zuul-cloner? it may be that they aren't tickling gitpython in the same ways if not | 17:27 |
| *** fedexo has quit IRC | 17:27 | |
| *** vgridnev has joined #openstack-infra | 17:27 | |
| *** krotscheck is now known as krotscheck_dr | 17:27 | |
| clarkb | fungi: they do to setup d-g | 17:27 |
| *** jlk has left #openstack-infra | 17:28 | |
| boris-42 | pc_m: https://github.com/openstack/rally/blob/master/tox.ini#L42-L43 here is tox.ini configure | 17:28 |
| jeblair | but not on large repos like nova or keystone | 17:28 |
| clarkb | correct | 17:28 |
| fungi | true. so far examples all point to timeouts cloning large repositorues | 17:28 |
| fungi | repositories | 17:28 |
| *** vgridnev has quit IRC | 17:28 | |
| *** achanda has joined #openstack-infra | 17:28 | |
| wznoinsk | clarkb, I know, I meant what command to run in the tunnelling ssh connection I invoke from command line, running gerrit stream-events is a slight overkill as it won't be consumed by amnything | 17:28 |
| chem | AJaeger: I was waiting for the tests to run, but I'm leaving now. If you have time to look again at https://review.openstack.org/#/c/272156/ (puppet cookiecutter), that would be great :) | 17:28 |
| wznoinsk | clarkb, it looks like port forwarding is not enabled on ssh server on review.openstack.org | 17:29 |
| wznoinsk | I run | 17:29 |
| *** samuelBartel has quit IRC | 17:29 | |
| wznoinsk | ssh -L 29418:review.openstack.org:29418 review.openstack.org -p 29418 gerrit stream-events | 17:29 |
| wznoinsk | and get: ssh -L 29418:review.openstack.org:29418 review.openstack.org -p 29418 gerrit stream-events | 17:29 |
| wznoinsk | channel 3: open failed: administratively prohibited: Connection denied | 17:29 |
| *** EricGonczer_ has joined #openstack-infra | 17:30 | |
| clarkb | wznoinsk: for the first thing you dont need a command, for the other this reminds me you need a host on the other side hrm | 17:30 |
| jeblair | clarkb, fungi: i have some nodepool log parsing scripts, i'll see if i can adapt them for this | 17:30 |
| jeblair | we have nodepool logs that go back to feb 7 | 17:30 |
| kozhukal` | guys, could you please take a look again https://review.openstack.org/#/c/279074/ | 17:30 |
| * clarkb vlouds so much I forget that not everyone has hosts runnign all over | 17:30 | |
| *** kozhukal` has quit IRC | 17:30 | |
| *** kozhukal` has joined #openstack-infra | 17:31 | |
| jeblair | clarkb: i vlike your accent. | 17:31 |
| clarkb | jeblair: cool | 17:31 |
| *** EricGonc_ has joined #openstack-infra | 17:31 | |
| clarkb | jeblair: juggling two sick babies and phone irc :) | 17:31 |
| *** kozhukal` is now known as kozhukalov | 17:31 | |
| wznoinsk | clarkb, ah, it's relying on yet another external ssh server somewhere, not liking it | 17:31 |
| jeblair | clarkb: yeah, just know your voice in my head sounds completely different now :) | 17:31 |
| clarkb | jeblair: haha | 17:31 |
| clarkb | wznoinsk: another alternative is socks if you have such a proxy | 17:33 |
| clarkb | then you can tsocks maybr? | 17:33 |
| wznoinsk | clarkb, yes, I thought of using tsocks | 17:33 |
| wznoinsk | clarkb, ;-), will do | 17:33 |
| *** achanda has quit IRC | 17:33 | |
| *** vgridnev has joined #openstack-infra | 17:34 | |
| *** EricGonczer_ has quit IRC | 17:34 | |
| *** vgridnev has quit IRC | 17:35 | |
| *** baoli has joined #openstack-infra | 17:36 | |
| *** sfinucan has quit IRC | 17:36 | |
| *** rcernin has joined #openstack-infra | 17:36 | |
| *** wolverineav has joined #openstack-infra | 17:37 | |
| jeblair | clarkb, fungi: script written; slow parsing of log files proceeding. | 17:38 |
| *** matrohon has quit IRC | 17:38 | |
| phschwartz | Anyone on know of any contacts inside Vexxhost? | 17:39 |
| *** ccrouch1 has joined #openstack-infra | 17:39 | |
| *** ccrouch has quit IRC | 17:41 | |
| *** sabeen3 has quit IRC | 17:41 | |
| openstackgerrit | Clint Adams proposed openstack-infra/project-config: Clone all openstack/ projects before trying to search them for devstack plugins https://review.openstack.org/281981 | 17:41 |
| *** Swami has joined #openstack-infra | 17:42 | |
| jeblair | phschwartz: mnaser | 17:42 |
| Clint | clarkb: so is that insane? ^ | 17:42 |
| *** wolverineav has quit IRC | 17:42 | |
| *** vgridnev has joined #openstack-infra | 17:42 | |
| phschwartz | jeblair: ty, let me try pinging him | 17:44 |
| clarkb | Clint you can in theory have the job manage a cache for itself since host is persitent but otherwise not sure there is analternative | 17:44 |
| clarkb | Clint: you might get away with shallow clones? | 17:44 |
| clarkb | those dont work in a bunch of scenarios though | 17:45 |
| *** vgridnev has quit IRC | 17:45 | |
| Clint | can zuul-cloner do shallow clones? | 17:46 |
| clarkb | Maybe? hashar may have added that? | 17:46 |
| * Clint grins. | 17:47 | |
| jeblair | why would we shallow clone when we have the repo already? | 17:47 |
| hashar | it does not afaik | 17:47 |
| clarkb | jeblair: we dont have the repo on the proposal slave | 17:48 |
| jeblair | oh | 17:48 |
| hashar | possibly one might want to teach zuul-cloner to pass parameters to the underlying git clone | 17:48 |
| jeblair | then yes | 17:48 |
| jeblair | it's not sane :) | 17:48 |
| jeblair | we will likely run out of disk space | 17:48 |
| fungi | Clint: worth noting, the project-config list is not always guaranteed to be in sync with what's on the git farm. that assumption leads us to failing image updates if someone approves a new project creation change at the wrong moment, for example | 17:48 |
| Clint | ruh-roh | 17:48 |
| hashar | on wmf we do shallow clones a lot, that speed up things by a few order of magnitudes. For a single repo git archive is even faster | 17:48 |
| Clint | fungi: so, ignoring the disk-space problem for the moment, is there a way of getting the git farm list? | 17:49 |
| jeblair | Clint: you might also consider a network request to cgit if you're just looking for presence of a file in the git repo... | 17:49 |
| Clint | oh, hmm | 17:50 |
| fungi | i'm with jeblair on this one. if the filename is already going to be in a known place, you can just make http calls to the raw url for them | 17:50 |
| fungi | rather than trying to use git | 17:50 |
| Clint | that makes all kind of sense | 17:50 |
| jeblair | clarkb, fungi: https://etherpad.openstack.org/p/VywDNLAyvI updated with incidences of pep8 jobs taking longer than 30 mins | 17:50 |
| fungi | is this just a periodic job? | 17:50 |
| Clint | yep | 17:50 |
| fungi | so, yeah, no real benefit to relying on zuul-cloner anyway. there is no zuul ref to worry about | 17:51 |
| pc_m | boris-42: Thanks for all the info! | 17:51 |
| clarkb | well the idea was to have ability to use cache once zuulv3 happens but that may be premature | 17:51 |
| clarkb | and unnecessary if cgit works | 17:51 |
| Clint | so am i going to have to scrape cgit html to get the project list? | 17:52 |
| fungi | Clint: you could in theory check the gerrit api to get the list of active repos and then hit cgit to pull the files for them | 17:52 |
| *** vgridnev has joined #openstack-infra | 17:52 | |
| fungi | Clint: /projects is an unauthenticated method | 17:52 |
| fungi | you can just hit it anonymously | 17:53 |
| *** abregman has quit IRC | 17:53 | |
| Clint | fungi: is there not a race condition there too? | 17:53 |
| *** esikachev_afk is now known as esikachev | 17:53 | |
| fungi | Clint: in the safe direction though. i believe our ansible roles populate the repos on the git farm, then create them in gerrit so that it can immediately replicate into cgit? double-checking now | 17:54 |
| clarkb | yes | 17:54 |
| Clint | yay | 17:54 |
| fungi | and if so, then any which exist on gerrit (modulo rare circumstances during project rename maintenance windows) will already exist on the git farm | 17:55 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Roll up static Heat envs into CI directory https://review.openstack.org/280431 | 17:55 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Use netiso in the ha job https://review.openstack.org/273424 | 17:55 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 17:55 |
| * Clint wonders why the /projects/ json begins with )]}' | 17:57 | |
| fungi | Clint: confirmed... http://git.openstack.org/cgit/openstack-infra/system-config/tree/playbooks/remote_puppet_git.yaml | 17:57 |
| fungi | Clint: that leader is to thwart cross-site scripting attacks | 17:57 |
| fungi | Clint: to parse it as valid json you need to strip the first 3-4 characters | 17:57 |
| *** SumitNaiksatam has joined #openstack-infra | 17:57 | |
| Clint | interesting | 17:57 |
| jeblair | good word for it | 17:58 |
| AJaeger | armax: found it.. https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L1044 is the culprit | 17:58 |
| fungi | (3 vs 4 is just a question of whether you care about the newline or not) | 17:58 |
| armax | AJaeger: ah | 17:58 |
| AJaeger | that's the skip-if that appears in the log file - below the other one... | 17:58 |
| armax | AJaeger: so this means that the new skip if rules overrides the older? | 17:59 |
| AJaeger | armax: yes | 17:59 |
| armax | AJaeger: I see | 18:00 |
| AJaeger | armax, we need to move the rally skip if in the big skip-if with neutron... | 18:00 |
| armax | so would a fix entail adding neutron to the list of projects in line 1044 | 18:00 |
| boris-42 | pc_m: you are welcome | 18:00 |
| AJaeger | armax: the conditions are different for the all-files-match-any | 18:01 |
| armax | AJaeger: or what you said | 18:01 |
| AJaeger | armax: do you want to update your change? | 18:01 |
| armax | I can have a stub at it If I understood you correctly | 18:02 |
| *** jordanP has quit IRC | 18:02 | |
| openstackgerrit | Kevin Short proposed openstack-infra/jenkins-job-builder: An undefined label prevented building (make) in 'doc' directory. https://review.openstack.org/281992 | 18:02 |
| AJaeger | armax: I'll review. | 18:02 |
| *** mriedem has quit IRC | 18:02 | |
| armax | stand by | 18:02 |
| AJaeger | will - with interruptions ;) | 18:02 |
| armax | :) | 18:03 |
| *** keedya has joined #openstack-infra | 18:03 | |
| *** achanda has joined #openstack-infra | 18:03 | |
| *** mriedem has joined #openstack-infra | 18:04 | |
| pc_m | Hi again! Is there a known problem with gate failures due to timeouts? | 18:04 |
| *** jlanoux has quit IRC | 18:06 | |
| *** Jeffrey4l__ has quit IRC | 18:06 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Skip rally runs on Neutron's doc changes https://review.openstack.org/281665 | 18:06 |
| armax | AJaeger: ^ see if I got you correctyly | 18:06 |
| armax | correctly | 18:06 |
| *** e0ne has joined #openstack-infra | 18:07 | |
| *** cburgess has quit IRC | 18:07 | |
| *** cburgess has joined #openstack-infra | 18:08 | |
| AJaeger | boris-42: could you review 281665 , please? That change skips rally-dsvm-cli now also on documentation etc only changes | 18:09 |
| AJaeger | armax, boris-42: Not doing the skip for gate-rally-dsvm-cli seems very hard. So, I'm fine with that change. | 18:10 |
| *** flepied has quit IRC | 18:10 | |
| AJaeger | armax: PLease mention it in the commit message and explain the problem there. | 18:10 |
| armax | AJaeger: I thought I did | 18:10 |
| AJaeger | armax: Ah, you mentioned problem... | 18:10 |
| AJaeger | armax: you did ;) | 18:10 |
| clarkb | pc_m: yes jeblair and fungi are debugging | 18:10 |
| clarkb | pc_m: possibly related to new gitpython release or cloud networking or unknown cause | 18:11 |
| armax | AJaeger: thanks for your help | 18:11 |
| *** kzaitsev_mb has quit IRC | 18:11 | |
| pc_m | clarkb: cool. Seeing docs, then full, then docs fail on a commit. | 18:11 |
| pc_m | clarkb: Looks like my commit (in the gate) is getting re-run and so far has passed the docs test. | 18:12 |
| AJaeger | armax: thanks for fixing it;) | 18:13 |
| armax | AJaeger: yup, now I am eager to see how https://review.openstack.org/#/c/281633/ heals things, thanks for blessing it | 18:13 |
| *** sputnik13 has joined #openstack-infra | 18:13 | |
| armax | if I can have another friendly nudge, that’d be great | 18:14 |
| AJaeger | ;) | 18:14 |
| AJaeger | armax: if it's critical, you might want to ask clarkb ^ | 18:14 |
| *** samueldmq has quit IRC | 18:14 | |
| *** derekh has quit IRC | 18:16 | |
| clarkb | armax: AJaeger you can't live migrate with config drive iirc | 18:16 |
| clarkb | armax: AJaeger so the neutron multinode dsvm test will become 100% fail rate | 18:16 |
| *** e0ne has quit IRC | 18:16 | |
| clarkb | dvr multinode too | 18:17 |
| keedya | hi all, I made a new change to openstack/shovel 2 hrs ago but i don't see the jenkins jobs running yet | 18:17 |
| AJaeger | keedya: did you chekc http://status.openstack.org/zuul/ ? | 18:17 |
| clarkb | keedya: if you look at http://status.openstack.org/zuul/ you can see that there is a shovel change queued for alost 2 hours | 18:17 |
| *** hashar has quit IRC | 18:17 | |
| AJaeger | keedya: 281938? It's running... | 18:18 |
| AJaeger | keedya: I mean: Queued | 18:18 |
| keedya | thanks :) | 18:18 |
| clarkb | mriedem: sdague: is it still the case that config drive and live migrate are not compatibile iwth nova? | 18:19 |
| armax | clarkb: we’re trying to go through the troubleshooting chain | 18:19 |
| *** kushal has quit IRC | 18:19 | |
| sdague | clarkb: yes, I think so. It's being worked on atm though. | 18:19 |
| clarkb | armax: I get that, but it was a specific conscious decision to disable config drive on those tests because it could not work | 18:19 |
| mriedem | clarkb: for block migration | 18:19 |
| armax | clarkb: interesting | 18:20 |
| armax | clarkb: how so? | 18:20 |
| clarkb | sdague: mriedem thanks | 18:20 |
| mriedem | hence https://review.openstack.org/#/c/272999/ | 18:20 |
| clarkb | armax: nova does not support live migrate on hosts with config drive | 18:20 |
| clarkb | armax: so live migrate will fail | 18:20 |
| armax | clarkb: I see | 18:20 |
| armax | clarkb: ok, so we’d be trading one failure mode with another | 18:20 |
| armax | clarkb: I guess we then need to enable ec2-api in the multi-node setting | 18:21 |
| armax | um, actually no | 18:21 |
| sdague | armax: hmm... what now? because we kind of deleted that. | 18:21 |
| armax | sdague: no, I am mistaken | 18:21 |
| armax | what I mean is this | 18:21 |
| armax | I believe there’s a bug in grenade | 18:22 |
| *** notmorgan has quit IRC | 18:22 | |
| sdague | armax: ok, cool. In what way? | 18:22 |
| sdague | because it's pretty much an out of the box devstack | 18:22 |
| armax | sdague: it looks like that after upgrade the api-paste still points to ec2 stuff | 18:22 |
| armax | http://logs.openstack.org/78/279378/9/experimental/gate-grenade-dsvm-neutron-multinode/40a5659/logs/etc/nova/api-paste.ini.txt.gz | 18:22 |
| *** notmorgan has joined #openstack-infra | 18:23 | |
| *** notmorgan has quit IRC | 18:23 | |
| *** notmorgan has joined #openstack-infra | 18:23 | |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add trace option https://review.openstack.org/280220 | 18:23 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add show-all-changes option https://review.openstack.org/280221 | 18:23 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add depth option https://review.openstack.org/280217 | 18:23 |
| armax | I see now upgrade script here: | 18:23 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add reset-first option https://review.openstack.org/280218 | 18:23 |
| openstackgerrit | Lucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add no-tags option https://review.openstack.org/280219 | 18:23 |
| armax | https://github.com/openstack-dev/grenade/tree/master/projects/60_nova | 18:23 |
| *** jed56 has quit IRC | 18:23 | |
| armax | that removes the stuff from api paste | 18:23 |
| AJaeger | armax: could you bless https://review.openstack.org/281240 and https://review.openstack.org/#/c/269174/, please? | 18:23 |
| armax | sdague: is that the reason why the nova api returns 404 to the neutron metadata proxy here: | 18:23 |
| sdague | don't know, but that's not the ec2api | 18:24 |
| armax | http://logs.openstack.org/78/279378/9/experimental/gate-grenade-dsvm-neutron-multinode/40a5659/logs/new/screen-q-meta.txt.gz#_2016-02-17_22_33_12_082 | 18:24 |
| armax | sdague: right, I take that back entirely | 18:24 |
| armax | AJaeger: looking | 18:24 |
| sdague | it's the metadata server | 18:24 |
| sdague | armax: that should be talking to the nova metadata server right? | 18:25 |
| armax | sdague: correct | 18:26 |
| sdague | so figuring out why that isn't allowed is the root issue | 18:26 |
| *** bardia has joined #openstack-infra | 18:26 | |
| sdague | that's all running in n-api in this config | 18:26 |
| armax | sdague: yup | 18:26 |
| *** achanda has quit IRC | 18:26 | |
| armax | sdague: we’ll have to dig deeper | 18:26 |
| clarkb | mtreinish: and you mentioned we need to restart the subunit log worker? | 18:27 |
| armax | sdague: my initial hunch was that the api pipeline was getting confused after the upgrade | 18:27 |
| bardia | anyone know how I can re-trigger Jenkins gate on my review. its python34 test failed to start: http://logs.openstack.org/66/279766/2/gate/gate-cinder-python34/44b54e4/console.html | 18:27 |
| mgagne | live migration + config drive: https://review.openstack.org/#/c/234659/ | 18:27 |
| armax | mgagne: nice | 18:28 |
| mtreinish | clarkb: I think so, I'm still seeing a bunch of empty runs being added to the db, and I think that's caused by 404s when trying to get the subunit stream | 18:28 |
| armax | sdague: ok, let me dig deeper, I’ll reach out for your expertise later if you don’t mind | 18:28 |
| mgagne | armax: we fixed it on our side by using rsync to copy over the config drive | 18:28 |
| clarkb | mtreinish: and that should be fixed in a current version? | 18:28 |
| mtreinish | clarkb: yeah: http://git.openstack.org/cgit/openstack-infra/puppet-subunit2sql/commit/?id=5327cc64a87f63958fd24929bb5d2765c61f752e | 18:28 |
| mtreinish | I just don't think we ever restarted the worker after that landed | 18:29 |
| sdague | armax: if you can get the request id for the request to nova, that is probably going to be helpful | 18:29 |
| sdague | I don't seem to see that in the logs | 18:29 |
| AJaeger | bardia: http://docs.openstack.org/infra/manual/developers.html#automated-testing | 18:29 |
| clarkb | mtreinish: looks like someone restarted it around 0109 UTC today | 18:29 |
| *** esker has joined #openstack-infra | 18:29 | |
| mtreinish | ok, cool | 18:29 |
| armax | sdague: actually a better run is this: | 18:29 |
| mtreinish | it's been a couple of days since I looked for empty runs | 18:29 |
| mtreinish | I'll keep my eyes on it | 18:29 |
| bardia | AJaeger: thanks | 18:30 |
| sdague | armax: oh, I found a thing - http://logs.openstack.org/78/279378/9/experimental/gate-grenade-dsvm-neutron-multinode/40a5659/logs/new/screen-n-api.txt.gz#_2016-02-17_22_32_00_638 | 18:30 |
| *** e0ne has joined #openstack-infra | 18:30 | |
| armax | http://logs.openstack.org/00/281600/1/experimental/gate-grenade-dsvm-neutron-multinode/94a9a2c/logs/new/screen-q-meta.txt.gz#_2016-02-18_02_28_05_436 | 18:30 |
| sdague | and yep, there is a nova bug there for sure | 18:30 |
| sdague | I thought there was a patch for this | 18:30 |
| *** esker has quit IRC | 18:30 | |
| armax | sdague: yeah we tried to fix that with https://review.openstack.org/#/c/281600/ | 18:30 |
| sdague | dims: weren't you working on that? | 18:30 |
| armax | sdague: there was | 18:30 |
| armax | mriedem has a patch | 18:30 |
| armax | with that path | 18:30 |
| *** ccrouch1 has quit IRC | 18:30 | |
| armax | now the neutron metadata agent gets a 404 | 18:31 |
| mriedem | indeed | 18:31 |
| *** esker has joined #openstack-infra | 18:31 | |
| mriedem | sdague: https://review.openstack.org/#/c/279721/ | 18:31 |
| *** vivekd has joined #openstack-infra | 18:31 | |
| sdague | ok, is there a log there? | 18:31 |
| sdague | find me a 404 log file, and I'll look, and maybe be able to help | 18:32 |
| *** _sarob has joined #openstack-infra | 18:32 | |
| armax | that’s where I left yesterday | 18:32 |
| armax | sdague: right on it | 18:32 |
| armax | sdague: you mean in the nova api log | 18:32 |
| sdague | armax: yeh | 18:32 |
| mriedem | sdague: the 404 didn't merge yet | 18:32 |
| mriedem | my patch failed the gate | 18:32 |
| mriedem | i'm fixing the TypeError to return a 404 | 18:33 |
| armax | mriedem: I have a depends-on on a patch of mine | 18:33 |
| mriedem | https://review.openstack.org/#/c/279721/3/nova/api/ec2/__init__.py | 18:33 |
| mriedem | yeah, i suspected | 18:33 |
| armax | sdague: here http://logs.openstack.org/00/281600/1/experimental/gate-grenade-dsvm-neutron-multinode/94a9a2c/logs/new/screen-n-api.txt.gz#_2016-02-18_02_28_05_433 | 18:33 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Initiate testing for puppet-openstack-cookiecutter https://review.openstack.org/272156 | 18:33 |
| armax | sdague: not a ton to go by | 18:33 |
| *** bardia has left #openstack-infra | 18:33 | |
| armax | sdague: so my next step was to drill down | 18:34 |
| mriedem | armax: well it's a 404 b/c the ec2api isn't in nova anymore | 18:34 |
| armax | mriedem: ok | 18:34 |
| armax | mriedem: so we’re back to a misconfiguration after upgrade | 18:34 |
| *** eharney has quit IRC | 18:34 | |
| sdague | mriedem: that's not the ec2api | 18:34 |
| sdague | right? | 18:35 |
| *** kushal has joined #openstack-infra | 18:35 | |
| armax | all I know right now is that metadata agent and nova-api talk fine | 18:35 |
| sdague | yeh | 18:35 |
| armax | when running Liberty | 18:35 |
| armax | as soon as we move to trunk things go astray | 18:35 |
| armax | I have no frame of comparison because on single nodes we use config drive everywhere | 18:36 |
| mtreinish | clarkb: yeah, it looks like the last one was from a couple of days ago: http://paste.openstack.org/show/487463/ | 18:36 |
| *** austin81 has joined #openstack-infra | 18:36 | |
| sdague | right, agreed, we have a high bug tracking this? I'm going to get some lunch then I'll put this on the top of my stack. | 18:36 |
| openstackgerrit | Merged openstack-infra/project-config: Add experimental rally job for networking-odl. https://review.openstack.org/269174 | 18:36 |
| mtreinish | clarkb: so from before the restart | 18:36 |
| clarkb | mtreinish: great, just checking in on that | 18:36 |
| sdague | armax: yeh, the postgresql job used to run in this config | 18:36 |
| mriedem | sdague: still does | 18:36 |
| armax | sdague: not yet, because I am not sure where to file it as I haven’t pinpointed the culprit | 18:36 |
| mriedem | just not with neutron | 18:36 |
| sdague | mriedem: right, there is no such job on neutron at the moment | 18:36 |
| clarkb | the neutron postgres job does too I think | 18:36 |
| mriedem | sdague: the typeerror bug is https://bugs.launchpad.net/nova/+bug/1545101 | 18:37 |
| openstack | Launchpad bug 1545101 in OpenStack Compute (nova) ""TypeError: __init__() takes exactly 3 arguments (2 given)" in n-api logs for nova metadata api request" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | 18:37 |
| clarkb | since armax's change updates it to not | 18:37 |
| dims | sdague : "GET /2009-04-04/meta-data/instance-id HTTP/1.1" is that openstack compute API? | 18:37 |
| sdague | dims: it's the metadata api | 18:37 |
| clarkb | https://review.openstack.org/#/c/281633/1/jenkins/jobs/devstack-gate.yaml | 18:37 |
| sdague | in liberty we run 3 apis on different ports | 18:37 |
| sdague | openstack, metadata, ec2 | 18:37 |
| mriedem | dims: no, it's the ec2 route for metadata https://github.com/openstack/nova/blob/master/nova/api/metadata/base.py#L238 | 18:37 |
| *** eharney has joined #openstack-infra | 18:37 | |
| mriedem | so i suspect it's asking for an ec2 route for the instance-id | 18:37 |
| *** achanda has joined #openstack-infra | 18:37 | |
| mriedem | we nixed the middleware code that provided that | 18:37 |
| mriedem | and give a 404 now | 18:37 |
| openstackgerrit | Merged openstack-infra/project-config: Remove python-constraints-jobs from ofagent job definition https://review.openstack.org/281240 | 18:37 |
| sdague | ok, so this we need to unwind | 18:38 |
| *** wolverineav has joined #openstack-infra | 18:38 | |
| sdague | I was kind of wondering if the hatchet on some of the metadata bits was going to cause issues, and now we know | 18:38 |
| sdague | armax: this is a nova bug for sure | 18:38 |
| armax | sdague: https://review.openstack.org/#/c/232325/ | 18:38 |
| sdague | and a release blocker I'd say | 18:38 |
| armax | not that I can tell | 18:38 |
| sdague | ok, I need to get some lunch | 18:38 |
| sdague | then I'll put this on top of my stack | 18:39 |
| armax | sdague: ok | 18:39 |
| armax | sdague: ping me if you need me | 18:39 |
| jeblair | google is returning "'Forever, Interrupted' a novel by Taylor Jenkins Reid" as the top result for my current query. | 18:40 |
| fungi | must be a really good book? | 18:41 |
| fungi | is it about interrupted git remote calls? | 18:42 |
| jeblair | i hope so! we'll see if it's any help when amazon delivers it | 18:42 |
| * fungi would read that book if so | 18:42 | |
| *** BobBall has quit IRC | 18:42 | |
| *** gyee has quit IRC | 18:44 | |
| *** shashank_hegde has joined #openstack-infra | 18:46 | |
| austin81 | AJaeger, ianw, sdague, anteaya: Would you mind taking a look at https://review.openstack.org/#/c/279317/ when you get a minute? Thanks | 18:48 |
| openstackgerrit | Victor Morales proposed openstack/requirements: Add schemadisplay-sphinx to global requirements https://review.openstack.org/281880 | 18:48 |
| AJaeger | austin81: I'm not familiar enough with that to give it a good review - better wait for ianw... | 18:49 |
| clarkb | with zfs on ubuntu this lts I wonder if we can test cinder + zfs | 18:49 |
| austin81 | AJaeger: No problem. If you're interested: https://etherpad.openstack.org/p/BKgWlKIjgQ | 18:49 |
| austin81 | mtreinish: https://review.openstack.org/#/c/279317/ | 18:50 |
| jeblair | clarkb, fungi: git is deleting the workspace directory because it is interrupted during a clone operation | 18:51 |
| clarkb | jeblair: oh! | 18:51 |
| *** zeih has quit IRC | 18:51 | |
| fungi | got it, so git will remove the clone target if the clone fails, even if the target preexisted | 18:51 |
| clarkb | if we used ./foo it would delete ./foo | 18:52 |
| jeblair | clarkb, fungi: they thing i got hung up on for a while is that it's not cloning to ".", it's cloning to /path/to/workspace. rmdir will refuse to remove '.' but not /path/to/dot. | 18:52 |
| clarkb | since we use . it deletes . | 18:52 |
| fungi | i suppose we could recreate that directory if it doesn't exist, prior to attempting to collect logs | 18:52 |
| fungi | that wuold be a relatively simple hackaround | 18:52 |
| *** vgridnev has quit IRC | 18:52 | |
| clarkb | jeblair: gotcha | 18:53 |
| jeblair | fungi: we could even put it in the cleanup wrapper we have around zuul-cloner there | 18:53 |
| jeblair | mkdir -p $WORKSPACE | 18:53 |
| fungi | ahh, yep, we already trap | 18:53 |
| fungi | good call | 18:53 |
| AJaeger | austin81: gave one comment - but others need to chime in. pabelanger, do you have time to look at 279317? | 18:54 |
| fungi | so that's an even better one-liner | 18:54 |
| pabelanger | AJaeger: sure | 18:54 |
| AJaeger | pabelanger: see austin81's etherpad reference 5 mins ago as well | 18:54 |
| AJaeger | thanks, pabelanger | 18:54 |
| thiagop | Guys, another question wrt CI deployment: I see that puppet creates a clouds.yaml on nodepool user's home, but don't understand how I manage to put something there through project-config (since it'll be overwritten every time I change somehting) | 18:55 |
| * esikachev is now away: Away from keyboard | 18:55 | |
| *** esikachev is now known as esikachev_afk | 18:55 | |
| jeblair | clarkb, fungi: though, i wonder if that's racy -- i think we saw the post build actions happening before all of the cleanup in some examples... | 18:55 |
| thiagop | any hints? | 18:55 |
| jeblair | esikachev_afk: would you please disable your away message? thanks. | 18:55 |
| asselin | thiagop, you need to put it in heira | 18:55 |
| *** ybathia has joined #openstack-infra | 18:55 | |
| asselin | thiagop, https://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/contrib/single_node_ci_data.yaml#n30 | 18:56 |
| pabelanger | AJaeger: austin81: Why is this in diskimage-builder over having puppet do this? | 18:56 |
| fungi | jeblair: can you point to an example of the trap in the builder firing after a publisher started? | 18:57 |
| AJaeger | pabelanger: I have no idea - and too little knowledge to review in depth | 18:57 |
| *** pfallenop has quit IRC | 18:57 | |
| *** darvon has quit IRC | 18:57 | |
| sdague | armax: I'd suggest we take the debug on this one over into the #nova channel. | 18:57 |
| fungi | jeblair: i wouldn't expect it, but that of course doesn't make it particularly unlikely... | 18:57 |
| jeblair | fungi: https://jenkins03.openstack.org/job/gate-nova-pep8/5558/console | 18:57 |
| armax | sdague: agreed | 18:57 |
| sdague | just so we don't clog up other things here | 18:57 |
| jeblair | fungi: at least, the *log messages* appear in that order | 18:57 |
| *** rossella_s has quit IRC | 18:57 | |
| pabelanger | AJaeger: austin81: will leave comments in review, but we tend to use puppet to manage job dependencies. | 18:58 |
| jeblair | fungi: i can't say for sure they actually happened in that order | 18:58 |
| *** sabeen has joined #openstack-infra | 18:58 | |
| fungi | huh, yep | 18:58 |
| *** rossella_s has joined #openstack-infra | 18:58 | |
| fungi | oh, wait... are we retrieving the console log in the builder trap rather than in the publisher? | 18:58 |
| fungi | revisiting my grokkage | 18:58 |
| jeblair | fungi: no, it's a postbuildstep | 18:58 |
| fungi | k | 18:59 |
| *** kzaitsev_mb has joined #openstack-infra | 18:59 | |
| clarkb | with the timing out we cant necessarily rely on the z-c script | 18:59 |
| fungi | yeah | 18:59 |
| fungi | also a good point | 18:59 |
| *** sabeen3 has joined #openstack-infra | 19:00 | |
| fungi | any solid workaround probably needs to happen in the publisher, except the publisher may fail to even start if its cwd is mia | 19:00 |
| jeblair | it seems to reliably run. but i agree, it _may_ be racy, and it _may_ be early terminated if jenkins is lucky | 19:00 |
| pabelanger | AJaeger: okay, comments left | 19:00 |
| AJaeger | thanks, pabelanger | 19:01 |
| jeblair | clarkb, fungi: probably the most solid workaround would be to adjust how we do that cloning. maybe clone to a subdir then move up a level or something. | 19:01 |
| *** dizquierdo has quit IRC | 19:01 | |
| *** elo has joined #openstack-infra | 19:01 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-openstackci: Clarify oscc_file_contents instructions https://review.openstack.org/282010 | 19:01 |
| fungi | i don't suppose we can avoid dereferencing . from the clonemap? | 19:02 |
| *** yonglihe has quit IRC | 19:02 | |
| asselin | thiagop, ^^ does that help? | 19:02 |
| *** fkautz has joined #openstack-infra | 19:02 | |
| asselin | thiagop, https://review.openstack.org/282010 | 19:02 |
| jeblair | (we could also change the expectations of the jobs to operate in a subdir; $WORKSPACE/$ZUUL_PROJECT. there may be dragons there.) | 19:02 |
| *** sabeen has quit IRC | 19:02 | |
| fungi | since it sounds like git avoids removing a target if the target is literally . | 19:02 |
| clarkb | yea.... | 19:02 |
| clarkb | there be dragons | 19:02 |
| jeblair | fungi: heh, actually, git doesn't avoid it, glibc refuses to do it. :) but same effect. | 19:02 |
| *** sridhar_ram1 has joined #openstack-infra | 19:02 | |
| thiagop | asselin__: I think so, now I'm nothing the same line that is on clouds.yaml on the oscc section | 19:02 |
| fungi | oh, amusing | 19:03 |
| jeblair | rmdir(".") = -1 EINVAL (Invalid argument) | 19:03 |
| jeblair | :) | 19:03 |
| thiagop | asselin__: not very intuitive or explicit on docs though, but thank you very much sir | 19:03 |
| *** achanda has quit IRC | 19:03 | |
| fungi | jeblair: but git ignores that error? | 19:03 |
| fkautz | is infra the right place to log a bug as a new user where contact info cannot be saved on openstack.org? I get a server error and it is blocking me from contributing. | 19:03 |
| thiagop | asselin__: (or miss) | 19:03 |
| *** flepied has joined #openstack-infra | 19:03 | |
| *** tzn has quit IRC | 19:03 | |
| thiagop | :) | 19:03 |
| fungi | jeblair: oh, i guess it doesn't matter if it ignores it. it's failing and trying to clean up at that point anyway | 19:03 |
| asselin | thiagop, please don't hesitate to upload doc fixes, etc. | 19:03 |
| *** sridhar_ram has quit IRC | 19:04 | |
| asselin | thiagop, also, please +1 the change if satisfied or -1 if it can still be more clear. | 19:04 |
| jeblair | fungi: yes; it does ignore it since it's in a git internal cleanup function | 19:04 |
| *** jaosorior has quit IRC | 19:04 | |
| jeblair | (it's in an atexit) | 19:04 |
| *** pfallenop has joined #openstack-infra | 19:05 | |
| *** darvon has joined #openstack-infra | 19:05 | |
| thiagop | asselin__: sure | 19:05 |
| fungi | yeah, so that seems like a safe path out of this if we can wrangle it | 19:05 |
| jeblair | fungi: which? clone to subdir and move, or avoid deref '.' in zc? | 19:06 |
| fungi | jeblair: avoid dereferencing . | 19:06 |
| fungi | it's less change to the structure of the jobs | 19:06 |
| fungi | though possibly hard to test for regression | 19:07 |
| *** kzaitsev_mb has quit IRC | 19:07 | |
| jeblair | fungi: yes; it is a change to zuul that we'll want to run by hashar and maybe others. we would also almost certainly need a unit test with some good comments for this, because it seems really easy to regress on "use '.' to get a specific side effect of glibc+git behavior" :) | 19:07 |
| *** Guest20178 has quit IRC | 19:08 | |
| fungi | can we chmod a-w .. or chown 0:0 .. before revoking sudo? | 19:10 |
| fungi | that should prevent unlinking the cwd | 19:10 |
| jeblair | fungi: it makes jenkins own homedir unwritable by jenkins :( | 19:10 |
| fungi | well, not the homedir | 19:11 |
| fungi | but a subdirectory of it | 19:11 |
| jeblair | right, sorry, ~/workspace | 19:11 |
| fungi | the actual path in question is /home/jenkins/workspace/gate-nova-pep8 | 19:11 |
| fungi | yeah | 19:11 |
| *** andymaier has quit IRC | 19:11 | |
| fungi | i agree that too is an ugly hack | 19:11 |
| *** annegentle has joined #openstack-infra | 19:12 | |
| fungi | so from what i can see in these examples, grab_console_log.sh is running consistently. perhaps we can simply make its first command to be to recreate the cwd if missing? | 19:13 |
| *** wolverineav has quit IRC | 19:13 | |
| fungi | what worries me is i still don't get _why_ it's not breaking | 19:14 |
| *** tzn has joined #openstack-infra | 19:14 | |
| *** wolverineav has joined #openstack-infra | 19:14 | |
| *** _sarob has quit IRC | 19:14 | |
| jeblair | fungi: that probably points to these actually happening in parallel, so $W hasn't been removed yet | 19:14 |
| fungi | right, so also racy | 19:15 |
| jeblair | which suggests that the cleanup trap is the best place to put the mkdir since it's closest to the deletion. basically, smallest race window. | 19:15 |
| fungi | what if we mkdir -p . _both_ at the start of the publishers _and_ in the cleanup trap? | 19:15 |
| jeblair | (if we were to take that appoarch) | 19:15 |
| jeblair | fungi: if my mental model is right, i think that almost by definition, putting the mkdir in the publishers will never be used. | 19:16 |
| fungi | that way we hedge our bets on the race that we don't try to start a new process in the split moment between the git cleanup and the trap wrapping it | 19:16 |
| *** eharney has quit IRC | 19:16 | |
| *** achanda has joined #openstack-infra | 19:17 | |
| austin81 | AJaeger, pabelanger: Thanks for the comments guys. greghaynes, ianw what do you guys think of a puppet deploy for stackviz rather than a dib element? https://review.openstack.org/#/c/279317 | 19:17 |
| fungi | yeah, i guess if the builder is forcibly terminated after git cleanup but before the trap fires, and for whatever reason the publisher hasn't started yet, the publisher would be unable to start anyway | 19:17 |
| jeblair | fungi: for that to have an effect, the publisher would have to start, git perform the delete, the publisher perform the mkdir, another publisher start, then the cleanup run | 19:17 |
| *** wolverineav has quit IRC | 19:18 | |
| fungi | we've both managed to convince me on the trap as the only sane place if we want to recreate the directory after cleanup | 19:18 |
| *** tzn has quit IRC | 19:18 | |
| jeblair | fungi: we make a good team | 19:19 |
| *** rubasov has quit IRC | 19:19 | |
| fungi | this will also take some jjb babysitting if we go that route. probably many hundreds of jobs affected | 19:19 |
| dansmith | clarkb: fungi: am I reading the scrollback correctly that this is still going on, only on some nodes, but probably high enough probability that we should not be rechecking things? | 19:20 |
| jeblair | dansmith: i think the underlying cause is intermittent network problems, so a recheck may be effective | 19:20 |
| *** angdraug has quit IRC | 19:20 | |
| fungi | dansmith: i think it's either recheck and live with the failures until we can get a better picture of what's going on, or live without the capacity of rax-iad (maybe all of rax?) assuming we can narrow it to just there | 19:20 |
| jeblair | fungi: agreed | 19:21 |
| dansmith | okay, sounds like rechecking is just going to further absorb bandwidth | 19:21 |
| fungi | i haven't seen any jobs running outside of rackspace impacted yet | 19:21 |
| dansmith | I mean, I hit it like three times in a row this morning | 19:21 |
| clarkb | and we are working through how to fix logging | 19:21 |
| dansmith | so I'll just hold off | 19:21 |
| fungi | dansmith: i personally saw a nova change in the gate fail FOUR jobs on this, and then get restarted in a gate reset because a keystone change ahead of it also failed a job on the same problem | 19:22 |
| AJaeger | fkautz: did you follow all steps in order at http://docs.openstack.org/infra/manual/developers.html#account-setup ? | 19:22 |
| jeblair | lucky nova change? | 19:22 |
| fungi | indeed | 19:22 |
| dansmith | fungi: yeah :/ | 19:22 |
| fkautz | AJaeger: i've been following that document | 19:23 |
| fkautz | when i try to add contact info, i get a server error on the website | 19:23 |
| openstackgerrit | Khai Do proposed openstack-infra/git-review: WIP: push change with a vote https://review.openstack.org/281504 | 19:23 |
| fungi | so anyway, i'm torn on whether we're wasting more capacity on rechecks than we'd lose by temporarily turning off rackspace | 19:23 |
| AJaeger | fkautz: additional troubleshooting tips are at https://ask.openstack.org/question/56720 | 19:24 |
| *** _sarob has joined #openstack-infra | 19:24 | |
| *** _sarob has quit IRC | 19:24 | |
| jeblair | fungi, clarkb: my preference would be to perform a job change -- either (a) mkdir -p; or (b) clone to subdir then move. because we can get those in faster than (c) change z-c. i'm not ruling out (c); just saying it's not a good fit for fixing log collection. | 19:24 |
| clarkb | fungi: only rackspace can run these jobs | 19:24 |
| *** _sarob has joined #openstack-infra | 19:24 | |
| fungi | clarkb: oh! | 19:24 |
| openstackgerrit | Khai Do proposed openstack-infra/git-review: remove non-supported python versions https://review.openstack.org/282015 | 19:24 |
| fungi | right, these are bare-trusty | 19:25 |
| fungi | so not dib-able | 19:25 |
| fungi | so that explains why we'd only see this in rackspace | 19:25 |
| jeblair | it does? | 19:25 |
| clarkb | we can switch to ubuntu trusty without reliable rax uploads or add snapshot builds to the other clouds | 19:25 |
| *** asettle has joined #openstack-infra | 19:25 | |
| jeblair | clarkb: hasn't "reliable uploads" landed? | 19:26 |
| clarkb | jeblair: yes but not released | 19:26 |
| jeblair | clarkb: what's the holdup? | 19:26 |
| clarkb | we need the release | 19:26 |
| fungi | well, the only jobs which seem to be impacted are jobs running on bare-trusty. i've found no examples of this failing for jobs run on devstack-trusty nodes in rackspace | 19:26 |
| clarkb | of oscc then update on nodepool and restart services | 19:27 |
| jeblair | fungi: yes, but this will only affect zuul-cloner jobs with constraints... how does that map to images? | 19:27 |
| fungi | however, now that i think about it. the tox-based jobs are the only ones probably using zuul-cloner to clone to the cwd itself | 19:27 |
| jeblair | clarkb: i mean, who do we poke to release oscc? | 19:27 |
| clarkb | jeblair: dhellmann | 19:27 |
| AJaeger | jeblair: or dims | 19:27 |
| clarkb | the releases change was pushed yesterday | 19:27 |
| jeblair | dhellmann! dims! :) | 19:27 |
| dhellmann | hi | 19:27 |
| openstackgerrit | Khai Do proposed openstack-infra/git-review: Update gerrit version for testing https://review.openstack.org/282018 | 19:28 |
| fungi | so for devstack-gate jobs the issue is likely just manifesting as job timeouts | 19:28 |
| AJaeger | clarkb: it was released today! | 19:28 |
| jeblair | dhellmann: will you release oscc in exchange for a cookie? | 19:28 |
| clarkb | and I habent seen a thrilled to announce email so pretty sure it hasnt happened | 19:28 |
| *** flip214 has quit IRC | 19:28 | |
| clarkb | AJaeger: oh was no email sent about it? | 19:28 |
| AJaeger | clarkb: https://review.openstack.org/281526 merged nearly 2 hours ago... | 19:28 |
| dhellmann | jeblair : I just did a release of os-client-config, is that what you want? | 19:28 |
| jeblair | dhellmann: yes! thanks! now i don't know if i should keep the cookie for a future bribe or give it to you in thanks. :) | 19:29 |
| *** dtantsur is now known as dtantsur|afk | 19:29 | |
| clarkb | yup, dhellmann was email sent? I dont see the email if so | 19:29 |
| dhellmann | apparently not, it seems that job didn't run http://logs.openstack.org/1b/1bb476d04fd3ef92c22fa50c0c7b26236af41a8b/release/ | 19:29 |
| openstackgerrit | Khai Do proposed openstack-infra/git-review: WIP: push change with a vote https://review.openstack.org/281504 | 19:29 |
| *** d0ugal has quit IRC | 19:29 | |
| *** asettle has quit IRC | 19:29 | |
| * dims catches up | 19:30 | |
| *** Apsu has left #openstack-infra | 19:30 | |
| jeblair | how about i look up the zuul log for that as thanks and eat the cookie | 19:30 |
| *** ddieterly has quit IRC | 19:30 | |
| clarkb | sounds good | 19:30 |
| dims | haha :) | 19:30 |
| *** d0ugal has joined #openstack-infra | 19:30 | |
| *** d0ugal has quit IRC | 19:30 | |
| *** d0ugal has joined #openstack-infra | 19:30 | |
| dhellmann | jeblair, clarkb : I don't see an announce job configured for that project. | 19:30 |
| clarkb | dhellmann: oh its configured on project sid | 19:31 |
| fungi | well, it also didn't upload to pypi https://pypi.python.org/pypi/os-client-config | 19:31 |
| dhellmann | oh, nevermind, I misread | 19:31 |
| dhellmann | fungi : 503 error from pypi | 19:31 |
| dhellmann | http://logs.openstack.org/1b/1bb476d04fd3ef92c22fa50c0c7b26236af41a8b/release/os-client-config-pypi-both-upload/dec6118/console.html | 19:31 |
| fungi | gnar | 19:31 |
| jeblair | 2016-02-18 17:54:06.093 | requests.exceptions.HTTPError: 503 Server Error: No healthy backends | 19:32 |
| jeblair | nice error | 19:32 |
| fungi | yeah, love that wording | 19:32 |
| pleia2 | heh | 19:32 |
| fungi | i believe that's been explained as fastly proxying the post and having an intermittent connectivity failure to the pypi backend | 19:32 |
| *** dtantsur|afk is now known as dtantsur | 19:33 | |
| *** eharney has joined #openstack-infra | 19:33 | |
| jeblair | should we "while !twine upload;" ? | 19:33 |
| fungi | we seem to hit this extremely often. i don't know if this is a widespread problem on pypi or if we really do just upload way more often than any other projects | 19:33 |
| fungi | jeblair: newer versions of twine than we're running, i believe, include an option to no-op if the file is already there | 19:33 |
| fungi | i need to look that up again and double-check | 19:33 |
| jeblair | fungi: is it bad for us to reupload the same file? | 19:34 |
| dhellmann | jeblair : sigmavirus24 did have some changes to twine to retry the upload, but getting that release into a system package for our secure build servers will take time, so yeah, I think we need to write a retry loop | 19:34 |
| AJaeger | anybody wants to +2A clarkb's twine upload check https://review.openstack.org/275431 ? | 19:34 |
| fungi | well, sometimes this manifests as a 5xx error back even though it uploaded successfully | 19:34 |
| *** jsavak has quit IRC | 19:34 | |
| fungi | but pypi will also error if you try to upload a file that's already on pypi | 19:34 |
| fungi | so we would loop indefinitely in those cases | 19:34 |
| dhellmann | AJaeger : needs a loop, right? | 19:34 |
| *** wolverineav has joined #openstack-infra | 19:34 | |
| jeblair | yeah, but looks like maybe we should loop around with clarkb's logic | 19:35 |
| dhellmann | fungi : there's a flag to twine to tell it to ignore existing files | 19:35 |
| *** jsavak has joined #openstack-infra | 19:35 | |
| jeblair | so, upload, check, if not exist, retry | 19:35 |
| dhellmann | though I don't know what version that flag is available in | 19:35 |
| fungi | dhellmann: right, but not in the version of twine packaged for trusty i thinkl | 19:35 |
| dhellmann | ok | 19:35 |
| *** flip214 has joined #openstack-infra | 19:35 | |
| *** ccrouch has joined #openstack-infra | 19:35 | |
| dhellmann | so check first, only try twine if the file doesn't exist, only fail if it doesn't exist after twine | 19:35 |
| fungi | so we'd be back to installing twine via pip and then breaking every time requests/urllib3/somebody decided to do something "fun" in a new release | 19:35 |
| dtantsur | dhellmann, sorry for jumping it, but could you elaborate on that "announcement jobs" thing? does it mean that projects are no longer announced without additional efforts? | 19:35 |
| sigmavirus24 | dhellmann: we have other functionality that will complement what's proposed for twine (like adding an option to verify something was uploaded and the SHA's match) | 19:35 |
| jeblair | fungi: nope! not gonna! | 19:35 |
| dtantsur | dhellmann, i.e. I don't see an announcement for ironic-inspector even though it seems released | 19:36 |
| sigmavirus24 | would that I had more time to work on things like twine | 19:36 |
| clarkb | fwiw I really cant update changes right now/today. I am doing my best to help what I can from phone while also doing my best to take care of sick kids | 19:36 |
| clarkb | so feel free to update as needed | 19:36 |
| dhellmann | dtantsur : you have the job. maybe there was a failure in the chain somewhere, like with oscc | 19:36 |
| *** ddieterly has joined #openstack-infra | 19:36 | |
| fungi | but yeah, clarkb's idea of checking /simple/whatever/ for the file we just uploaded and retrying after some wait seems like the best compromise | 19:37 |
| jeblair | clarkb: :( will do thanks and good luck | 19:37 |
| AJaeger | dtantsur: http://lists.openstack.org/pipermail/openstack-announce/2016-January/000938.html | 19:37 |
| dtantsur | AJaeger, that's 3.0, we just released 2.2.4 for liberty | 19:37 |
| dhellmann | AJaeger : that's old | 19:37 |
| AJaeger | dtantsur: ah... | 19:37 |
| fungi | dhellmann: anyway, since the upload never happened, i'll reenqueue that os-client-config tag | 19:37 |
| fungi | unless jeblair was already working on that | 19:38 |
| AJaeger | dtantsur: might be same problem ;( let's check logs... | 19:38 |
| jeblair | fungi: nope | 19:38 |
| fungi | on it then | 19:38 |
| dhellmann | dtantsur, fungi : the ironic-inspector upload failed, too: http://logs.openstack.org/e5/e5c61b11ba85a1f8b03f3859fc7ce2c7461848f7/release/ironic-inspector-pypi-both-upload/a7aac06/console.html | 19:38 |
| AJaeger | dtantsur: https://pypi.python.org/pypi/ironic-inspector/2.2.4 | 19:38 |
| dhellmann | AJaeger : ^^ | 19:38 |
| fungi | AJaeger: did the wheel make it onto pypi? | 19:38 |
| *** jkraj has quit IRC | 19:38 | |
| *** claudiub has quit IRC | 19:38 | |
| dtantsur | dhellmann, I see the wheel on pypi.. | 19:38 |
| AJaeger | yes, the wheel is there | 19:38 |
| *** exploreshaifali has quit IRC | 19:39 | |
| fungi | yeah, so for that one i probably need to manually upload the sdist to pypi and then retrigger teh announce job | 19:39 |
| dtantsur | dhellmann, also, if openstack-server-release-jobs is what you mean, I don't see it on inspector | 19:39 |
| dhellmann | dtantsur : sometimes pypi says the upload failed even when it works | 19:39 |
| fungi | i'll work on that as soon as i'm done with the oscc reenqueue | 19:39 |
| jeblair | okay: todo list: 1) manually run the oscc upload; 2) make a new patch on top of 275431 to loop; 3) add mkdir -p to z-c jobs; 4) restart nodepool and upload some images to rax; 5) something about bare-trusty nodes?; 6) actually think about what's causing these jobs to time out in the first place. | 19:39 |
| dhellmann | dtantsur : you have "publish-to-pypi" which includes "{name}-announce-release" | 19:39 |
| *** daemontool has joined #openstack-infra | 19:39 | |
| dtantsur | dhellmann, ah thanks :) sorry for confusion | 19:40 |
| dhellmann | dtantsur : the pypi upload job exited with an error, so the subsequent jobs didn't run | 19:40 |
| dtantsur | yeah, got it | 19:40 |
| jeblair | clarkb: can you clarify 5 a bit? | 19:40 |
| fungi | jeblair: i'm popping 1 off that list now. will pop whatever else is at top once i'm done with 1 | 19:40 |
| jeblair | fungi: cool, you do 2; i'll do 3. | 19:40 |
| fungi | sounds good | 19:41 |
| clarkb | jeblair: ya, bare-trusty only build via snapshot because of puppet service resource interactions with dib and mysql | 19:41 |
| jeblair | (unless someone else wants 2) | 19:41 |
| *** dtantsur is now known as dtantsur|afk | 19:41 | |
| openstackgerrit | Lucas Alvares Gomes proposed openstack/requirements: Add "virtualbmc" dependency https://review.openstack.org/282026 | 19:41 |
| clarkb | jeblair: becausr we have said no snapshots on new clouds this means no bare-trusty images on new clouds | 19:41 |
| fungi | i am not opposed to one of our 7 other core reviewers pitching in on this pile | 19:41 |
| jeblair | clarkb: how does getting rax uploads fix that? we can switch these jobs to ubuntu-trusty once they are on rax? | 19:41 |
| clarkb | plan is to switch to bindep + ubuntu-trusty which is dib based everywhere | 19:42 |
| fungi | or even non-core reviewers can submit and review patches | 19:42 |
| clarkb | jeblair: yup that | 19:42 |
| clarkb | jeblair: we didnt want to rely on uploads across the board like that until rax was reliabler for it | 19:42 |
| jeblair | clarkb: if we make the switch right now, will it work? (we'll just stop using rax for these but will use ovh/bb/internap, right?) | 19:42 |
| fungi | yeah, my experimental bindep trials across a bunch of jobs/projects seem to pan out according to pabelanger's reviews of logs | 19:42 |
| clarkb | jeblair: yes and yes | 19:42 |
| jeblair | that actually would temporarily alleviate the network issues. | 19:43 |
| fungi | we'll of course need to step the ubuntu-trusty min-ready up sufficiently to take on most of the bare-trusty volume | 19:43 |
| *** MarkAtwood has joined #openstack-infra | 19:44 | |
| *** SumitNaiksatam has quit IRC | 19:44 | |
| jeblair | fungi: i don't think min-ready makes too much of a difference under load with the current allocator. i mean, we should change it, but i don't think it's a prereq. | 19:44 |
| pabelanger | fungi: so, I only found 1 py34 job that was failing, I mentioned it before. It looked related to subunit2html. See 281427 | 19:44 |
| fungi | jeblair: oh, yeah i guess we're running at full capacity regardless | 19:44 |
| fungi | pabelanger: was that project voting on gate-{name}-python34? | 19:45 |
| *** ccrouch has quit IRC | 19:45 | |
| fungi | if not, i'd probably ignore it, but will take a closer look in a bit | 19:45 |
| pabelanger | fungi: Yes, the normal python34 job passed | 19:46 |
| pabelanger | but bindep failed | 19:46 |
| pabelanger | that was the only review I noticed a failure on bindep but pass on the original job | 19:46 |
| fungi | cool, thanks. definitely something we'll want to suss out then | 19:46 |
| clarkb | jeblair note that rax is in the ubuntu-tristy label the images may just be old | 19:47 |
| clarkb | so we want to remove rax from there if we do this to avoid network trouble | 19:47 |
| *** ccrouch has joined #openstack-infra | 19:47 | |
| *** waynr has quit IRC | 19:47 | |
| *** waynr has joined #openstack-infra | 19:48 | |
| *** waynr has quit IRC | 19:48 | |
| *** waynr has joined #openstack-infra | 19:48 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Recreate workspace in zuul-cloner cleanup https://review.openstack.org/282034 | 19:48 |
| *** jpr has quit IRC | 19:48 | |
| *** bhunter_ has joined #openstack-infra | 19:49 | |
| fungi | well, it may be we're also hitting similar network issues on devstack jobs in rackspace, just manifesting differently? so we'd probably turn it all down there | 19:49 |
| *** rfolco has quit IRC | 19:50 | |
| *** _nadya_ has joined #openstack-infra | 19:50 | |
| *** achanda has quit IRC | 19:51 | |
| bhunter_ | a question: does the URL '/cli' intentionally not work on a infra/puppet-jenkins created jenkins server? | 19:52 |
| clarkb | bhunter_: yes, there was a security advisory about how it shluldnt be exposed | 19:53 |
| fungi | bhunter_: i believe that may be a workaround to avoid a serious remote code execution vulnerability in the cli endpoint's deserialization | 19:53 |
| bhunter_ | ooh, thanks, both of you! | 19:53 |
| *** lucasagomes is now known as lucas-dinner | 19:54 | |
| bhunter_ | is there a flag for this, or a particular line of code you could point to? | 19:54 |
| bhunter_ | you know, for the 'daring' | 19:54 |
| fungi | bhunter_: if memory serves, it's just being masked by the apache vhost config, so you could in theory undo that (but i wouldn't recommend exposing your jenkins api to any untrusted networks if you do) | 19:54 |
| jeblair | clarkb, fungi: back on the question of whether gitpython updates could be involved -- we have updated bare-trusty in rax-iad every day since the 10th except for the 15th. so i think we would have gotten the new release asap. | 19:54 |
| clarkb | fungi: we have a groovy init acript to disable it | 19:54 |
| fungi | oh, right, that was the published "fix" | 19:55 |
| *** rajinir has quit IRC | 19:55 | |
| *** rcernin has quit IRC | 19:55 | |
| fungi | though also possibly better constrained in newer jenkins releases too? i'd look through their security vulnerability list for confirmation | 19:55 |
| clarkb | jeblair: so wpuldve been noticed earlier? | 19:55 |
| fungi | clarkb: but maybe only obvious when coupled with serious network issues | 19:56 |
| fungi | i'm also unconvinced that avoiding running nodes in rackspace will work around that, since the git servers, zuul mergers, et cetera are also in rackspace (but maybe it's mostly iad that's the problem, and our servers are in dfw) | 19:56 |
| jeblair | clarkb: the error rates before and after the release are about the same. it's not a lot of data, and i still can't say that the behavior isn't worse in the new release with severe network issues that started on tuesday. | 19:56 |
| fungi | i did see at least one failure like this on a worker in dfw though | 19:57 |
| *** daemontool has quit IRC | 19:57 | |
| jeblair | clarkb, fungi: i think it does at least confirm that the peak starting on tuesday really is pointing at the network issues as the trigger rather than anything like gitpython being coincidentally updated on the same day. | 19:57 |
| *** gyee has joined #openstack-infra | 19:58 | |
| *** SumitNaiksatam has joined #openstack-infra | 19:58 | |
| openstackgerrit | Merged openstack-infra/project-config: Check twine uploads for success https://review.openstack.org/275431 | 19:58 |
| fungi | i concur | 19:58 |
| *** achanda has joined #openstack-infra | 19:58 | |
| *** ociuhandu has quit IRC | 19:58 | |
| jeblair | clarkb, fungi: so: network issues making git behave badly is one theory. network issues making gitpython behave badly is another. we may be able to distinguish between them by looking at d-g jobs. | 19:58 |
| fungi | also i have faint hopes that the network maintenance starting in a few hours will fix it and not just make it worse | 19:58 |
| *** ddieterl_ has joined #openstack-infra | 19:59 | |
| *** annegentle has quit IRC | 19:59 | |
| *** krtaylor has quit IRC | 19:59 | |
| *** amrith is now known as _amrith_ | 19:59 | |
| jeblair | http://status.openstack.org/ has both "QA Health" and "Openstack-Health" on the navbar | 20:00 |
| *** daemontool has joined #openstack-infra | 20:00 | |
| *** ddieterly has quit IRC | 20:01 | |
| *** dkliban has joined #openstack-infra | 20:01 | |
| fungi | sounds doubly-healthy? | 20:02 |
| pleia2 | I seem to recall a couple patches in flight around the same time to add openstack-health, so I think the got their streams crossed | 20:02 |
| jeblair | a disturbing number of bugs on e-r with recent failures are described as hpcloud specific. | 20:02 |
| * pleia2 has a look | 20:02 | |
| *** dkliban has left #openstack-infra | 20:02 | |
| *** burgerk has joined #openstack-infra | 20:02 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 20:02 |
| *** waynr has quit IRC | 20:03 | |
| pabelanger | fungi: looks like pip and setuptools are installed on passing py34 in tox, but not on bindep. That appears to be the only difference I see with dependencies atm | 20:03 |
| fungi | clarkb: jeblair: dhellmann: oscc seems to be successfully released now | 20:03 |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/system-config: Revert "Update common.js to include OpenStack-Health" https://review.openstack.org/282038 | 20:03 |
| pabelanger | need to step away for a bit | 20:03 |
| pleia2 | mtreinish: ^^ this one that we looked at at LCA did end up being a bit of a duplicate | 20:04 |
| *** waynr has joined #openstack-infra | 20:04 | |
| dhellmann | fungi : does that include the announce email? if not, I can do that by hand | 20:04 |
| fungi | dhellmann: yes, http://logs.openstack.org/1b/1bb476d04fd3ef92c22fa50c0c7b26236af41a8b/release/os-client-config-announce-release/41416a0/console.html | 20:04 |
| clarkb | fungi woot, we just need to double check it is installed on nodepool.o.o and rsstart the builder (for oscc) and nodepoold for the other recent bigfixes | 20:04 |
| dhellmann | excellent | 20:04 |
| fungi | dhellmann: dtantsur|afk: i've also uploaded the missing ironic-inspector sdist to pypi and have queued the announcement job for it (though it hasn't run yet) | 20:05 |
| dhellmann | fungi : great, thank you! | 20:05 |
| *** aysyd has quit IRC | 20:05 | |
| fungi | working on the twine and chect of pypi/simple loop next | 20:06 |
| fungi | er, check | 20:06 |
| *** eric_lopez has joined #openstack-infra | 20:07 | |
| *** annegentle has joined #openstack-infra | 20:07 | |
| *** wolverin_ has joined #openstack-infra | 20:07 | |
| *** jswarren_ has joined #openstack-infra | 20:07 | |
| *** |-paul-| has joined #openstack-infra | 20:07 | |
| *** markvoelker_ has joined #openstack-infra | 20:07 | |
| *** EricGonc_ has quit IRC | 20:08 | |
| jeblair | clarkb, fungi: http://status.openstack.org//openstack-health/#/g/node_provider/rax-iad?groupKey=node_provider&resolutionKey=hour&duration=P7D | 20:08 |
| *** lindsayk__ has joined #openstack-infra | 20:08 | |
| jeblair | i don't see a noticable increase in rax-iad failures | 20:09 |
| jeblair | (that, for the moment at least, still basically only includes the dsvm jobs) | 20:09 |
| fungi | agreed | 20:09 |
| *** flip214_ has joined #openstack-infra | 20:09 | |
| *** notmorga1 has joined #openstack-infra | 20:09 | |
| *** notmorga1 has quit IRC | 20:09 | |
| *** notmorga1 has joined #openstack-infra | 20:09 | |
| jeblair | and in fact, 0 failures for gate-tempest-dsvm-full | 20:10 |
| *** EricGonczer_ has joined #openstack-infra | 20:10 | |
| clarkb | ya dsvm joba look generally fine | 20:10 |
| *** krtaylor has joined #openstack-infra | 20:11 | |
| jeblair | we do have those git operations in a retry loop | 20:11 |
| *** xwizard___ has joined #openstack-infra | 20:11 | |
| jeblair | however, it is starting to suggest that it isn't just that git alone that is getting stuck, yeah? | 20:11 |
| *** SlickN1k has joined #openstack-infra | 20:11 | |
| *** wolverineav has quit IRC | 20:12 | |
| *** lindsayk_ has quit IRC | 20:12 | |
| *** flip214 has quit IRC | 20:12 | |
| *** notmorgan has quit IRC | 20:12 | |
| *** jcooley has quit IRC | 20:12 | |
| *** devananda has quit IRC | 20:12 | |
| *** markvoelker has quit IRC | 20:12 | |
| *** tlbr_ has quit IRC | 20:12 | |
| *** xwizard_ has quit IRC | 20:12 | |
| *** bnemec has quit IRC | 20:12 | |
| *** morgabra has quit IRC | 20:12 | |
| *** elo has quit IRC | 20:12 | |
| *** rossella_s has quit IRC | 20:12 | |
| *** dkranz has quit IRC | 20:12 | |
| *** mattoliverau has quit IRC | 20:12 | |
| *** jswarren has quit IRC | 20:12 | |
| *** kevinbenton has quit IRC | 20:12 | |
| *** ams_ has quit IRC | 20:12 | |
| *** SlickNik has quit IRC | 20:12 | |
| *** spiffxp has quit IRC | 20:12 | |
| *** lindsayk__ is now known as lindsayk_ | 20:12 | |
| *** flepied has quit IRC | 20:12 | |
| *** rlandy has quit IRC | 20:12 | |
| *** SlickN1k is now known as SlickNik | 20:12 | |
| *** mknawabi has quit IRC | 20:12 | |
| *** stevemar has quit IRC | 20:12 | |
| *** tristanC has quit IRC | 20:12 | |
| *** dkranz_ has joined #openstack-infra | 20:12 | |
| *** dkranz_ has quit IRC | 20:12 | |
| *** dkranz_ has joined #openstack-infra | 20:12 | |
| *** tlbr has joined #openstack-infra | 20:12 | |
| *** spiffxp has joined #openstack-infra | 20:12 | |
| *** mknawabi has joined #openstack-infra | 20:12 | |
| *** vivekd has quit IRC | 20:12 | |
| *** matt6434 has joined #openstack-infra | 20:12 | |
| jeblair | oh, heh | 20:12 |
| *** rlandy has joined #openstack-infra | 20:12 | |
| jeblair | we run it in a timeout too | 20:12 |
| *** adreznec has quit IRC | 20:12 | |
| jeblair | because "git can sometimes get itself infinitely stuck with transient network errors" | 20:12 |
| *** jcooley has joined #openstack-infra | 20:12 | |
| *** rossella_s has joined #openstack-infra | 20:12 | |
| *** flepied has joined #openstack-infra | 20:12 | |
| *** kevinbenton has joined #openstack-infra | 20:12 | |
| *** tristanC has joined #openstack-infra | 20:13 | |
| fungi | we've added a retry loop around zc i think, but not a timeout inside the loop )yet anyway) | 20:13 |
| jeblair | http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/functions.sh#n265 | 20:13 |
| *** stevemar has joined #openstack-infra | 20:13 | |
| *** bnemec has joined #openstack-infra | 20:13 | |
| *** morgabra has joined #openstack-infra | 20:13 | |
| *** devananda has joined #openstack-infra | 20:13 | |
| *** ams_ has joined #openstack-infra | 20:14 | |
| *** jpr has joined #openstack-infra | 20:14 | |
| jeblair | fungi: there are timeout options we can pass to gitpython, so we may be able to improve this in zuul-cloner | 20:14 |
| *** _amrith_ is now known as amrith | 20:15 | |
| *** kgalanov has quit IRC | 20:15 | |
| fungi | ahh, yeah a --timeout option or envvar or something for zc would be swell | 20:15 |
| fungi | --retries might be nice too! ;) | 20:15 |
| *** daemontool has quit IRC | 20:16 | |
| *** adreznec has joined #openstack-infra | 20:16 | |
| *** boris-42 has quit IRC | 20:16 | |
| *** kgalanov has joined #openstack-infra | 20:17 | |
| jeblair | i'm leaning toward saying that what we're seeing is rax network errors causing git to hang; devstack runs are less-affected because of the timeout/retry logic, but zuul-cloner jobs are not because they lack that. there is no evidence that changes in gitpython 1.0.2 are related, but can not rule it out. we should be able to mitigate it by adding timeout/retry options to zuul-cloner. | 20:17 |
| fungi | yeah, the evidence points to that as likely | 20:18 |
| jeblair | so maybe we can switch image types quickly to stop running these jobs on rax and meanwhile add those options to z-c. | 20:18 |
| *** boris-42 has joined #openstack-infra | 20:18 | |
| jeblair | then start using rax again for these jobs once they land | 20:19 |
| jeblair | (and, hopefully, the network gets better at some point) | 20:19 |
| *** daemontool has joined #openstack-infra | 20:19 | |
| *** jsavak has quit IRC | 20:20 | |
| *** daemontool has quit IRC | 20:21 | |
| jeblair | i need to run to lunch; someone should volunteer to take the zuul-cloner hacking task :) | 20:21 |
| fungi | if we're ready to kick the bindep macros into real job-templates i'll start writing that job update (after i look into the python34 failure pabelanger spotted) | 20:21 |
| *** zeih has joined #openstack-infra | 20:21 | |
| *** Sukhdev has joined #openstack-infra | 20:21 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 20:22 |
| mtreinish | pleia2: hah, nice | 20:22 |
| *** boris-42 has quit IRC | 20:24 | |
| openstackgerrit | Khai Do proposed openstack-infra/git-review: WIP: push change with a vote https://review.openstack.org/281504 | 20:24 |
| *** aeng has joined #openstack-infra | 20:25 | |
| *** eil397 has joined #openstack-infra | 20:27 | |
| *** cdent has quit IRC | 20:28 | |
| *** jsavak has joined #openstack-infra | 20:28 | |
| *** annegentle has quit IRC | 20:29 | |
| *** annegentle has joined #openstack-infra | 20:29 | |
| *** annegentle has quit IRC | 20:30 | |
| *** ociuhandu has joined #openstack-infra | 20:31 | |
| pleia2 | mtreinish: my fault, didn't read closey enough /o\ | 20:31 |
| pleia2 | closely too | 20:31 |
| *** sabeen3 has quit IRC | 20:32 | |
| mtreinish | I prefer closey :) | 20:32 |
| pleia2 | hehe | 20:33 |
| *** BobBall has joined #openstack-infra | 20:34 | |
| *** kushal has quit IRC | 20:35 | |
| *** BobBall is now known as Guest30445 | 20:35 | |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Add heat lbaas specific test for v1 and v2 https://review.openstack.org/281354 | 20:35 |
| *** cody-somerville has quit IRC | 20:35 | |
| *** cody-somerville has joined #openstack-infra | 20:35 | |
| *** cody-somerville has quit IRC | 20:35 | |
| *** cody-somerville has joined #openstack-infra | 20:35 | |
| *** yamahata has quit IRC | 20:36 | |
| *** EricGonczer_ has quit IRC | 20:36 | |
| *** notmorga1 is now known as notmorgan | 20:38 | |
| *** notmorgan is now known as Guest8046 | 20:38 | |
| *** Guest8046 is now known as morganfainberg | 20:39 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 20:39 |
| *** morganfainberg is now known as Guest99464 | 20:40 | |
| *** Guest99464 is now known as maelfius | 20:40 | |
| *** maelfius is now known as notmorgan1 | 20:40 | |
| *** notmorgan1 is now known as notmorgan | 20:44 | |
| *** notmorgan has quit IRC | 20:44 | |
| *** notmorgan has joined #openstack-infra | 20:44 | |
| *** notmorgan is now known as morgan | 20:44 | |
| *** morgan is now known as notmorgan | 20:45 | |
| *** e0ne has quit IRC | 20:45 | |
| *** e0ne has joined #openstack-infra | 20:45 | |
| *** EricGonczer_ has joined #openstack-infra | 20:46 | |
| *** notmorgan is now known as captainmorgan | 20:46 | |
| *** captainmorgan is now known as notmorgan | 20:46 | |
| *** sabeen has joined #openstack-infra | 20:46 | |
| *** notmorgan is now known as needscoffee | 20:46 | |
| *** e0ne has quit IRC | 20:46 | |
| *** needscoffee is now known as notmorgan | 20:47 | |
| *** ddieterl_ is now known as ddieterly | 20:47 | |
| *** ldnunes has quit IRC | 20:48 | |
| *** jswarren_ is now known as jswarren | 20:50 | |
| openstackgerrit | Roland Hochmuth proposed openstack/requirements: Upperbound on kafka-python https://review.openstack.org/282053 | 20:53 |
| *** infra-re_ has joined #openstack-infra | 20:53 | |
| *** infra-red has quit IRC | 20:55 | |
| *** harlowja_at_home has joined #openstack-infra | 20:56 | |
| *** rguillebert has quit IRC | 20:57 | |
| *** esp has joined #openstack-infra | 20:59 | |
| *** apoorvad has quit IRC | 20:59 | |
| *** ccrouch1 has joined #openstack-infra | 20:59 | |
| *** tcammann_ has joined #openstack-infra | 21:00 | |
| *** ccrouch has quit IRC | 21:01 | |
| *** sabeen3 has joined #openstack-infra | 21:01 | |
| *** samueldmq1 has joined #openstack-infra | 21:01 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add support for provider network options https://review.openstack.org/280417 | 21:02 |
| *** NobodyCa1 has joined #openstack-infra | 21:02 | |
| *** sridhar_ram1 has quit IRC | 21:03 | |
| *** sabeen has quit IRC | 21:04 | |
| *** sridhar_ram has joined #openstack-infra | 21:04 | |
| *** jsavak has quit IRC | 21:05 | |
| *** tcammann_ has quit IRC | 21:05 | |
| *** esp has quit IRC | 21:05 | |
| *** jsavak has joined #openstack-infra | 21:05 | |
| *** roxanaghe has quit IRC | 21:06 | |
| *** roxanaghe has joined #openstack-infra | 21:06 | |
| *** tcammann_ has joined #openstack-infra | 21:07 | |
| *** NobodyCa1 has quit IRC | 21:07 | |
| *** samueldmq1 has quit IRC | 21:07 | |
| *** EricGonc_ has joined #openstack-infra | 21:07 | |
| *** EricGonczer_ has quit IRC | 21:07 | |
| *** Sukhdev has quit IRC | 21:07 | |
| *** deva_ has joined #openstack-infra | 21:08 | |
| *** tzn has joined #openstack-infra | 21:08 | |
| *** sabeen3 has quit IRC | 21:08 | |
| *** Guest30445 has quit IRC | 21:08 | |
| *** tcammann1 has joined #openstack-infra | 21:08 | |
| *** NobodyCa1 has joined #openstack-infra | 21:09 | |
| *** sabeen has joined #openstack-infra | 21:09 | |
| *** dstanek has quit IRC | 21:09 | |
| *** amrith is now known as _amrith_ | 21:09 | |
| *** dstanek has joined #openstack-infra | 21:10 | |
| *** zeih has quit IRC | 21:10 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Retry twine uploads in a loop https://review.openstack.org/282060 | 21:10 |
| fungi | jeblair: clarkb: as discussed ^ | 21:10 |
| fungi | probably atrociously inefficiently written, so don't hold back picking it apart | 21:11 |
| fungi | though i have at least locally tested the logic introduced there | 21:12 |
| *** tzn has quit IRC | 21:12 | |
| *** esker has quit IRC | 21:13 | |
| *** deva_ has quit IRC | 21:13 | |
| clarkb | looks fine, you added the set +e to handle errors directly | 21:14 |
| *** NobodyCa1 has quit IRC | 21:14 | |
| *** tcammann1 has quit IRC | 21:14 | |
| *** tcammann_ has quit IRC | 21:14 | |
| *** burgerk_ has joined #openstack-infra | 21:15 | |
| *** amitgandhinz has quit IRC | 21:17 | |
| *** amitgandhinz has joined #openstack-infra | 21:18 | |
| *** burgerk has quit IRC | 21:18 | |
| *** jcooley_ has joined #openstack-infra | 21:19 | |
| *** ashleighfarnham has joined #openstack-infra | 21:19 | |
| *** ashleighfarnham has quit IRC | 21:19 | |
| *** burgerk_ has quit IRC | 21:20 | |
| *** bpokorny_ has joined #openstack-infra | 21:20 | |
| *** bpokorn__ has joined #openstack-infra | 21:21 | |
| *** bpokorny_ has quit IRC | 21:21 | |
| *** kgiusti has left #openstack-infra | 21:22 | |
| *** bpokorny has quit IRC | 21:23 | |
| *** ashleighfarnham has joined #openstack-infra | 21:23 | |
| fungi | right, by the time i was done littering everything that might short-circuit the loop too soon, there was basically nothing left aside from assignments and logic | 21:24 |
| *** yamahata has joined #openstack-infra | 21:24 | |
| fungi | so just dropped the ||true masking instead | 21:24 |
| jhesketh | Morning | 21:24 |
| pleia2 | morning jhesketh | 21:24 |
| jhesketh | o/ | 21:25 |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Adding integration test support for lbaas v2 dashboard https://review.openstack.org/275394 | 21:25 |
| *** matt6434 is now known as mattoliverau | 21:26 | |
| fungi | Clint: thanks for the suggestions on 282060. i'm particularly tempted to update it for replacing \!= with -ne since the latter is what i actually wanted (integer comparison not string comparison) though technically both work in this case | 21:28 |
| *** jpr has quit IRC | 21:28 | |
| *** jpr has joined #openstack-infra | 21:29 | |
| fungi | also builtins ftw | 21:29 |
| Clint | fungi: yeah, everything there should work the same.. your syntax just makes me twitch for no good reason | 21:30 |
| fungi | Clint: heh, as it was i was forcing myself to remember to use [ instead of test | 21:30 |
| fungi | too many years of writing posix shell, i get a mild rash when using bashisms | 21:30 |
| openstackgerrit | Ronald Bradford proposed openstack-infra/project-config: Add reno job for oslo.log https://review.openstack.org/282066 | 21:31 |
| Clint | yeah, i know what you mean | 21:31 |
| *** jcooley_ has quit IRC | 21:32 | |
| Clint | i get itchy from export VAR="blah" | 21:32 |
| ianw | austin81: pabelanger makes a pretty good point. it does probably fit better as a puppet module | 21:34 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/project-config: zuul_swift_upload add support for --verbose https://review.openstack.org/269288 | 21:35 |
| *** dkranz_ has quit IRC | 21:36 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Retry twine uploads in a loop https://review.openstack.org/282060 | 21:38 |
| *** ccrouch1 has quit IRC | 21:38 | |
| fungi | Clint: ^ updated with your syntax | 21:38 |
| *** krtaylor has quit IRC | 21:38 | |
| *** e0ne has joined #openstack-infra | 21:38 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/puppet-openstackci: Clarify oscc_file_contents instructions https://review.openstack.org/282010 | 21:39 |
| *** ccrouch has joined #openstack-infra | 21:39 | |
| fungi | also retested locally to make sure it really still worked the way i was expecting | 21:39 |
| *** jcooley_ has joined #openstack-infra | 21:39 | |
| Clint | whee | 21:40 |
| *** ociuhandu has quit IRC | 21:41 | |
| *** sridhar_ram has quit IRC | 21:41 | |
| *** jcooley_ has quit IRC | 21:42 | |
| *** infra-re_ has quit IRC | 21:42 | |
| *** jcooley_ has joined #openstack-infra | 21:42 | |
| *** bpokorn__ has quit IRC | 21:43 | |
| *** austin81 has left #openstack-infra | 21:43 | |
| *** bpokorny has joined #openstack-infra | 21:43 | |
| *** maishsk has joined #openstack-infra | 21:44 | |
| *** bpokorny has quit IRC | 21:44 | |
| *** sridhar_ram has joined #openstack-infra | 21:44 | |
| fungi | if we still see pypi upload failures after 282060 is in place, at least we have some easy values to tune (number of retries, delay between) | 21:45 |
| fungi | dhellmann: ^ | 21:45 |
| *** bpokorny has joined #openstack-infra | 21:45 | |
| dhellmann | fungi : woot! | 21:45 |
| * pleia2 fingers crossed | 21:46 | |
| *** suhsyed has joined #openstack-infra | 21:47 | |
| *** alivigni has quit IRC | 21:47 | |
| *** infra-red has joined #openstack-infra | 21:47 | |
| sdake | fungi quick question, suhsyed is getting an invalid token when trying to register his ocrrect email adresss with gerrit | 21:48 |
| *** jcooley_ has quit IRC | 21:48 | |
| *** gildub has joined #openstack-infra | 21:48 | |
| sdake | suhsyed can you explain in a little more detail? | 21:48 |
| *** jcooley_ has joined #openstack-infra | 21:48 | |
| sdake | he gets an email from gerrit to change his email | 21:49 |
| sdake | he clicks that | 21:49 |
| sdake | then invalid token | 21:49 |
| markvan | AJaeger: hi, thx for your help with my infra patches. Was wondering about the usage of the PROJECTS="openstack/blah $PROJECTS". What is this used for? I see it being used in the yaml jabs files, but jobs that use pre/post hook scripts don't seem be using it. | 21:49 |
| clarkb | sdake: is this with microsoft outlook? | 21:50 |
| clarkb | sdake: there is a known issue with how it interprets the gerrit url | 21:50 |
| *** |-paul-| has quit IRC | 21:51 | |
| *** krtaylor has joined #openstack-infra | 21:51 | |
| clarkb | if you look at the email source and copy pasta directly that should work | 21:51 |
| *** jcoufal has quit IRC | 21:51 | |
| sdake | clarkb thanks a bunch | 21:51 |
| clarkb | sdake: fwiw I believ eour investigating resulted in "gerrit is technically correct and doesn't do anything the rfcs forbid, outlook is just being silly" | 21:52 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add support for provider network options https://review.openstack.org/280417 | 21:54 |
| *** maishsk has quit IRC | 21:55 | |
| *** maishsk has joined #openstack-infra | 21:57 | |
| *** chlong has quit IRC | 21:57 | |
| *** chlong_ has joined #openstack-infra | 21:58 | |
| *** amitgandhinz has quit IRC | 21:59 | |
| *** EricGonc_ has quit IRC | 21:59 | |
| *** apoorvad has joined #openstack-infra | 21:59 | |
| sdake | clarkb thanks for htat, super helpful :) | 22:00 |
| *** EricGonczer_ has joined #openstack-infra | 22:00 | |
| *** krtaylor has quit IRC | 22:02 | |
| *** sarob_ has joined #openstack-infra | 22:02 | |
| *** asettle has joined #openstack-infra | 22:02 | |
| *** salv-orlando has joined #openstack-infra | 22:03 | |
| asselin | dstufft, are you around? I have a bandersnatch issue I hope you can help me with: http://paste.openstack.org/show/487500/ | 22:04 |
| *** apoorvad has quit IRC | 22:04 | |
| *** thiagop has quit IRC | 22:04 | |
| rhedlind | One of my CI tests started failing in the devstack setup stage yesterday as it is trying to setup a venv. Would appreciate if anyone has any input as to what might have happened. The CI environment hasn't changed. Log excerpt http://paste.openstack.org/show/487498/ | 22:04 |
| dstufft | asselin: hi | 22:04 |
| *** _sarob has quit IRC | 22:05 | |
| *** sc68cal has quit IRC | 22:05 | |
| *** salv-orl_ has quit IRC | 22:05 | |
| asselin | dstufft, Hi, we're having issues last week with packages failing to update. I pasted a more current log file of new failures. (I need to dig up the ones from last week). jeblair said you might be able to help me. | 22:05 |
| *** thorst has quit IRC | 22:05 | |
| dstufft | bleh | 22:05 |
| dstufft | looks like that got deleted in AWS but failed to record the delete in PyPI | 22:06 |
| *** cznewt has quit IRC | 22:06 | |
| *** sarob_ has quit IRC | 22:06 | |
| dstufft | I hate legacy PyPI | 22:06 |
| *** thorst has joined #openstack-infra | 22:06 | |
| clarkb | rhedlind: are you reusing machines? | 22:06 |
| *** cznewt has joined #openstack-infra | 22:07 | |
| sdake | clarkb so suhsyed got his email matching his launchpad id, but it still wants to log him in from launchpad as from gmail | 22:07 |
| sdake | the issue is git review -s isn't working for him | 22:07 |
| *** asettle has quit IRC | 22:07 | |
| rhedlind | clarkb: no | 22:07 |
| clarkb | rhedlind: ok that throws out one theory (preexisting old pbr that didn't get updated) | 22:07 |
| clarkb | sdake: so matching the launchpad id isn't what needs to happen... | 22:08 |
| sdake | he is matching his gerrit-id | 22:08 |
| clarkb | sdake: your primary email address in gerrit must match your primary email address on your foundation profile | 22:08 |
| clarkb | sdake: that does not matter | 22:08 |
| clarkb | sdake: can you paste the output of git review -s -v? | 22:08 |
| dstufft | asselin: try now | 22:08 |
| sdake | clarkb he is checking foundation profile and will do git review -s paste in a moment | 22:08 |
| clarkb | sdake: with -v too | 22:09 |
| clarkb | adds debug output | 22:09 |
| *** ganso has quit IRC | 22:09 | |
| *** thorst_ has joined #openstack-infra | 22:10 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test overcloud SSL https://review.openstack.org/281988 | 22:10 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Use netiso in the ha job https://review.openstack.org/273424 | 22:10 |
| *** thorst has quit IRC | 22:10 | |
| *** wwwbukolaycom has joined #openstack-infra | 22:10 | |
| asselin | dstufft, more fun: http://paste.openstack.org/show/487503/ | 22:12 |
| *** thorst has joined #openstack-infra | 22:12 | |
| clarkb | rhedlind: Download error on http://pypi.dfw.openstack.org/simple/: [Errno 101] Network is unreachable is the problem | 22:12 |
| *** infra-red has quit IRC | 22:12 | |
| clarkb | rhedlind: we deleted that mirror, they are not intended for public consumption | 22:12 |
| *** rkukura has joined #openstack-infra | 22:13 | |
| openstackgerrit | Khai Do proposed openstack-infra/git-review: Allow git review to set votes https://review.openstack.org/281504 | 22:13 |
| dstufft | asselin: try now | 22:13 |
| asselin | dstufft, issues from last week: http://paste.openstack.org/show/487504/ | 22:13 |
| *** jsavak has quit IRC | 22:13 | |
| *** thorst_ has quit IRC | 22:14 | |
| *** doug-fis_ has joined #openstack-infra | 22:15 | |
| *** sridhar_ram has quit IRC | 22:15 | |
| *** krtaylor has joined #openstack-infra | 22:15 | |
| clarkb | sdake: also what setup docs are you following? are you using http://docs.openstack.org/infra/manual/developers.html#account-setup ? | 22:16 |
| openstackgerrit | Khai Do proposed openstack-infra/git-review: Update gerrit version for testing https://review.openstack.org/282018 | 22:16 |
| *** thorst has quit IRC | 22:16 | |
| sdake | clarkb that is what he is following yes | 22:16 |
| rhedlind | clarkb: so how come the script uses that? I assume the latest devstack is pulled down as part of the tempest setup? | 22:16 |
| clarkb | ok I don't think that doc says anything about making gerrit userid and lp id match /me dobule checks | 22:16 |
| *** eharney has quit IRC | 22:16 | |
| clarkb | rhedlind: devstack should use pypi.python.org | 22:16 |
| clarkb | rhedlind: by default | 22:17 |
| openstackgerrit | Khai Do proposed openstack-infra/git-review: Allow git review to set votes https://review.openstack.org/281504 | 22:17 |
| sdake | does the gerrit id userid and lp id need to match? | 22:17 |
| asselin | dstufft, DEBUG: Getting https://pypi.python.org/packages/source/r/rest_client/rest_client-0.2.0.tar.gz (serial None) | 22:17 |
| asselin | fails | 22:17 |
| clarkb | sdake: "You can enter your Launchpad username here, or something else if you want." is what it says | 22:17 |
| sdake | cool | 22:17 |
| clarkb | sdake: no thats what I am saying,but you said you just changed things to make them match | 22:17 |
| clarkb | so I am confused where that is coming from | 22:17 |
| sdake | he had an old invalid email | 22:18 |
| *** sridhar_ram has joined #openstack-infra | 22:18 | |
| sdake | he is afk atm so I can't ask him more :) | 22:18 |
| clarkb | ok | 22:18 |
| sdake | he will be back | 22:18 |
| *** doug-fish has quit IRC | 22:18 | |
| *** maishsk has quit IRC | 22:18 | |
| sdake | clarkb we got his email to match reality ;) | 22:18 |
| sdake | that was painful (the outlook thing) | 22:18 |
| sdake | our workaround was to forward to gmail and click through there | 22:19 |
| *** maishsk has joined #openstack-infra | 22:19 | |
| *** doug-fis_ has quit IRC | 22:19 | |
| sdake | had trouble finding view sourcein new version of office on his end | 22:19 |
| sdake | outlook 2010 has it but not 2015 from what i can tell | 22:19 |
| clarkb | ok so it was just email that was made to match | 22:20 |
| sdake | his login id in his linux machine matches his gerrit user id though | 22:20 |
| clarkb | gotcha | 22:20 |
| sdake | that was problem #1 | 22:20 |
| sdake | now onto problem number two :) | 22:20 |
| sdake | i'll get back to you on that | 22:20 |
| clarkb | ok | 22:20 |
| *** Sukhdev has joined #openstack-infra | 22:20 | |
| sdake | all my stuff matches, makes life so much simpler ;-) | 22:21 |
| *** apoorvad has joined #openstack-infra | 22:21 | |
| sdake | I guess I look boring at a party ;) | 22:21 |
| sdake | to also fix he had to delete his old invalid email from launchpad, as it kept wanting to log into gerrit as that even though it was set as secondary | 22:22 |
| dstufft | asselin: try now | 22:22 |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: Keystone-only uwsgi job https://review.openstack.org/264991 | 22:23 |
| asselin | 2016-02-18 22:23:04,090 DEBUG: Getting https://pypi.python.org/packages/2.7/r/rest_client/rest_client-0.1.0-py2-none-any.whl (serial None) | 22:23 |
| asselin | 2016-02-18 22:23:04,203 ERROR: Error syncing package: rest_client@1955908 | 22:23 |
| clarkb | sdake: git review -s works now? | 22:23 |
| sdake | clarkb he is still afk | 22:23 |
| clarkb | ok I don't think doing that was or is necessary fwiw | 22:23 |
| asselin | dstufft, is there something else I should be doing to avoid this? | 22:23 |
| *** mriedem has quit IRC | 22:24 | |
| *** rguillebert has joined #openstack-infra | 22:25 | |
| dstufft | asselin: no | 22:25 |
| dstufft | it's a pypi problem | 22:25 |
| dstufft | I have to fix each file | 22:25 |
| rhedlind | clarkb: ok, I will rebuild the image used for this test and hopefully the new image will use the correct address. Thanks! | 22:25 |
| asselin | dstufft, :( | 22:26 |
| *** kozhukalov has quit IRC | 22:26 | |
| *** harlowja_at_home has quit IRC | 22:26 | |
| dstufft | asselin: try now | 22:28 |
| asselin | dstufft, success! | 22:28 |
| dstufft | great | 22:28 |
| asselin | dstufft, but now those other packages...I had to hack my todo file to get our system running again. | 22:28 |
| *** thorst has joined #openstack-infra | 22:29 | |
| dstufft | I fixed all the ones that were in that paste | 22:29 |
| asselin | ok, so what's the best way to get back in sync. Redo a fully mirror sync? or restore that previous (bad) todo file? | 22:29 |
| dstufft | I'd just redo a full sync | 22:30 |
| *** thorst has quit IRC | 22:30 | |
| *** thorst has joined #openstack-infra | 22:31 | |
| *** thorst has quit IRC | 22:31 | |
| asselin | ok will do. thank you. I'll start it now. hopefully there's nothing else broken. | 22:31 |
| *** thorst has joined #openstack-infra | 22:31 | |
| craige | o/ | 22:31 |
| *** EricGonczer_ has quit IRC | 22:33 | |
| *** tpsilva has quit IRC | 22:36 | |
| *** sabeen has quit IRC | 22:37 | |
| *** thorst has quit IRC | 22:38 | |
| *** thorst has joined #openstack-infra | 22:38 | |
| *** camunoz has joined #openstack-infra | 22:39 | |
| *** ociuhandu has joined #openstack-infra | 22:41 | |
| *** thorst has quit IRC | 22:43 | |
| *** baoli has quit IRC | 22:45 | |
| *** baoli has joined #openstack-infra | 22:45 | |
| asselin | dstufft, 2016-02-18 22:45:39,330 DEBUG: Getting https://pypi.python.org/packages/any/p/pyriscope/pyriscope-1.0.6.macosx-10.6-intel.tar.gz (serial None) | 22:46 |
| *** lucas-dinner has quit IRC | 22:47 | |
| *** sabeen1 has joined #openstack-infra | 22:47 | |
| *** swat30 has joined #openstack-infra | 22:48 | |
| *** edmondsw has quit IRC | 22:49 | |
| asselin | dstufft, http://paste.openstack.org/show/487506/ | 22:49 |
| *** sabeen2 has joined #openstack-infra | 22:49 | |
| *** yamamoto has quit IRC | 22:49 | |
| jeblair | clarkb, fungi: i'm working on the zuul-cloner timeout changes | 22:50 |
| clarkb | jeblair: ok | 22:51 |
| *** sabeen1 has quit IRC | 22:51 | |
| *** lucasagomes has joined #openstack-infra | 22:53 | |
| *** mknawabi has quit IRC | 22:54 | |
| *** mknawabi has joined #openstack-infra | 22:54 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Fixed downstream_ext comparison conditions https://review.openstack.org/274104 | 22:54 |
| dstufft | asselin: try now | 22:58 |
| *** maishsk has quit IRC | 22:59 | |
| *** gildub has quit IRC | 22:59 | |
| asselin | dstufft, http://paste.openstack.org/show/487508/ | 23:00 |
| mordred | clarkb, fungi: back online - anything I should look at? | 23:00 |
| *** xyang1 has quit IRC | 23:01 | |
| *** sabeen2 has quit IRC | 23:02 | |
| *** jpr has quit IRC | 23:02 | |
| *** suhsyed has quit IRC | 23:05 | |
| *** thorst has joined #openstack-infra | 23:05 | |
| *** jtomasek has quit IRC | 23:05 | |
| *** e0ne has quit IRC | 23:06 | |
| *** gordc has quit IRC | 23:06 | |
| dstufft | asselin: try again | 23:06 |
| asselin | dstufft, 2016-02-18 23:07:01,956 DEBUG: Getting https://pypi.python.org/packages/source/d/dndclib/dndclib-0.1.tar.gz (serial None) | 23:07 |
| *** keedya has quit IRC | 23:07 | |
| clarkb | mordred: if you are in a place where you can make sure oscc 1.15.0 is installed on nodepool.o.o and restart nodepoold and the nodepool-builder that would be helpful | 23:08 |
| dstufft | asselin: does that have an associated error? | 23:08 |
| dstufft | that file isn't hitting the same bug | 23:08 |
| dstufft | maybe a different bug | 23:08 |
| clarkb | mordred: otherwise jeblair is updated zuul cloner to timout git clones and fungi has a change to improve our publishing job | 23:08 |
| mordred | clarkb: kk | 23:08 |
| clarkb | https://review.openstack.org/#/c/282060/ has the needed reviews so you can ignore it if yo uwant | 23:08 |
| *** dingyichen has joined #openstack-infra | 23:08 | |
| asselin | dstufft, http://paste.openstack.org/show/487510/ | 23:08 |
| *** dingyichen has quit IRC | 23:09 | |
| dstufft | oh | 23:09 |
| dstufft | nevermind | 23:09 |
| dstufft | it was cached | 23:09 |
| dstufft | asselin: ok, try again | 23:10 |
| asselin | dstufft, ok | 23:11 |
| *** sputnik13 has quit IRC | 23:11 | |
| asselin | dstufft, ok will now delete the status file and restart a full mirror sync. | 23:11 |
| openstackgerrit | Merged openstack-infra/project-config: Keep Gerrit ACL lines deduplicated https://review.openstack.org/281562 | 23:12 |
| *** eric_lopez has quit IRC | 23:12 | |
| *** sputnik13 has joined #openstack-infra | 23:13 | |
| *** thorst has quit IRC | 23:13 | |
| *** dingyichen has joined #openstack-infra | 23:14 | |
| *** thorst has joined #openstack-infra | 23:14 | |
| *** xiexs___ has quit IRC | 23:16 | |
| *** regXboi has quit IRC | 23:17 | |
| *** yamamoto has joined #openstack-infra | 23:17 | |
| *** dingyichen has quit IRC | 23:17 | |
| *** thorst has quit IRC | 23:17 | |
| *** thorst has joined #openstack-infra | 23:17 | |
| *** ddieterly has quit IRC | 23:18 | |
| *** thorst has quit IRC | 23:18 | |
| *** dingyichen has joined #openstack-infra | 23:18 | |
| clarkb | "Labels with value=0 not working in 2.12 or "open source" master (working in Google Code review)" | 23:18 |
| clarkb | that is a fun thread on gerrit list | 23:18 |
| *** thorst has joined #openstack-infra | 23:18 | |
| *** jamielennox is now known as jamielennox|away | 23:20 | |
| *** jcooley_ has quit IRC | 23:21 | |
| pabelanger | TIL: 2 gerrit master branches | 23:23 |
| *** thorst has quit IRC | 23:23 | |
| clarkb | pabelanger: I think it is more that the google source is internal and secret | 23:23 |
| clarkb | and google pushes up and pulls down as necessary | 23:23 |
| pabelanger | clarkb: Ya, not sure why there isn't more outrage on that | 23:23 |
| clarkb | but they rip out much of the internals to replace it with googly things | 23:24 |
| *** tzn has joined #openstack-infra | 23:24 | |
| clarkb | (that thread talks about this briefly where googles search thing queries differently than lucene) | 23:24 |
| pabelanger | ya, seen that | 23:24 |
| mordred | clarkb: ok. nodepool restarted - logs look good | 23:25 |
| clarkb | mordred: nodepool-builder too? | 23:26 |
| mordred | yah | 23:26 |
| clarkb | woot | 23:26 |
| clarkb | in theory this makes rax uploads much more betterer | 23:26 |
| clarkb | and the main process can restart when images are in a building state | 23:26 |
| mordred | woot | 23:26 |
| *** _sarob has joined #openstack-infra | 23:28 | |
| *** cody-somerville has quit IRC | 23:28 | |
| *** cody-somerville has joined #openstack-infra | 23:28 | |
| *** cody-somerville has quit IRC | 23:28 | |
| *** cody-somerville has joined #openstack-infra | 23:28 | |
| asselin | clarkb, that is nice | 23:28 |
| *** Sukhdev has quit IRC | 23:29 | |
| *** yamamoto has quit IRC | 23:31 | |
| *** ganso has joined #openstack-infra | 23:31 | |
| notmorgan | ok. i am NOT liking the fact that every nick color just changed on me in weechat... escept clarkb and dstufft. oddly, they are the same :P | 23:32 |
| *** Qiming has joined #openstack-infra | 23:33 | |
| Clint | go on | 23:33 |
| notmorgan | Clint: nope. | 23:33 |
| notmorgan | Clint: /me puts soapbox away. | 23:33 |
| *** dims_ has joined #openstack-infra | 23:36 | |
| pleia2 | tough times over there, notmorgan :P | 23:36 |
| notmorgan | pleia2: right?! | 23:37 |
| *** dimtruck is now known as zz_dimtruck | 23:37 | |
| notmorgan | pleia2: oh but your nick is now a lovely purple color | 23:37 |
| notmorgan | pleia2: so i approve of that | 23:37 |
| openstackgerrit | Merged openstack-infra/project-config: Create ansible-role-ssh https://review.openstack.org/281867 | 23:37 |
| *** dims has quit IRC | 23:38 | |
| clarkb | I am disappointed my nick is that boring | 23:38 |
| dstufft | notmorgan: I hateee when colors change on people | 23:38 |
| dstufft | it's kind of crazy that it makes that much difference | 23:38 |
| notmorgan | dstufft: i know. it's frustrating, because i expect everyone's name to be pretty much the same color - since there are ~100 colors i use for nicks, there is little overlap | 23:39 |
| notmorgan | dstufft: s/same color/ concistent | 23:39 |
| pleia2 | purple is nice | 23:39 |
| *** tzn has quit IRC | 23:39 | |
| notmorgan | dstufft: the sad part is.. if i was using irssi or something w/o color, i'd actually read names vs. rely on colors :P | 23:40 |
| mordred | clarkb and pleia2 are the same color for me | 23:40 |
| mordred | notmorgan: you are the same color as openstackgerrit | 23:40 |
| dstufft | the worst is when two people with similar names get the same color | 23:40 |
| pleia2 | everyone is the same color for me, except when they say my nickname, then they are pink <3 | 23:40 |
| mordred | dstufft: ++ | 23:40 |
| mtreinish | notmorgan: everyone looks green to me... | 23:40 |
| sc` | every nick looks white to me | 23:41 |
| sc` | <- irssi | 23:41 |
| notmorgan | mordred: you're now a cyan color... the only person in all the channels i have seen so far with it :P | 23:41 |
| notmorgan | pleia2: go go weechat? | 23:41 |
| pleia2 | notmorgan: I like irssi | 23:41 |
| dstufft | mordred is dark red | 23:41 |
| dstufft | I just use textual though | 23:42 |
| dstufft | I'm too lazy to console IRC | 23:42 |
| pleia2 | but I also like Perl, and all my scripts for irssi are written in it already, so changing is hard | 23:42 |
| *** jlk has joined #openstack-infra | 23:42 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add timeout/retries to zuul cloner https://review.openstack.org/282099 | 23:42 |
| notmorgan | dstufft: i used to use textual, moved to weechat when i was tired of trying to make it work on multiple clinets at the same time w/o missing messages | 23:42 |
| *** bhunter_ has quit IRC | 23:43 | |
| notmorgan | pleia2: fair enough, i think weechat aslo supports perl fwiw. | 23:43 |
| dstufft | I rarely use multiple clients, I am the easiest use case | 23:43 |
| notmorgan | pleia2: but irssi is good too | 23:43 |
| jeblair | clarkb, fungi, mordred: https://review.openstack.org/282099 | 23:43 |
| * notmorgan just needs to move from screen to tmux | 23:43 | |
| dstufft | I sometimes think I about trying to use weechat + glowing bear + textual | 23:43 |
| notmorgan | dstufft: well i also now only have 2 clients sometimes. | 23:43 |
| notmorgan | dstufft: ios or android phone? [i assume glowing bear for phone] | 23:44 |
| dstufft | ios | 23:44 |
| openstackgerrit | Paul Van Eck proposed openstack-infra/puppet-refstack: Make puppet pull releases from Pypi https://review.openstack.org/281737 | 23:44 |
| notmorgan | dstufft: cause i just tried glowing bear...and on android it's not as good as the real weechat client | 23:44 |
| jeblair | pleia2: i like irssi too | 23:44 |
| dstufft | I own so many apple devices | 23:44 |
| clarkb | the one feature that keeps me on weechat is smartfilters | 23:44 |
| dstufft | (if you count my entire family) | 23:44 |
| dstufft | it'd be hard to swap to something other than ios | 23:45 |
| dstufft | well "hard" | 23:45 |
| notmorgan | dstufft: oh totally, just asking cause i hear glowing bear on ios is a bout as good as it gets. i used to use irccloud but got tired of the outages | 23:46 |
| notmorgan | and glowing bear isn't bad by any stretch | 23:46 |
| notmorgan | but on android, glowing bear didn't work well for me | 23:47 |
| *** openstackgerrit has quit IRC | 23:47 | |
| *** openstackgerrit_ is now known as openstackgerrit | 23:47 | |
| *** openstackgerrit_ has joined #openstack-infra | 23:47 | |
| * notmorgan cries for openstackgerrit bot. | 23:47 | |
| notmorgan | poor bot. | 23:48 |
| *** openstackgerrit_ is now known as openstackgerrit | 23:48 | |
| *** openstackgerrit_ has joined #openstack-infra | 23:49 | |
| dmsimard | Glowing bear works really well on my Nexus 6P. | 23:49 |
| dmsimard | Moved my bouncer from ZNC to Weechat, works great. | 23:49 |
| dmsimard | and I host the glowing bear interface on the same server Weechat runs on so anywhere I have a browser I can hop on. | 23:49 |
| dmsimard | best discovery in a while | 23:50 |
| *** gildub has joined #openstack-infra | 23:50 | |
| pleia2 | bot just needed a config update :) | 23:50 |
| fungi | i think i forgot to say i was going to dinner, but am back now | 23:51 |
| *** chlong_ has quit IRC | 23:52 | |
| *** _sarob has quit IRC | 23:52 | |
| *** _sarob has joined #openstack-infra | 23:53 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:53 | |
| mordred | fungi: I, for one, am shocked, shocked I tell you | 23:54 |
| *** openstackgerrit_ has quit IRC | 23:55 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add timeout/retries to zuul cloner https://review.openstack.org/282102 | 23:55 |
| fungi | notmorgan: tmux is pretty awesome. as a long-time gnu screen user, i made the switch a few years ago and was quite pleased. the main impetus for switching was the configurable activity highlights | 23:55 |
| * notmorgan welcomes openstackgerrit back to life! | 23:56 | |
| *** openstackgerrit_ has joined #openstack-infra | 23:56 | |
| *** rbrndt_ has quit IRC | 23:56 | |
| notmorgan | what if whenever fungi is not on IRC he's just getting a config update ... what if ... we are all just irc bots *whoa* :P | 23:57 |
| fungi | mordred: yeah, taco thursdays at the bad bean. unsurprisingly, while did have a couple pork belly tacos, i really went for the beer | 23:57 |
| fungi | notmorgan: it's a variant of solipsism where the simplest explanation of reality is that everyone else is just an irc bot | 23:58 |
| * notmorgan is reminded to get beer from cabinet and drink beer with late lunch | 23:58 | |
| * fungi hopes this is a dedicated "beer cabinet" | 23:58 | |
| notmorgan | fungi: eh, it might be mostly dedicated to beer | 23:59 |
| *** sputnik13 has quit IRC | 23:59 | |
| notmorgan | there are other things in there... like a bottle of wine, some ginger drinks and maybe a soda | 23:59 |
| *** ashleighfarnham has quit IRC | 23:59 | |
| fungi | "beer mixers" | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!