| *** sdake_ has joined #openstack-infra | 00:00 | |
| *** ajmiller_ has quit IRC | 00:00 | |
| bkero | I think apache reload will serve old requests with old certs and new requests with new certs. And as long as you don't have long-running transactions or state that matters on something like a LB it's okay | 00:00 |
|---|---|---|
| bkero | but I imagine clients can have state that gets pretty grumpy if the server cert changes during a live connection | 00:01 |
| clarkb | ya I don't have details on why the new cert broke things today | 00:01 |
| clarkb | and it was broken for new connections not existing ones | 00:01 |
| bkero | So...whenever my letsencrypt upgrade cert script runs, I need to cat the entire chain into the cert file to be served | 00:02 |
| fungi | apache needs a restart to load new certs, reload won't do it last i checked. but anyway i expect that wasn't the root of the problem | 00:02 |
| *** zhurong has quit IRC | 00:02 | |
| bkero | Otherwise there's an incomplete chain to my end cert | 00:02 |
| *** sdake has quit IRC | 00:02 | |
| bkero | super lazymode script: http://paste.openstack.org/show/505759/ | 00:03 |
| *** earlephilhower has quit IRC | 00:03 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Add tracing flag to dib-buildimage-atomic https://review.openstack.org/321900 | 00:03 |
| madhuvishy | zaro: yup! I understand it needs to be +2-ed :) It's currently blocking some of my work at Wikimedia on automating maven jar releases, so was wondering if someone could help it get merged sooner! Thank you :) | 00:04 |
| fungi | i wish i'd had an opportunity to point openssl s_client at it while broken, but i missed the excitement so no idea what was wrong with it really | 00:04 |
| *** nelsnelson has joined #openstack-infra | 00:11 | |
| *** ddieterly is now known as ddieterly[away] | 00:12 | |
| *** banix has joined #openstack-infra | 00:12 | |
| *** dims has quit IRC | 00:13 | |
| *** vhosakot has quit IRC | 00:14 | |
| *** SumitNaiksatam has quit IRC | 00:14 | |
| *** mtanino has joined #openstack-infra | 00:16 | |
| *** denisra has quit IRC | 00:17 | |
| *** Jeffrey4l has joined #openstack-infra | 00:17 | |
| *** _sarob has quit IRC | 00:18 | |
| *** vhosakot has joined #openstack-infra | 00:20 | |
| *** vhosakot has quit IRC | 00:21 | |
| *** ddieterly[away] is now known as ddieterly | 00:21 | |
| openstackgerrit | Sachi King proposed openstack-dev/pbr: Restore warnerrors behavior https://review.openstack.org/229951 | 00:22 |
| *** xarses has joined #openstack-infra | 00:22 | |
| *** vhosakot has joined #openstack-infra | 00:22 | |
| *** baoli has quit IRC | 00:22 | |
| *** baoli has joined #openstack-infra | 00:23 | |
| *** dimtruck is now known as zz_dimtruck | 00:24 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add support for xenial-backports https://review.openstack.org/321904 | 00:24 |
| pabelanger | fungi: clarkb: jeblair: xenial-backports are a thing now^ | 00:24 |
| fungi | yay progress! | 00:25 |
| *** matrohon has quit IRC | 00:25 | |
| *** r-mibu has quit IRC | 00:26 | |
| *** r-mibu has joined #openstack-infra | 00:26 | |
| *** pvaneck has quit IRC | 00:27 | |
| *** markvoelker has joined #openstack-infra | 00:29 | |
| *** Qiming has quit IRC | 00:29 | |
| *** bpokorny has quit IRC | 00:31 | |
| *** cody-somerville has joined #openstack-infra | 00:33 | |
| *** nadya has joined #openstack-infra | 00:34 | |
| *** markvoelker has quit IRC | 00:36 | |
| *** zz_dimtruck is now known as dimtruck | 00:37 | |
| *** mixos has joined #openstack-infra | 00:38 | |
| *** nadya has quit IRC | 00:38 | |
| *** ddieterly is now known as ddieterly[away] | 00:40 | |
| *** mtanino has quit IRC | 00:41 | |
| *** shashank_hegde has quit IRC | 00:42 | |
| JayF | fungi: another fun email responder gerrit case. Someone subscribed to all of nova is apparently autoresponding directly to me, and it's from the same organization/domain as the other problem from the other day. | 00:45 |
| fungi | JayF: another "i don't work here" autoresponder? let me know the e-mail address and i'll null it out | 00:47 |
| JayF | fungi: not an "i don't work here", it's an "I'm on vacation" responder, but went directly to me. I think the other day you indicated that would be bad behavior by that domain (to send to me instead of to gerrit) | 00:47 |
| fungi | JayF: er, yeah or responding to it at all for a number of reasons | 00:48 |
| *** vhosakot has quit IRC | 00:50 | |
| *** ddieterly[away] is now known as ddieterly | 00:54 | |
| *** cody-somerville has quit IRC | 00:56 | |
| *** dims has joined #openstack-infra | 00:56 | |
| *** ddieterly is now known as ddieterly[away] | 00:57 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: move puppet4 jobs into check pipeline https://review.openstack.org/321837 | 00:58 |
| *** ddieterly[away] is now known as ddieterly | 01:00 | |
| *** gyee has quit IRC | 01:03 | |
| *** asettle has joined #openstack-infra | 01:10 | |
| *** zhurong has joined #openstack-infra | 01:11 | |
| *** esker has joined #openstack-infra | 01:15 | |
| *** Daisy has joined #openstack-infra | 01:16 | |
| *** Daisy_ has joined #openstack-infra | 01:17 | |
| *** asettle has quit IRC | 01:17 | |
| *** esker has quit IRC | 01:19 | |
| *** Daisy has quit IRC | 01:20 | |
| *** rhallisey has quit IRC | 01:21 | |
| *** gomarivera has joined #openstack-infra | 01:22 | |
| *** vhosakot has joined #openstack-infra | 01:23 | |
| *** kzaitsev_mb has quit IRC | 01:24 | |
| *** mixos has quit IRC | 01:27 | |
| *** mixos has joined #openstack-infra | 01:28 | |
| *** baoli has quit IRC | 01:29 | |
| *** vhosakot has quit IRC | 01:30 | |
| *** baoli has joined #openstack-infra | 01:31 | |
| *** gomarivera has quit IRC | 01:34 | |
| *** claudiub has joined #openstack-infra | 01:35 | |
| *** Daisy_ has quit IRC | 01:35 | |
| *** Daisy has joined #openstack-infra | 01:36 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/nodepool: Python 3 Fix: Use six.ByesIO https://review.openstack.org/321918 | 01:36 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/nodepool: Python 3 Fix: cmp -> key function https://review.openstack.org/321919 | 01:36 |
| *** yanyanhu has joined #openstack-infra | 01:38 | |
| *** Qiming has joined #openstack-infra | 01:38 | |
| *** claudiub|2 has quit IRC | 01:39 | |
| *** yamahata has quit IRC | 01:42 | |
| *** sdake has joined #openstack-infra | 01:45 | |
| *** hichihara has joined #openstack-infra | 01:45 | |
| *** SumitNaiksatam has joined #openstack-infra | 01:47 | |
| *** amitgandhinz has joined #openstack-infra | 01:47 | |
| *** Daisy has quit IRC | 01:48 | |
| *** sdake_ has quit IRC | 01:48 | |
| *** Daisy has joined #openstack-infra | 01:48 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/nodepool: Python 3 fix: Use new-style raise syntax https://review.openstack.org/321926 | 01:49 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/nodepool: Python 3 Fixes: Encode config write in tests https://review.openstack.org/321927 | 01:49 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/nodepool: Python 3 fixes: dict.iteritems https://review.openstack.org/321928 | 01:49 |
| *** esker has joined #openstack-infra | 01:51 | |
| *** amrith is now known as _amrith_ | 01:51 | |
| *** _amrith_ is now known as amrith | 01:52 | |
| *** amitgandhinz has quit IRC | 01:53 | |
| *** Daisy has quit IRC | 01:53 | |
| *** esker has quit IRC | 01:55 | |
| *** Daisy has joined #openstack-infra | 01:57 | |
| *** claudiub|2 has joined #openstack-infra | 02:01 | |
| *** Apoorva has quit IRC | 02:02 | |
| *** amrith is now known as _amrith_ | 02:04 | |
| *** claudiub has quit IRC | 02:04 | |
| *** _amrith_ is now known as amrith | 02:06 | |
| *** bpokorny has joined #openstack-infra | 02:06 | |
| *** Daisy has quit IRC | 02:07 | |
| *** Daisy has joined #openstack-infra | 02:07 | |
| *** kushal has quit IRC | 02:07 | |
| *** amrith is now known as _amrith_ | 02:08 | |
| *** bpokorny_ has joined #openstack-infra | 02:08 | |
| *** _amrith_ is now known as amrith | 02:09 | |
| *** ddieterly is now known as ddieterly[away] | 02:10 | |
| *** amrith is now known as _amrith_ | 02:11 | |
| *** bpokorny has quit IRC | 02:11 | |
| *** _amrith_ is now known as amrith | 02:11 | |
| *** bpokorny_ has quit IRC | 02:12 | |
| *** hparekh has quit IRC | 02:15 | |
| *** tlian has quit IRC | 02:18 | |
| *** nwkarsten has joined #openstack-infra | 02:19 | |
| *** Daisy_ has joined #openstack-infra | 02:25 | |
| *** sdake_ has joined #openstack-infra | 02:26 | |
| *** amrith is now known as _amrith_ | 02:27 | |
| *** sdake has quit IRC | 02:28 | |
| *** Sam-I-Am has quit IRC | 02:28 | |
| *** Daisy has quit IRC | 02:29 | |
| *** _amrith_ is now known as amrith | 02:29 | |
| *** Daisy has joined #openstack-infra | 02:32 | |
| *** antonym has quit IRC | 02:32 | |
| *** markvoelker has joined #openstack-infra | 02:32 | |
| *** nwkarsten has quit IRC | 02:32 | |
| *** Madasi has quit IRC | 02:32 | |
| *** Daisy_ has quit IRC | 02:33 | |
| *** cody-somerville has joined #openstack-infra | 02:34 | |
| *** nwkarsten has joined #openstack-infra | 02:36 | |
| *** openstackgerrit has quit IRC | 02:36 | |
| *** hockeynut has quit IRC | 02:36 | |
| *** markvoelker has quit IRC | 02:36 | |
| *** hockeynut has joined #openstack-infra | 02:37 | |
| *** erikwilson has quit IRC | 02:37 | |
| mwhahaha | so where'd gerrit go? | 02:38 |
| *** Madasi has joined #openstack-infra | 02:38 | |
| yanyanhu | it's broken? | 02:39 |
| mwhahaha | seems down? | 02:40 |
| *** erikmwilson has joined #openstack-infra | 02:40 | |
| *** nwkarsten has quit IRC | 02:40 | |
| *** openstackgerrit has joined #openstack-infra | 02:42 | |
| ianw | mwhahaha : seems ok here | 02:42 |
| mwhahaha | just came back | 02:42 |
| ianw | heh, i guess jhesketh's fix for http://cacti.openstack.org/ hasn't hit | 02:43 |
| jhesketh | ianw: yeah, you can still access it at http://cacti.openstack.org/cacti/graph_view.php though | 02:43 |
| mwhahaha | also i'm getting 500s when i review | 02:44 |
| jhesketh | but if people want to review https://review.openstack.org/#/c/321352/ , that'll solve it | 02:44 |
| jhesketh | mwhahaha: any particular reviews? | 02:44 |
| *** Madasi has quit IRC | 02:45 | |
| mwhahaha | i reviewed https://review.openstack.org/#/c/312280/ and https://review.openstack.org/#/c/321860/, when i hit submit both 500ed but it seemed to still work | 02:45 |
| jhesketh | hmm | 02:45 |
| *** antonym has joined #openstack-infra | 02:47 | |
| ianw | yeah, actually just reviewing that change i got a 500 | 02:48 |
| *** amitgandhinz has joined #openstack-infra | 02:49 | |
| ianw | load average, memory usuage look about right | 02:49 |
| jhesketh | ianw: which of the two did you review? | 02:50 |
| jhesketh | it looks like yours may not have been saved | 02:50 |
| *** yuanying has quit IRC | 02:50 | |
| *** Madasi has joined #openstack-infra | 02:50 | |
| ianw | https://review.openstack.org/#/c/321352/ | 02:50 |
| jhesketh | oh right | 02:50 |
| jhesketh | and that one was saved | 02:51 |
| jhesketh | yep, got it too | 02:51 |
| jhesketh | so it's possibly all reviews | 02:51 |
| mwhahaha | wondering if it was/is a network issue because it was down-down for me and i did one of those is it down or just me things and it was reporting down | 02:51 |
| fungi | java gc chewing up the system again? | 02:51 |
| jhesketh | fungi: what's the best way to tell? | 02:52 |
| fungi | javamelody graphs | 02:52 |
| jhesketh | yeah I was looking at those and they seem okay | 02:53 |
| *** amitgandhinz has quit IRC | 02:54 | |
| fungi | the garbage collection graph doesn't show it grinding continually? | 02:55 |
| mestery | mwhahaha: I'm also seeing issues doign "git review", getting "unpack failed: error Read-onlyu file system" | 02:55 |
| *** openstackgerrit has quit IRC | 02:56 | |
| mwhahaha | :o | 02:56 |
| *** mahito has joined #openstack-infra | 02:56 | |
| mestery | mwhahaha: http://paste.openstack.org/show/505797/ | 02:56 |
| *** rfolco has quit IRC | 02:56 | |
| *** antonym has quit IRC | 02:57 | |
| ianychoi | Neither do I. I cannot do code-review now. | 02:57 |
| mwhahaha | can't load paste.openstack.org now heh | 02:57 |
| fungi | [Fri May 27 02:37:49 2016] end_request: I/O error, dev xvdc, sector 63126424 | 02:57 |
| mestery | fungi: That doesn't look good ;) | 02:57 |
| ianw | there is this sort of odd square-wave for i/o on several mounted drives -> http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=4588&rra_id=all | 02:57 |
| fungi | we've got block storage errors from cinder | 02:57 |
| fungi | i'll offline i and fsck | 02:57 |
| ianychoi | draft comments are fine.. | 02:58 |
| *** Madasi has quit IRC | 02:58 | |
| jhesketh | fungi: ouch.. | 02:58 |
| jhesketh | should we send a status | 02:58 |
| fungi | #status alert Gerrit is going offline briefly to check possible filesystem corruption | 02:58 |
| openstackstatus | fungi: sending alert | 02:58 |
| *** woodster_ has quit IRC | 02:58 | |
| *** Sukhdev has joined #openstack-infra | 02:58 | |
| *** Daisy_ has joined #openstack-infra | 02:59 | |
| *** mahito has quit IRC | 02:59 | |
| fungi | probably network maintenance or an unplanned outage in rackspace dfw impacting connectivity between the nova host and cinder backend | 02:59 |
| fungi | or at least that's the usual cause | 03:00 |
| ianychoi | I see, thanks! Hope that gerrit will recover soon.. | 03:00 |
| -openstackstatus- NOTICE: Gerrit is going offline briefly to check possible filesystem corruption | 03:00 | |
| *** ChanServ changes topic to "Gerrit is going offline briefly to check possible filesystem corruption" | 03:00 | |
| *** jamielennox is now known as jamielennox|away | 03:01 | |
| fungi | okay, gerrit is on its way back up now | 03:01 |
| fungi | see if it's any better and then we can #status ok | 03:01 |
| *** Daisy has quit IRC | 03:02 | |
| fungi | mwhahaha: mestery: ianychoi: ianw: jhesketh: ^ | 03:03 |
| openstackstatus | fungi: finished sending alert | 03:03 |
| *** hockeynut has quit IRC | 03:04 | |
| *** thorst_ has joined #openstack-infra | 03:07 | |
| *** shashank_hegde has joined #openstack-infra | 03:07 | |
| *** erikmwilson has quit IRC | 03:07 | |
| mwhahaha | didn't get a 500 on one review, so that's an improvement :D | 03:07 |
| *** erikmwilson has joined #openstack-infra | 03:07 | |
| *** sdake has joined #openstack-infra | 03:07 | |
| fungi | mestery: hopefully your git review command works now if you retry? | 03:08 |
| *** anteaya has quit IRC | 03:08 | |
| *** hockeynut has joined #openstack-infra | 03:08 | |
| ianw | git.openstack.org[0: 104.130.246.128]: errno=Connection timed out | 03:09 |
| fungi | nice | 03:09 |
| ianw | might be unrelated, that's from a dib build i'm doing | 03:09 |
| fungi | "On 26 May 2016, at 21:26 CDT, engineers were alerted to a switching loop occurring in the DFW1 data center. Engineers are engaged and working to resolve the issue. During this time, Customers may be unable to access their Cloud instances hosted within the DFW1 data center." | 03:10 |
| fungi | https://status.rackspace.com/ | 03:10 |
| *** antonym has joined #openstack-infra | 03:10 | |
| *** ddieterly[away] has quit IRC | 03:10 | |
| *** Madasi has joined #openstack-infra | 03:11 | |
| fungi | so i'm going with "unplanned outage" as the cause here ;) | 03:11 |
| *** sdake_ has quit IRC | 03:11 | |
| * ianw takes an unplanned afternoon tea break | 03:13 | |
| fungi | there's also... | 03:13 |
| fungi | "The Rackspace Open Cloud system engineers will perform a priority maintenance to the control infrastructure of our Next Generation Cloud Servers regions during the following dates and times: [...] DFW Region - May 26th from 10:00 PM CDT - May 27th 5:00 AM CDT" | 03:13 |
| *** SumitNaiksatam has quit IRC | 03:13 | |
| fungi | though that one's only supposed to impact api endpoints | 03:14 |
| fungi | so this is more likely the bridge loop impact (somebody got drunk and turned off stp?) | 03:14 |
| fungi | (i used to do that all the time, just for laughs) | 03:15 |
| *** Daisy_ has quit IRC | 03:15 | |
| *** Daisy has joined #openstack-infra | 03:16 | |
| *** antonym has quit IRC | 03:16 | |
| fungi | anyway, no new errors reported for gerrit's cinder volume since the fsck and remount | 03:16 |
| fungi | i'm going to go with it should be all better now | 03:17 |
| *** Madasi has quit IRC | 03:17 | |
| fungi | #status ok after a quick check, gerrit and its filesystem have been brought back online and should be working again | 03:18 |
| openstackstatus | fungi: sending ok | 03:18 |
| *** antonym has joined #openstack-infra | 03:18 | |
| *** openstackgerrit has joined #openstack-infra | 03:18 | |
| fungi | mestery: thanks for saying "Read-only file system" since that helped me zero in on the problem instantly | 03:19 |
| *** amotoki has quit IRC | 03:20 | |
| *** ChanServ changes topic to "[sprint in progress on #openstack-sprint] Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 03:20 | |
| -openstackstatus- NOTICE: after a quick check, gerrit and its filesystem have been brought back online and should be working again | 03:20 | |
| mwhahaha | 500s again :( | 03:20 |
| fungi | no new filesystem errors, so that may just be network still broken in parts of rackspace | 03:21 |
| mwhahaha | k | 03:21 |
| fungi | are the 5xx errors intermittent? | 03:23 |
| openstackstatus | fungi: finished sending ok | 03:23 |
| *** Madasi has joined #openstack-infra | 03:23 | |
| *** Sam-I-Am has joined #openstack-infra | 03:25 | |
| fungi | oh, you know what? i bet it's also disrupting connectivity to gerrit's trove instance | 03:25 |
| fungi | some of the messages in gerrit's error log (ones that i don't recognize as the usual flood of benign noise in there) imply database socket timeouts | 03:26 |
| fungi | anyway, it's well past my bedtime | 03:27 |
| fungi | sorry jhesketh to leave you dealing with broken rackspace | 03:27 |
| jhesketh | no worries, it's not your fault | 03:27 |
| jhesketh | probably isn't much I can do in regards to the network | 03:27 |
| fungi | i don't think there's much we can do 'till this storm blows over | 03:27 |
| jhesketh | yeah | 03:28 |
| jhesketh | thanks for fixing the filesystem though | 03:28 |
| jhesketh | fungi: get some sleep :-) | 03:28 |
| fungi | thanks. hopefully they'll have figured out how to use spanning tree by the time i wake up ;) | 03:28 |
| fungi | night all! | 03:29 |
| ianychoi | :) Thanks! | 03:31 |
| jhesketh | o. | 03:32 |
| jhesketh | *o/ | 03:32 |
| *** yamahata has joined #openstack-infra | 03:34 | |
| mestery | fungi: Thanks for the help! :) | 03:36 |
| *** thorst_ has quit IRC | 03:38 | |
| *** thorst_ has joined #openstack-infra | 03:39 | |
| *** Douhet has quit IRC | 03:40 | |
| *** bpokorny has joined #openstack-infra | 03:41 | |
| *** Douhet has joined #openstack-infra | 03:43 | |
| *** Daisy has quit IRC | 03:43 | |
| *** Daisy has joined #openstack-infra | 03:44 | |
| *** baoli has quit IRC | 03:44 | |
| *** phschwartz has joined #openstack-infra | 03:45 | |
| *** baoli has joined #openstack-infra | 03:45 | |
| *** Sukhdev has quit IRC | 03:45 | |
| *** links has joined #openstack-infra | 03:46 | |
| *** thorst_ has quit IRC | 03:48 | |
| *** gomarivera has joined #openstack-infra | 03:48 | |
| *** Daisy has quit IRC | 03:48 | |
| *** yuanying has joined #openstack-infra | 03:48 | |
| *** baoli has quit IRC | 03:50 | |
| *** amitgandhinz has joined #openstack-infra | 03:50 | |
| *** baoli has joined #openstack-infra | 03:51 | |
| *** amitgandhinz has quit IRC | 03:55 | |
| *** fawadkhaliq has joined #openstack-infra | 04:04 | |
| *** nadya has joined #openstack-infra | 04:05 | |
| *** sree has joined #openstack-infra | 04:06 | |
| *** fawadkhaliq has quit IRC | 04:07 | |
| *** Daisy has joined #openstack-infra | 04:08 | |
| *** zhurong has quit IRC | 04:09 | |
| *** nadya has quit IRC | 04:09 | |
| *** cody-somerville has quit IRC | 04:10 | |
| *** claudiub|2 has quit IRC | 04:10 | |
| *** Sam-I-Am has quit IRC | 04:12 | |
| *** banix has quit IRC | 04:12 | |
| *** yamamoto has quit IRC | 04:13 | |
| *** zhurong has joined #openstack-infra | 04:14 | |
| *** amotoki has joined #openstack-infra | 04:14 | |
| *** Daisy has quit IRC | 04:17 | |
| *** Sam-I-Am has joined #openstack-infra | 04:18 | |
| *** Daisy has joined #openstack-infra | 04:18 | |
| *** cody-somerville_ has joined #openstack-infra | 04:18 | |
| Qiming | jhesketh, still there? | 04:19 |
| *** yamahata has quit IRC | 04:20 | |
| Qiming | thanks for w+1 this patch: https://review.openstack.org/#/c/318453/ | 04:20 |
| jhesketh | Qiming: yep, I'm around | 04:20 |
| Qiming | but I'm afraid the gate job was not in queue due to the gerrit reboot? could you please help re-approve it? ... not sure if it is necessary | 04:21 |
| Qiming | thanks! | 04:21 |
| *** armax has quit IRC | 04:21 | |
| jhesketh | Qiming: I've left a recheck which should get it picked up again | 04:21 |
| Qiming | okay, great! thank you. | 04:22 |
| jhesketh | because of network issues at rackspace though it's likely the system will be under a little bit of turbulence so it may take a bit still | 04:22 |
| Qiming | jhesketh, will keep an eye on the progress | 04:23 |
| *** kdas__ has joined #openstack-infra | 04:23 | |
| *** psachin has joined #openstack-infra | 04:24 | |
| *** sdake_ has joined #openstack-infra | 04:25 | |
| *** nwkarsten has joined #openstack-infra | 04:27 | |
| *** sdake has quit IRC | 04:27 | |
| *** sree_ has joined #openstack-infra | 04:28 | |
| *** sree_ is now known as Guest39411 | 04:28 | |
| *** amotoki has quit IRC | 04:28 | |
| *** sree has quit IRC | 04:29 | |
| *** kdas__ is now known as kushal | 04:29 | |
| *** yamahata has joined #openstack-infra | 04:29 | |
| *** kushal has quit IRC | 04:29 | |
| *** kushal has joined #openstack-infra | 04:29 | |
| *** sdake has joined #openstack-infra | 04:31 | |
| *** markvoelker has joined #openstack-infra | 04:32 | |
| *** Sukhdev has joined #openstack-infra | 04:33 | |
| *** sdake_ has quit IRC | 04:33 | |
| *** Daisy_ has joined #openstack-infra | 04:34 | |
| *** yfried has quit IRC | 04:34 | |
| *** Daisy has quit IRC | 04:35 | |
| *** markvoelker has quit IRC | 04:37 | |
| *** Douhet has quit IRC | 04:38 | |
| *** maishsk has quit IRC | 04:41 | |
| *** nwkarsten has quit IRC | 04:42 | |
| *** gomarivera has quit IRC | 04:42 | |
| *** bpokorny has quit IRC | 04:44 | |
| *** nwkarsten has joined #openstack-infra | 04:44 | |
| *** jamesmcarthur has joined #openstack-infra | 04:45 | |
| *** thorst_ has joined #openstack-infra | 04:45 | |
| *** roxanaghe has joined #openstack-infra | 04:48 | |
| *** jamesmcarthur has quit IRC | 04:49 | |
| *** jaosorior has joined #openstack-infra | 04:50 | |
| *** amitgandhinz has joined #openstack-infra | 04:51 | |
| *** thorst_ has quit IRC | 04:52 | |
| *** amitgandhinz has quit IRC | 04:56 | |
| *** flwang1 has quit IRC | 04:56 | |
| *** yamamot__ has joined #openstack-infra | 04:57 | |
| *** Guest39411 has quit IRC | 04:58 | |
| *** dimtruck is now known as zz_dimtruck | 04:58 | |
| *** gomarivera has joined #openstack-infra | 04:58 | |
| *** hparekh has joined #openstack-infra | 05:00 | |
| *** gomarivera has quit IRC | 05:03 | |
| *** nadya has joined #openstack-infra | 05:03 | |
| *** nwkarsten has quit IRC | 05:03 | |
| *** nwkarsten has joined #openstack-infra | 05:06 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-bandersnatch: Fix acceptance tests https://review.openstack.org/320068 | 05:08 |
| *** maishsk has joined #openstack-infra | 05:08 | |
| *** amotoki has joined #openstack-infra | 05:11 | |
| *** maishsk has quit IRC | 05:12 | |
| *** ilyashakhat has joined #openstack-infra | 05:14 | |
| *** roxanaghe has quit IRC | 05:14 | |
| *** maishsk has joined #openstack-infra | 05:15 | |
| *** zhurong has quit IRC | 05:17 | |
| *** zhurong has joined #openstack-infra | 05:20 | |
| *** armax has joined #openstack-infra | 05:23 | |
| *** armax has quit IRC | 05:23 | |
| *** gildub has joined #openstack-infra | 05:28 | |
| *** baoli has quit IRC | 05:29 | |
| *** sarob has joined #openstack-infra | 05:31 | |
| *** salv-orlando has joined #openstack-infra | 05:33 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/nodepool: Python 3 Fixes: use bytes instead of str https://review.openstack.org/321957 | 05:34 |
| *** sarob has quit IRC | 05:35 | |
| *** nwkarsten has quit IRC | 05:35 | |
| *** xwizard has quit IRC | 05:35 | |
| *** xwizard has joined #openstack-infra | 05:36 | |
| rakhmerov | hi, seems like new jobs don't start | 05:37 |
| rakhmerov | is it being taken care of? | 05:37 |
| rakhmerov | fungi: ^ | 05:38 |
| openstackgerrit | zhurong proposed openstack-infra/project-config: Add check-requirements for solum https://review.openstack.org/321958 | 05:38 |
| *** ramishra has quit IRC | 05:38 | |
| *** nwkarsten has joined #openstack-infra | 05:38 | |
| *** ramishra has joined #openstack-infra | 05:40 | |
| zaro | madhuvishy: you might want to ping hashar for a review | 05:44 |
| *** mixos has quit IRC | 05:45 | |
| zaro | rakhmerov: which ones? have you tried 'recheck'? | 05:45 |
| rakhmerov | zaro: I sent https://review.openstack.org/#/c/317879/ ~ 20 mins ago and still see 15 check jobs in Zuul | 05:46 |
| rakhmerov | mine didn't start yet | 05:46 |
| *** bhavik has joined #openstack-infra | 05:47 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Cleanup source-repositories output https://review.openstack.org/321961 | 05:49 |
| *** binbincong has quit IRC | 05:49 | |
| *** thorst_ has joined #openstack-infra | 05:50 | |
| *** nwkarsten has quit IRC | 05:50 | |
| *** nwkarsten has joined #openstack-infra | 05:51 | |
| *** amitgandhinz has joined #openstack-infra | 05:51 | |
| *** sdake_ has joined #openstack-infra | 05:53 | |
| *** sdake_ has quit IRC | 05:53 | |
| *** sdake_ has joined #openstack-infra | 05:53 | |
| *** nwkarsten has quit IRC | 05:55 | |
| *** nadya has quit IRC | 05:55 | |
| *** amitgandhinz has quit IRC | 05:56 | |
| *** sdake has quit IRC | 05:56 | |
| *** thorst_ has quit IRC | 05:57 | |
| rakhmerov | zaro: do I need to let someone else know about it? | 05:58 |
| rakhmerov | don't know exactly who to ping | 05:58 |
| *** sdake_ has quit IRC | 05:59 | |
| *** ilyashakhat has quit IRC | 06:04 | |
| *** rcernin has joined #openstack-infra | 06:05 | |
| rakhmerov | SergeyLukjanov: hi Sergey, do you happen to know about what I wrote above? | 06:06 |
| *** ilyashakhat has joined #openstack-infra | 06:06 | |
| *** binbincong has joined #openstack-infra | 06:10 | |
| *** aeng has quit IRC | 06:10 | |
| jaosorior | yep, seems that jobs are gettings stuck | 06:10 |
| jaosorior | rechecks won't help | 06:10 |
| *** YorikSar has quit IRC | 06:13 | |
| *** YorikSar has joined #openstack-infra | 06:15 | |
| *** rcernin has quit IRC | 06:15 | |
| *** ffrank has joined #openstack-infra | 06:17 | |
| *** yfried has joined #openstack-infra | 06:17 | |
| *** binbincong has quit IRC | 06:18 | |
| *** rcernin has joined #openstack-infra | 06:20 | |
| *** salv-orlando has quit IRC | 06:24 | |
| *** ilyashakhat has quit IRC | 06:24 | |
| jhesketh | Rackspace has had some networking trouble so I suspect zuul is stuck in a bad state | 06:25 |
| jhesketh | I'll take a look adn see if I can get it moving along | 06:25 |
| *** Sukhdev has quit IRC | 06:27 | |
| *** javeriak has joined #openstack-infra | 06:27 | |
| rakhmerov | jhesketh: yes, thanks | 06:27 |
| *** nadya has joined #openstack-infra | 06:30 | |
| *** binbincong has joined #openstack-infra | 06:31 | |
| *** markvoelker has joined #openstack-infra | 06:33 | |
| jhesketh | zuul isn't processing it's queues, but I'm not sure why... it likely got stuck talking to gerrit when we had to restart it | 06:34 |
| *** megm has quit IRC | 06:34 | |
| jhesketh | I think it'll require a restart to fix but that'll lose 4000+ events... | 06:35 |
| jhesketh | yolanda: ping | 06:35 |
| *** mikelk has joined #openstack-infra | 06:35 | |
| *** megm has joined #openstack-infra | 06:36 | |
| *** markvoelker has quit IRC | 06:38 | |
| jhesketh | I'm going to shut down zuul and hope it writes out its events.. otherwise people will need to recheck their patches | 06:39 |
| rakhmerov | ok | 06:40 |
| *** daemontool has joined #openstack-infra | 06:44 | |
| *** flepied has joined #openstack-infra | 06:46 | |
| *** kushal has quit IRC | 06:47 | |
| *** Daisy has joined #openstack-infra | 06:47 | |
| jhesketh | the queue wasn't able to be reloaded... it's back up and running though so I'm going to watch some results before sending a notice for people to recheck missing jobs | 06:48 |
| jhesketh | hmm the multinode jobs aren't registered... | 06:48 |
| *** kushal has joined #openstack-infra | 06:48 | |
| *** ffrank has quit IRC | 06:49 | |
| openstackgerrit | Martin André proposed openstack-dev/cookiecutter: Add missing license info to requirements.txt https://review.openstack.org/321975 | 06:50 |
| *** Daisy_ has quit IRC | 06:51 | |
| *** amitgandhinz has joined #openstack-infra | 06:52 | |
| openstackgerrit | Martin André proposed openstack-dev/cookiecutter: Add missing license info to requirements.txt https://review.openstack.org/321975 | 06:55 |
| *** thorst_ has joined #openstack-infra | 06:55 | |
| *** amitgandhinz has quit IRC | 06:57 | |
| openstackgerrit | Merged openstack-infra/project-config: Add Senlin support to rally-gate https://review.openstack.org/318453 | 07:02 |
| *** thorst_ has quit IRC | 07:02 | |
| *** maishsk has quit IRC | 07:04 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: Allow to pass OS_TEST_TIMEOUT for grenade job https://review.openstack.org/316662 | 07:05 |
| *** maishsk has joined #openstack-infra | 07:06 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW https://review.openstack.org/315499 | 07:06 |
| *** tdasilva has quit IRC | 07:07 | |
| *** ilyashakhat has joined #openstack-infra | 07:08 | |
| *** Daisy has quit IRC | 07:09 | |
| *** Daisy has joined #openstack-infra | 07:09 | |
| *** frickler has quit IRC | 07:10 | |
| *** vincentll has joined #openstack-infra | 07:10 | |
| *** Mmike has quit IRC | 07:10 | |
| *** Daisy has quit IRC | 07:11 | |
| *** Daisy has joined #openstack-infra | 07:11 | |
| *** Daisy has quit IRC | 07:11 | |
| *** Daisy has joined #openstack-infra | 07:12 | |
| jhesketh | #status notice zuul required a restart due to network outages. If your change is not listed on http://status.openstack.org/zuul/ and is missing results, please issue a 'recheck'. | 07:12 |
| openstackstatus | jhesketh: sending notice | 07:12 |
| -openstackstatus- NOTICE: zuul required a restart due to network outages. If your change is not listed on http://status.openstack.org/zuul/ and is missing results, please issue a 'recheck'. | 07:13 | |
| *** ccamacho has quit IRC | 07:14 | |
| openstackstatus | jhesketh: finished sending notice | 07:15 |
| *** ifarkas has joined #openstack-infra | 07:15 | |
| *** ccamacho has joined #openstack-infra | 07:16 | |
| *** Mmike has joined #openstack-infra | 07:16 | |
| openstackgerrit | Martin André proposed openstack-dev/cookiecutter: Add missing license info to requirements.txt https://review.openstack.org/321975 | 07:16 |
| *** ilyashakhat has quit IRC | 07:17 | |
| *** Daisy has quit IRC | 07:17 | |
| *** camunoz has quit IRC | 07:19 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add MysqlInternal endpoint to enable-tls https://review.openstack.org/321363 | 07:21 |
| *** schang has quit IRC | 07:24 | |
| *** kushal has quit IRC | 07:24 | |
| *** bhavik has quit IRC | 07:25 | |
| *** tdasilva has joined #openstack-infra | 07:27 | |
| *** flepied has quit IRC | 07:27 | |
| *** frickler has joined #openstack-infra | 07:31 | |
| *** schang has joined #openstack-infra | 07:31 | |
| *** daemontool has quit IRC | 07:32 | |
| *** oanson has joined #openstack-infra | 07:33 | |
| *** strigazi has quit IRC | 07:33 | |
| *** bhavik has joined #openstack-infra | 07:35 | |
| *** bauzas is now known as bauwser | 07:35 | |
| *** tesseract has joined #openstack-infra | 07:36 | |
| *** claudiub|2 has joined #openstack-infra | 07:40 | |
| *** amotoki_ has joined #openstack-infra | 07:41 | |
| *** hashar has joined #openstack-infra | 07:43 | |
| *** amotoki has quit IRC | 07:43 | |
| *** amoralej|off is now known as amoralej | 07:44 | |
| *** yamahata has quit IRC | 07:46 | |
| *** arxcruz has joined #openstack-infra | 07:49 | |
| *** salv-orlando has joined #openstack-infra | 07:50 | |
| *** ilyashakhat has joined #openstack-infra | 07:51 | |
| *** amitgandhinz has joined #openstack-infra | 07:53 | |
| *** ilyashakhat has quit IRC | 07:55 | |
| *** gildub has quit IRC | 07:56 | |
| *** ifarkas_ has joined #openstack-infra | 07:56 | |
| *** ifarkas has quit IRC | 07:56 | |
| *** sarob has joined #openstack-infra | 07:58 | |
| *** amitgandhinz has quit IRC | 07:58 | |
| *** sree has joined #openstack-infra | 07:59 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** thorst_ has joined #openstack-infra | 08:00 | |
| *** sarob has quit IRC | 08:02 | |
| *** afazekas|sick is now known as afazekas | 08:03 | |
| *** pilgrimstack has joined #openstack-infra | 08:03 | |
| *** shashank_hegde has quit IRC | 08:05 | |
| *** thorst_ has quit IRC | 08:07 | |
| *** flepied has joined #openstack-infra | 08:08 | |
| *** bhavik has quit IRC | 08:09 | |
| *** jordanP has joined #openstack-infra | 08:13 | |
| *** sree has quit IRC | 08:13 | |
| *** sree has joined #openstack-infra | 08:19 | |
| *** hichihar_ has joined #openstack-infra | 08:19 | |
| *** claudiub|2 has quit IRC | 08:19 | |
| *** pahuang has quit IRC | 08:20 | |
| *** slaweq has quit IRC | 08:21 | |
| *** hichihara has quit IRC | 08:22 | |
| *** sree has quit IRC | 08:24 | |
| *** asettle has joined #openstack-infra | 08:25 | |
| *** slaweq has joined #openstack-infra | 08:26 | |
| *** asettle has quit IRC | 08:26 | |
| *** tosky has joined #openstack-infra | 08:30 | |
| *** tosky has left #openstack-infra | 08:31 | |
| *** tosky has joined #openstack-infra | 08:31 | |
| *** markusry has joined #openstack-infra | 08:31 | |
| *** vincentll has quit IRC | 08:34 | |
| *** YorikSar has quit IRC | 08:35 | |
| *** derekh has joined #openstack-infra | 08:35 | |
| *** YorikSar has joined #openstack-infra | 08:36 | |
| *** zeih has joined #openstack-infra | 08:42 | |
| *** e0ne has joined #openstack-infra | 08:43 | |
| *** arxcruz has quit IRC | 08:43 | |
| *** dmk0202 has joined #openstack-infra | 08:46 | |
| *** strigazi has joined #openstack-infra | 08:47 | |
| *** amrith is now known as _amrith_ | 08:47 | |
| *** yuanying has quit IRC | 08:47 | |
| *** pbourke_ has quit IRC | 08:49 | |
| *** pbourke_ has joined #openstack-infra | 08:49 | |
| *** zhurong has quit IRC | 08:52 | |
| *** zhurong has joined #openstack-infra | 08:53 | |
| wznoinsk | hi all | 08:53 |
| *** yuanying has joined #openstack-infra | 08:53 | |
| *** yanyanhu has quit IRC | 08:54 | |
| *** amitgandhinz has joined #openstack-infra | 08:54 | |
| *** yanyanhu has joined #openstack-infra | 08:54 | |
| *** yuanying has quit IRC | 08:54 | |
| *** yanyanhu has quit IRC | 08:55 | |
| wznoinsk | I'm looking for a best way to 'pause' my CI, in case of a site-wide issue where all/most of the jobs are impacted I'd like to stop running any real jobs and do a testing to find a workaround, and when solution is found unpause the zuul and jobs... I'm wondering what's the best way to do it? | 08:55 |
| *** javeriak has quit IRC | 08:56 | |
| *** Qiming has quit IRC | 08:57 | |
| *** amitgandhinz has quit IRC | 08:59 | |
| *** jaosorior is now known as jaosorior_lunch | 08:59 | |
| *** dizquierdo has joined #openstack-infra | 08:59 | |
| rcarrillocruz | wznoinsk: http://git.openstack.org/cgit/openstack-infra/system-config/tree/doc/source/zuul.rst#n111 | 09:00 |
| *** flwang1 has joined #openstack-infra | 09:01 | |
| *** YorikSar has quit IRC | 09:01 | |
| *** eezhova has joined #openstack-infra | 09:01 | |
| *** YorikSar has joined #openstack-infra | 09:03 | |
| *** HeOS has joined #openstack-infra | 09:04 | |
| *** thorst_ has joined #openstack-infra | 09:05 | |
| openstackgerrit | Markus Zoeller (markus_z) proposed openstack-infra/release-tools: update README for the script to expire old bug reports https://review.openstack.org/322019 | 09:06 |
| *** amotoki_ has quit IRC | 09:10 | |
| *** yuanying has joined #openstack-infra | 09:10 | |
| *** pbourke_ has quit IRC | 09:10 | |
| *** esikachev has joined #openstack-infra | 09:10 | |
| *** nadya has quit IRC | 09:12 | |
| *** pbourke_ has joined #openstack-infra | 09:12 | |
| *** thorst_ has quit IRC | 09:12 | |
| *** yuanying has quit IRC | 09:15 | |
| wznoinsk | rcarrillocruz that's not exactly what I was looking for... I know how to restart zuul, I'm more interested in how to keep it reading events while not executing the jobs till I give it a 'green' light that some site-wide issue is now resolved | 09:15 |
| *** vincentll has joined #openstack-infra | 09:15 | |
| rcarrillocruz | well, that's not just restart zuul, but saving the queues state and reloading them after a zuul restart. If you are looking for a way for zuul to start reading the events stream , I don't think there's a way to do that | 09:16 |
| rcarrillocruz | s/start/stop | 09:16 |
| wznoinsk | for the moment I've put Jenkins into shutdown mode hence it's not executing anything but the jobs are still registered with gearman server so does what I wanted to do but I would like to be able to still kick off some test jobs from jenkins while resovling the issue manually | 09:17 |
| *** bhavik has joined #openstack-infra | 09:17 | |
| strigazi | hi all, I'd like some feedback on https://review.openstack.org/#/c/321026/ | 09:18 |
| *** daemontool has joined #openstack-infra | 09:18 | |
| *** zzzeek has quit IRC | 09:20 | |
| *** amotoki has joined #openstack-infra | 09:20 | |
| rcarrillocruz | wznoinsk: zuul is now under some major change, switching from zuul-gearman-jenkins to zuul launching jobs via ansible | 09:22 |
| rcarrillocruz | check it out with jeblair to discuss that use case | 09:23 |
| rcarrillocruz | jhesketh too | 09:23 |
| *** ilyashakhat has joined #openstack-infra | 09:24 | |
| wznoinsk | thanks, will do | 09:25 |
| *** mhickey has joined #openstack-infra | 09:26 | |
| *** amotoki has quit IRC | 09:27 | |
| yolanda | good morning | 09:27 |
| *** electrofelix has joined #openstack-infra | 09:27 | |
| strigazi | Hi yolanda, I'm Spyros from magnum team. I want to add a non-voting job at the rally gate to test our benchmark scenarios. I think this change needs more work. Can you have a look: https://review.openstack.org/#/c/321026/ | 09:31 |
| *** arxcruz has joined #openstack-infra | 09:35 | |
| yolanda | strigazi, sure | 09:36 |
| *** jlanoux has joined #openstack-infra | 09:36 | |
| strigazi | yolanda: thanks | 09:36 |
| *** salv-orl_ has joined #openstack-infra | 09:38 | |
| *** Guest98278 has quit IRC | 09:38 | |
| openstackgerrit | Merged openstack-infra/project-config: Add Non-voting job for nodepool py34 https://review.openstack.org/321885 | 09:38 |
| *** gomarivera has joined #openstack-infra | 09:39 | |
| *** Hal has joined #openstack-infra | 09:40 | |
| *** salv-orlando has quit IRC | 09:40 | |
| *** Hal is now known as Guest53789 | 09:40 | |
| *** jyuso1 has quit IRC | 09:41 | |
| yolanda | strigazi, initially looks good. I see you miss the rally-plot publisher, you don't need it? | 09:41 |
| *** nadya has joined #openstack-infra | 09:41 | |
| strigazi | yolanda: we do, thanks | 09:42 |
| strigazi | yolanda: new patch comming | 09:43 |
| *** mpaolino has joined #openstack-infra | 09:43 | |
| *** gomarivera has quit IRC | 09:44 | |
| strigazi | yolanda: Isn't included in line 934 https://review.openstack.org/#/c/321026/4/jenkins/jobs/rally.yaml | 09:45 |
| yolanda | oh, last line was cut on my screen!!! | 09:46 |
| * yolanda switched from computers yesterday | 09:46 | |
| *** Qiming has joined #openstack-infra | 09:46 | |
| strigazi | :) | 09:47 |
| yolanda | the change looks good, my screen doesn't :) | 09:47 |
| strigazi | yolanda: thanks | 09:50 |
| jhesketh | wznoinsk: so it'd be a little hacky, but I think you should be able to do what you want with some reloads... You could configure a job that will never run with every project as its only job (it'll need to be registered with gearman, but you can do that via telnet or a simple gear client). Then play with your jenkins configuration/jobs however you want, and when you're ready configure the layout.yaml to have all the jobs | 09:50 |
| jhesketh | again and reload | 09:50 |
| jhesketh | wznoinsk: zuul will correct the jobs that should be ran for a change that is still in the pipeline. So if you've added or taken jobs while it is there it will figure out what to do | 09:51 |
| jhesketh | if that makes sense | 09:51 |
| odyssey4me | yolanda is there anyone available to add another review to yours for my stream of patches? https://review.openstack.org/#/q/owner:jesse-pretorius+status:open+project:openstack-infra/project-config | 09:51 |
| yolanda | odyssey4me, any infra core or projec-config core could help with that | 09:52 |
| odyssey4me | yolanda unfortunately it seems that everyone's been busy with sprints, so even though I've been asking no-one's managed to get to them | 09:53 |
| yolanda | jhesketh seem to be around... or ping pabelanger on few hours | 09:53 |
| *** permalac has joined #openstack-infra | 09:54 | |
| *** permalac has quit IRC | 09:54 | |
| jhesketh | odyssey4me: I can take a look | 09:54 |
| odyssey4me | thanks jhesketh | 09:54 |
| *** permalac has joined #openstack-infra | 09:54 | |
| *** amitgandhinz has joined #openstack-infra | 09:54 | |
| *** jianghuaw has joined #openstack-infra | 09:55 | |
| *** permalac has quit IRC | 09:55 | |
| *** permalac has joined #openstack-infra | 09:55 | |
| jianghuaw | Hi, anyone met this failure which nodepool image-update: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'logstash.openstack.org' ([Errno 101] Network is unreachable)") | 09:56 |
| wznoinsk | jhasketh I had the same idea, hence pointed my joubs to ubuntu-trusty-dummy instead of ubuntu-trusty (in jbb projects.yaml) only then to learn the jobs don't register when they don't have slave available during registration attempt (and I did restart zuul in the meantime)... if I would not restart zuul and registered jobs at gearman server would be still the old names jobA:ubuntu-trusty I guess I should be fine? (jobs will not run | 09:56 |
| wznoinsk | - not slave available but it's fixable by updating jbb projects.yaml -> jenkins slave info) ? | 09:56 |
| jianghuaw | my nodepool ran well until sometime back in today. it always failed with this error. | 09:57 |
| *** javeriak has joined #openstack-infra | 09:58 | |
| wznoinsk | jianghuaw check 'ip -r' for default gateway | 09:58 |
| wznoinsk | 'ip r' even | 09:59 |
| *** ociuhandu has quit IRC | 09:59 | |
| *** amitgandhinz has quit IRC | 09:59 | |
| jhesketh | jianghuaw: logstash may have changed ip's.. | 09:59 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Rename openstack-ansible-ironic to openstack-ansible-os_ironic https://review.openstack.org/299192 | 09:59 |
| jhesketh | wznoinsk: I didn't fully follow sorry... It depends what is in your layout... so long as the jobs in layout.yaml are registered with the gearman server at some point then you should be fine | 10:00 |
| jianghuaw | oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'logstash.openstack.org' ([Errno 101] Network is unreachable)") | 10:01 |
| jianghuaw | oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'logstash.openstack.org' ([Errno 101] Network is unreachable)") | 10:01 |
| jianghuaw | wznoinsk | 10:01 |
| jianghuaw | @wznoinsk: gateway has no change and it's ok to reach other internal address. | 10:01 |
| *** sdague has joined #openstack-infra | 10:01 | |
| jianghuaw | wznoinsk: there is no change with the route and it can reach other internal address. | 10:02 |
| openstackgerrit | Merged openstack-infra/project-config: Add api-ref job for Zaqar https://review.openstack.org/321324 | 10:02 |
| *** redixin has joined #openstack-infra | 10:03 | |
| redixin | hiyo. Tell me please where I can find some info about openstack proposal bot? fungi jeblair SergeyLukjanov | 10:04 |
| wznoinsk | jhesketh: yeah I think it see the full picture, I don't want to change layout unless I have to (I want to keep my current production layout) so I'll test only pointing jobs to a nonexistent node in jbb/jenkins hence jobs will be waiting in zuul for jenkins with the proper slave for the job allowing me to troubleshoot, i'll change project.yaml and node used for my jobs to the real slave label and zuul should kick off again, is my t | 10:04 |
| wznoinsk | hinking correct? | 10:04 |
| openstackgerrit | Merged openstack-infra/project-config: Add CloudKitty role to OpenStack-Ansible https://review.openstack.org/318836 | 10:07 |
| *** zhurong has quit IRC | 10:07 | |
| wznoinsk | jianghuaw: to check jhesketh suggestion go to https://toolbox.googleapps.com/apps/dig/#A/logstash.openstack.org and compare it with out of 'host logstash.openstack.org' on the machine with the problem | 10:07 |
| jianghuaw | wznoinsk: thanks. I will try. | 10:08 |
| wznoinsk | you may have the domain resolving to a diff/old ip | 10:08 |
| *** thorst_ has joined #openstack-infra | 10:09 | |
| jianghuaw | on the node, I can successfully ping to logstash.openstack.org. | 10:09 |
| *** javeriak has quit IRC | 10:10 | |
| wznoinsk | can you connect to the mysql on it ? | 10:10 |
| *** ilyashakhat has quit IRC | 10:10 | |
| *** javeriak has joined #openstack-infra | 10:10 | |
| *** _degorenko|afk is now known as degorenko | 10:11 | |
| *** yuanying has joined #openstack-infra | 10:11 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Added new repo for fuel-plugin-datera-cinder https://review.openstack.org/315651 | 10:11 |
| jhesketh | wznoinsk: that's an interesting question... so I think zuul will only request the job, not a specific node (unless you name the job's in zuul with a :node-type suffix). So it's the specific node that is registering with gearman as able to do the job. | 10:11 |
| jianghuaw | I've restart the image-building, I will check it after the VM's up. | 10:11 |
| jhesketh | wznoinsk: so as long as another node previously registered with gearman, I think it'd be okay... but I'm not sure | 10:11 |
| *** oanson has quit IRC | 10:12 | |
| wznoinsk | jhesketh: I haven't checked the code but I think zuul will only send to a specific jenkins only if it sees nodepool providing a properly labeled slave to that jenkins | 10:13 |
| openstackgerrit | Merged openstack-infra/project-config: Retire openstack-ansible-py_from_git repository https://review.openstack.org/319322 | 10:14 |
| *** lezbar has quit IRC | 10:16 | |
| jhesketh | wznoinsk: I don't think that's the case, but I may be wrong... | 10:17 |
| *** ociuhandu has joined #openstack-infra | 10:17 | |
| jhesketh | odyssey4me: I've reviewed your patches and there are a couple requiring feedback | 10:17 |
| *** Na3iL has joined #openstack-infra | 10:17 | |
| odyssey4me | thanks jhesketh - resolving the resulting merge conflicts and updating the patches | 10:17 |
| *** thorst_ has quit IRC | 10:18 | |
| openstackgerrit | Kirill Bespalov proposed openstack-infra/project-config: add reno jobs for oslo projects https://review.openstack.org/320904 | 10:18 |
| openstackgerrit | Kirill Bespalov proposed openstack-infra/project-config: add reno jobs for oslo projects https://review.openstack.org/320904 | 10:19 |
| openstackgerrit | Kirill Bespalov proposed openstack-infra/project-config: add reno jobs for oslo projects https://review.openstack.org/320904 | 10:19 |
| wznoinsk | jhesketh: that was my understanding of gearman making use of multiple jenkins masters avaiable, it sends the job to the jenkins that has everything needed to run the job | 10:20 |
| *** ilyashakhat has joined #openstack-infra | 10:20 | |
| jhesketh | I'd have to look at the code sorry | 10:21 |
| jhesketh | and haven't got time right now :-( | 10:21 |
| *** jaosorior_lunch is now known as jaosorior | 10:22 | |
| wznoinsk | jhesketh what are the time-wise plans regarding the zuul ansible? | 10:23 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add release announcement jobs for major OSA repos https://review.openstack.org/319234 | 10:23 |
| jhesketh | wznoinsk: as soon as it's ready most likely | 10:23 |
| odyssey4me | jhesketh are my changes to https://review.openstack.org/319234 what you meant? | 10:23 |
| jhesketh | jeblair has been working really hard on it and made great progres... I'd say it's very close.. hopefully next week, but no idea :-) | 10:24 |
| wznoinsk | jhesketh ok, I'll touch base with him later then, thanks | 10:24 |
| *** sarob has joined #openstack-infra | 10:25 | |
| jhesketh | odyssey4me: yep, thanks | 10:26 |
| wznoinsk | jhesketh btw. I think stopping zuul-merger would have a similar effect to the solution we've discussed above, zuul will wait for merge to complete before it sends the job to jenkins... | 10:26 |
| jianghuaw | wznoinsk: yes. connection to mysql on logstash failed: ERROR 2003 (HY000): Can't connect to MySQL server on 'logstash.openstack.org' (101) | 10:27 |
| jianghuaw | maybe the mysql service failed on it? | 10:27 |
| jhesketh | wznoinsk: oh yeah, that's probably a much easier solution | 10:27 |
| wznoinsk | jianghuaw it looks like it says network unreachable again, if the mysql service would be down you'd get connection refused most likely | 10:29 |
| *** javeriak has quit IRC | 10:29 | |
| *** yamamot__ has quit IRC | 10:29 | |
| jianghuaw | wznoinsk: but same error code: 101. | 10:30 |
| *** sarob has quit IRC | 10:30 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add Sahara role to OpenStack-Ansible https://review.openstack.org/317931 | 10:30 |
| odyssey4me | thanks for picking up on that error jhesketh ^ updated | 10:30 |
| wznoinsk | jianghuaw; can you telnet logstash.openstack.org 3306 ? | 10:30 |
| jhesketh | no worries | 10:31 |
| wznoinsk | btw. are you running the nodepool in some sort of networking isolation? namespace/(docker) container etc? | 10:32 |
| jianghuaw | telnet logstash.openstack.org 3306 | 10:32 |
| jianghuaw | Trying 23.253.230.235... | 10:32 |
| jianghuaw | Trying 2001:4800:7817:103:be76:4eff:fe05:f1cc... | 10:32 |
| jianghuaw | telnet: Unable to connect to remote host: Network is unreachable | 10:32 |
| rcarrillocruz | yolanda: mind doing a quick review for https://review.openstack.org/#/c/322053/2/tasks/create_clouds_resources.yml ? | 10:33 |
| wznoinsk | there you go, solve this problem and you're good then ;-) try traceroute maybe http://www.howtogeek.com/134132/how-to-use-traceroute-to-identify-network-problems/ | 10:33 |
| rcarrillocruz | trivial, but as it's 'largish' i rather get another core +2 | 10:33 |
| yolanda | sure | 10:33 |
| rcarrillocruz | thx | 10:34 |
| *** markvoelker has joined #openstack-infra | 10:34 | |
| jianghuaw | wznoinsk: No, nodepool run in a VM from the RAX cloud. | 10:34 |
| *** lezbar has joined #openstack-infra | 10:34 | |
| wznoinsk | I'm surprised you were a ble to ping that ip tho | 10:35 |
| jianghuaw | wznoinsk: but it does work. | 10:37 |
| jianghuaw | ping logstash.openstack.org | 10:37 |
| jianghuaw | PING logstash.openstack.org (23.253.230.235) 56(84) bytes of data. | 10:37 |
| jianghuaw | 64 bytes from logstash.openstack.org (23.253.230.235): icmp_seq=1 ttl=48 time=172 ms | 10:37 |
| wznoinsk | ok, maybe try disabling ipv6 if you don't use it i.e.: 'sysctl -w net.ipv6.conf.all.disable_ipv6=1' | 10:38 |
| *** markvoelker has quit IRC | 10:39 | |
| *** kien-ha has joined #openstack-infra | 10:39 | |
| *** mpaolino has quit IRC | 10:39 | |
| jianghuaw | wznoinsk: got the same error | 10:40 |
| vponomaryov | Hello everyone, I need to add "debootstrap" package to ubuntu-trusty, where is correct place to do it? | 10:41 |
| *** javeriak has joined #openstack-infra | 10:41 | |
| odyssey4me | vponomaryov you make use of the file other-requirements.txt in your own repository | 10:43 |
| wznoinsk | jianghuaw telnet 23.253.230.235 3306 ? | 10:43 |
| odyssey4me | vponomaryov take a look at http://docs.openstack.org/infra/bindep/ for how it works | 10:44 |
| vponomaryov | odyssey4me: other-requirements.txt intended to install system packages? | 10:44 |
| vponomaryov | odyssey4me: reading doc, thanks | 10:45 |
| jianghuaw | wznoinsk: interesting... I can reach via the ip. - error is "Connection refused". | 10:46 |
| *** markusry has quit IRC | 10:46 | |
| wznoinsk | jianghuaw: or use the port your nodepool has configured for mysql on logstash.openstack.org instead of 3306... it looks like the network unreachable error comes back from the ipv6 connection attempt, and it tries ipv6 because the first attempt on ipv4 fails for a readon, it fails for me with refused hence the mysql service is either down or we try to connect to the wrong port | 10:46 |
| sdague | yolanda / jhesketh - either of you want to help me land enforcing unit tests - https://review.openstack.org/#/c/321176/ ? | 10:47 |
| wznoinsk | jianghuaw or logstash.openstack.org does not accept connections from outside world to their mysql? | 10:47 |
| *** openstackgerrit has quit IRC | 10:47 | |
| *** openstackgerrit has joined #openstack-infra | 10:48 | |
| *** esikachev has quit IRC | 10:48 | |
| yolanda | sure | 10:48 |
| odyssey4me | vponomaryov other-requirements.txt is intended to record binary dependencies for your project, and jenkins will install them all on the node prior to executing your job | 10:50 |
| *** abregman has joined #openstack-infra | 10:50 | |
| vponomaryov | odyssey4me: I assumed exactly this after reading doc, thank you very much! | 10:50 |
| odyssey4me | vponomaryov note that if you currently do not have an other-requirements.txt file then your jobs will be using the fallback deps, so you may find that once you populate the file you'll need to add a few more that you never needed to do before | 10:50 |
| jianghuaw | wznoinsk: Thanks. I think the problem is on the logstash.openstack.org which is out of my control. Will see if it will recover sometime later. | 10:51 |
| odyssey4me | jhesketh feedback in https://review.openstack.org/319381 | 10:51 |
| *** javeriak has quit IRC | 10:52 | |
| wznoinsk | jianghuaw it may be a planned change of configuration to disalow these connections (but double check the port you should be using whether it's 3306 or a different one), it may be 'just a failure' too | 10:53 |
| *** abregman has quit IRC | 10:53 | |
| *** abregman has joined #openstack-infra | 10:54 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Only pre install packages for master jobs https://review.openstack.org/322073 | 10:55 |
| *** amitgandhinz has joined #openstack-infra | 10:55 | |
| *** johnchalekson has joined #openstack-infra | 10:57 | |
| *** javeriak has joined #openstack-infra | 10:59 | |
| *** amitgandhinz has quit IRC | 11:00 | |
| openstackgerrit | Merged openstack-infra/project-config: add python 2.7 tests to os-api-ref https://review.openstack.org/321176 | 11:01 |
| *** maishsk_ has joined #openstack-infra | 11:01 | |
| *** maishsk has quit IRC | 11:02 | |
| *** maishsk_ is now known as maishsk | 11:02 | |
| *** lezbar__ has joined #openstack-infra | 11:03 | |
| *** lezbar has quit IRC | 11:04 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/project-config: Enable check/gate tests for ansible-role-cloud-launcher project https://review.openstack.org/322081 | 11:05 |
| rcarrillocruz | yolanda: does ^ look good? | 11:07 |
| *** johnchalekson has quit IRC | 11:07 | |
| *** kien-ha has quit IRC | 11:08 | |
| *** johnchalekson has joined #openstack-infra | 11:11 | |
| *** _amrith_ is now known as amrith | 11:13 | |
| yolanda | let me see | 11:15 |
| *** thorst_ has joined #openstack-infra | 11:16 | |
| yolanda | rcarrillocruz, do you need documentation? | 11:17 |
| *** rfolco has joined #openstack-infra | 11:18 | |
| yolanda | wondering if docs-on-rtfd is needed | 11:18 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Retire openstack-ansible-py_from_git repository https://review.openstack.org/319335 | 11:18 |
| *** yamamoto has joined #openstack-infra | 11:20 | |
| *** ddieterly has joined #openstack-infra | 11:22 | |
| *** Na3iL has quit IRC | 11:23 | |
| rcarrillocruz | nice to have, other roles docs are also published | 11:23 |
| *** ihrachys has joined #openstack-infra | 11:24 | |
| *** DevBox has joined #openstack-infra | 11:24 | |
| *** EricGonczer_ has joined #openstack-infra | 11:25 | |
| *** thorst_ has quit IRC | 11:25 | |
| *** yamamoto has quit IRC | 11:26 | |
| odyssey4me | yolanda please review https://review.openstack.org/317931 when you have a moment | 11:28 |
| *** lucasagomes is now known as lucas-hungry | 11:29 | |
| openstackgerrit | Merged openstack-infra/project-config: Add Ops repo to OpenStack-Ansible https://review.openstack.org/319381 | 11:30 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Use keystoneauth1.betamax for shade mocks https://review.openstack.org/298647 | 11:32 |
| *** johnchalekson has quit IRC | 11:32 | |
| *** EricGonczer_ has quit IRC | 11:33 | |
| *** ldnunes has joined #openstack-infra | 11:33 | |
| *** johnchalekson has joined #openstack-infra | 11:34 | |
| *** EricGonczer_ has joined #openstack-infra | 11:34 | |
| *** kzaitsev_mb has joined #openstack-infra | 11:36 | |
| *** Kennan has quit IRC | 11:36 | |
| *** johnchalekson has quit IRC | 11:36 | |
| *** dave-mcnally has quit IRC | 11:37 | |
| *** johnchalekson has joined #openstack-infra | 11:38 | |
| *** Kennan has joined #openstack-infra | 11:39 | |
| yolanda | odyssey4me, approved | 11:40 |
| odyssey4me | thanks yolanda | 11:40 |
| *** johnchalekson has quit IRC | 11:41 | |
| *** bhavik has quit IRC | 11:41 | |
| *** kzaitsev_mb has quit IRC | 11:41 | |
| *** johnchalekson has joined #openstack-infra | 11:41 | |
| *** johnchalekson has quit IRC | 11:42 | |
| odyssey4me | yolanda jhesketh https://review.openstack.org/319335 is now ready for workflow when you have a moment | 11:42 |
| *** thorst_ has joined #openstack-infra | 11:42 | |
| *** johnchalekson has joined #openstack-infra | 11:42 | |
| *** johnchalekson has quit IRC | 11:43 | |
| *** johnchalekson has joined #openstack-infra | 11:43 | |
| yolanda | approved | 11:44 |
| *** dizquierdo has quit IRC | 11:44 | |
| odyssey4me | thanks yolanda | 11:45 |
| *** rhallisey has joined #openstack-infra | 11:46 | |
| *** openstackgerrit has quit IRC | 11:47 | |
| *** openstackgerrit has joined #openstack-infra | 11:48 | |
| *** ddieterly is now known as ddieterly[away] | 11:48 | |
| *** ilyashakhat has quit IRC | 11:49 | |
| *** daemontool has quit IRC | 11:51 | |
| *** daemontool has joined #openstack-infra | 11:51 | |
| openstackgerrit | Merged openstack-infra/project-config: Add Sahara role to OpenStack-Ansible https://review.openstack.org/317931 | 11:53 |
| *** aysyd has joined #openstack-infra | 11:54 | |
| *** kzaitsev_mb has joined #openstack-infra | 11:56 | |
| *** amitgandhinz has joined #openstack-infra | 11:56 | |
| *** jaosorior has quit IRC | 12:01 | |
| *** amitgandhinz has quit IRC | 12:01 | |
| *** jaosorior has joined #openstack-infra | 12:01 | |
| *** yamamoto has joined #openstack-infra | 12:01 | |
| *** psilvad has joined #openstack-infra | 12:04 | |
| *** esikachev has joined #openstack-infra | 12:04 | |
| *** yolanda has quit IRC | 12:04 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: zuul/layout: add puppet-unit 4.5 jobs https://review.openstack.org/322124 | 12:04 |
| *** yamamoto has quit IRC | 12:06 | |
| *** yolanda has joined #openstack-infra | 12:06 | |
| *** ilyashakhat has joined #openstack-infra | 12:07 | |
| *** daemontool has quit IRC | 12:08 | |
| *** markvoelker has joined #openstack-infra | 12:08 | |
| *** amrith is now known as _amrith_ | 12:08 | |
| *** daemontool has joined #openstack-infra | 12:08 | |
| *** maishsk_ has joined #openstack-infra | 12:09 | |
| *** maishsk has quit IRC | 12:09 | |
| *** maishsk_ is now known as maishsk | 12:09 | |
| *** vgridnev has joined #openstack-infra | 12:09 | |
| *** ddieterly[away] is now known as ddieterly | 12:12 | |
| *** exploreshaifali has joined #openstack-infra | 12:13 | |
| *** salv-orl_ has quit IRC | 12:13 | |
| *** salv-orlando has joined #openstack-infra | 12:16 | |
| EmilienM | hello infra, can we get a review on https://review.openstack.org/#/c/322124/ please? | 12:16 |
| odyssey4me | jhesketh yolanda the regex used for the jobs - is that bash, python, perl, ?? | 12:17 |
| *** deadnull_ has joined #openstack-infra | 12:17 | |
| *** daemontool has quit IRC | 12:18 | |
| *** daemontool has joined #openstack-infra | 12:18 | |
| *** sarob has joined #openstack-infra | 12:19 | |
| *** banix has joined #openstack-infra | 12:20 | |
| *** dmellado is now known as dmellado|lunch | 12:20 | |
| *** dmellado|lunch is now known as dmellado | 12:20 | |
| *** trown|outtypewww is now known as trown | 12:23 | |
| haypo | hi. gate-tempest-dsvm-full failed on my tiny patch for nova http://logs.openstack.org/40/322040/1/check/gate-tempest-dsvm-full/6bdad07/console.html : "devstack-gate/devstack-vm-gate.sh: No such file or directory" | 12:23 |
| *** sarob has quit IRC | 12:24 | |
| haypo | is someone aware of this issue? i see also "/tmp/ansible/bin/ansible: No such file or directory" error and ".../logs/reproduce.sh: No such file or directory" error, no idea if it's related | 12:24 |
| *** markusry has joined #openstack-infra | 12:25 | |
| haypo | hum, it looks like many files are missing | 12:25 |
| *** lucas-hungry is now known as lucasagomes | 12:28 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Skip OSA CentOS-7/Xenial role jobs for Liberty/Mitaka https://review.openstack.org/322135 | 12:29 |
| *** kgiusti has joined #openstack-infra | 12:31 | |
| *** daemontool has quit IRC | 12:31 | |
| *** daemontool has joined #openstack-infra | 12:32 | |
| *** dprince has joined #openstack-infra | 12:33 | |
| *** rodrigods has quit IRC | 12:34 | |
| *** rodrigods has joined #openstack-infra | 12:34 | |
| *** johnchalekson has quit IRC | 12:34 | |
| *** zhurong has joined #openstack-infra | 12:37 | |
| wznoinsk | haypo something's completely wrong with that build, try recheck | 12:37 |
| haypo | wznoinsk: what about debugging random bugs? :-/ | 12:38 |
| wznoinsk | it looks like network issue, the VMs are already gone I suppose so troubleshooting would be done on log files only | 12:39 |
| haypo | wznoinsk: which log files? are you able to troubleshoot this issue? | 12:40 |
| haypo | why do we get network issues? | 12:40 |
| *** yolanda has quit IRC | 12:41 | |
| wznoinsk | it could be rax related network issue - in your logs: Retrying (Retry(total=4, connect=None, read=None, redirect=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='mirror.dfw.rax.openstack.org', port=80): Read timed out. | 12:41 |
| wznoinsk | where no issue in other job but on ovh: | 12:42 |
| *** daemontool has quit IRC | 12:42 | |
| wznoinsk | http://logs.openstack.org/40/322040/1/check/gate-tempest-dsvm-neutron-full/605b6d7/console.html | 12:42 |
| *** daemontool has joined #openstack-infra | 12:42 | |
| wznoinsk | Downloading http://mirror.bhs1.ovh.openstack.org/pypi/packages/25/90/a0baec87a353c4c5418ecc974d6cc3663d4404f367ea890f0f25ba968a83/paramiko-1.16.0-py2.py3-none-any.whl (169kB) | 12:42 |
| *** doug-fish has joined #openstack-infra | 12:43 | |
| jordanP | yes network issue, it happens, you should just recheck | 12:43 |
| jordanP | there's not a lot you can do, network issues happen and will happen again | 12:43 |
| *** amoralej is now known as amoralej|lunch | 12:44 | |
| *** nwkarsten has joined #openstack-infra | 12:44 | |
| *** banix has quit IRC | 12:44 | |
| wznoinsk | is logstash borken or I'm doing something wrong? http://logstash.openstack.org/ | 12:44 |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: Add install-distro-packages template to manila-image-elements jobs https://review.openstack.org/322143 | 12:44 |
| rcarrillocruz | wznoinsk: logstash has been in migration process since yesterday | 12:45 |
| rcarrillocruz | not sure if it's done yet | 12:45 |
| rcarrillocruz | clarkb and pabelanger were working on that yesterday | 12:45 |
| wznoinsk | ok, cheers | 12:45 |
| *** tlian has joined #openstack-infra | 12:46 | |
| *** zhurong has quit IRC | 12:46 | |
| *** openstackgerrit has quit IRC | 12:48 | |
| *** yolanda has joined #openstack-infra | 12:48 | |
| *** openstackgerrit has joined #openstack-infra | 12:48 | |
| *** zhurong has joined #openstack-infra | 12:48 | |
| *** nwkarsten has quit IRC | 12:48 | |
| openstackgerrit | Merged openstack-infra/release-tools: update README for the script to expire old bug reports https://review.openstack.org/322019 | 12:49 |
| *** edmondsw has joined #openstack-infra | 12:49 | |
| openstackgerrit | Ivan Kolodyazhny proposed openstack-infra/devstack-gate: Add python-brick-cinderclient-ext workspace setup https://review.openstack.org/321845 | 12:50 |
| *** pilgrimstack has quit IRC | 12:50 | |
| *** yamamoto has joined #openstack-infra | 12:51 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Use keystoneauth1.betamax for shade mocks https://review.openstack.org/298647 | 12:51 |
| *** pilgrimstack has joined #openstack-infra | 12:51 | |
| *** coreyob has joined #openstack-infra | 12:51 | |
| *** abregman has quit IRC | 12:51 | |
| wznoinsk | jianghuaw ^ see above | 12:53 |
| *** zhurong has quit IRC | 12:54 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Notify #puppet-openstack with puppet-ceph/stable changes https://review.openstack.org/322151 | 12:54 |
| *** baoli has joined #openstack-infra | 12:54 | |
| *** zhurong has joined #openstack-infra | 12:55 | |
| *** baoli_ has joined #openstack-infra | 12:56 | |
| *** amitgandhinz has joined #openstack-infra | 12:57 | |
| pabelanger | wznoinsk: rcarrillocruz: clarkb: I have restarted jenkins-log-client on logstash.o.o, I believe things will work better now | 12:59 |
| *** zhurong has quit IRC | 12:59 | |
| fungi | jhesketh: i guess we ended up needing a zuul restart eventually too? any other persistent impact? do the network issues seem to have subsided? | 12:59 |
| *** baoli has quit IRC | 12:59 | |
| jhesketh | fungi: they seem to have subsided... no other issues that I've noticed | 13:00 |
| *** piet has joined #openstack-infra | 13:00 | |
| jhesketh | (had to clean up a few nodepool nodes to get jobs to re-register so they'd be picked up in demand calcs) | 13:00 |
| fungi | makes sense | 13:00 |
| *** |-paul-| has joined #openstack-infra | 13:01 | |
| fungi | looks like rackspace has taken the incident off their status page entirely, so no mention of the resolution time | 13:01 |
| pabelanger | jhesketh: fungi Ya, it looks like we have leaked a lot of ready nodes in nodepool | 13:01 |
| jhesketh | fungi: if you missed it, we did lose a bunch of state including ~100 results and ~4000 events | 13:01 |
| pabelanger | I can clean them up if needed | 13:01 |
| jhesketh | hopefully people saw the notice and are rechecking | 13:01 |
| jhesketh | pabelanger: yeah I didn't clean them all up so if you want to figure out what are stale that might be handy | 13:02 |
| fungi | jhesketh: yep, not much we can do about that i guess | 13:02 |
| *** amitgandhinz has quit IRC | 13:02 | |
| *** matt-borland has joined #openstack-infra | 13:02 | |
| redixin | Hi all. Does anybody know where is sources of openstack proposal bot? | 13:02 |
| redixin | fungi: ^ | 13:02 |
| pabelanger | jhesketh: sure, let me do that now | 13:02 |
| fungi | redixin: it's not a piece of software, it's just a colloquial name for a bunch of different ci jobs that use a common gerrit account to propose changes for review | 13:03 |
| *** burgerk has joined #openstack-infra | 13:03 | |
| fungi | redixin: so you'll need to be more specific about what you're looking for | 13:04 |
| *** ddieterly has quit IRC | 13:04 | |
| redixin | fungi: i trying to make something similar | 13:05 |
| *** _ari_ has joined #openstack-infra | 13:05 | |
| *** zhurong has joined #openstack-infra | 13:05 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: move puppet4 jobs into check pipeline https://review.openstack.org/321837 | 13:05 |
| fungi | redixin: the rule of thumb is that most of the time having something propose git commits for review containing autogenerated content is a terrible idea | 13:06 |
| rcarrillocruz | pabelanger: added tests for the launcher, https://review.openstack.org/#/c/322081/ enables them | 13:06 |
| fungi | redixin: we rely on it in a few cases where there is no other option, and even for those we're constantly looking for alternatives so we can stop | 13:07 |
| *** electrofelix has quit IRC | 13:07 | |
| pabelanger | rcarrillocruz: nice! | 13:07 |
| *** maishsk has quit IRC | 13:07 | |
| fungi | redixin: basically adding generated content in a revision control system runs counter to expectations and is better served through some other means of publication | 13:08 |
| *** bhavik has joined #openstack-infra | 13:08 | |
| *** nwkarsten has joined #openstack-infra | 13:08 | |
| rcarrillocruz | thx | 13:09 |
| *** pilgrimstack has quit IRC | 13:09 | |
| redixin | fungi: the second option is -1 all patches if we have buggy/vulnerable dependency in requiements.txt | 13:09 |
| *** hichihar_ has quit IRC | 13:10 | |
| fungi | redixin: buggy/vulnerable in ways which don't impact testing? | 13:10 |
| openstackgerrit | Rodrigo Duarte proposed openstack-infra/project-config: Make keystone functional tests job voting https://review.openstack.org/321890 | 13:11 |
| fungi | redixin: there's been consensus for a long time that our community isn't going to use our coordinated requirements list to communicate security vulnerabilities in our dependencies, if that's what you're suggesting | 13:11 |
| redixin | fungi: it may help to save some time. to have proposed change instead of looking for a problem with new release of whatever-pythonclient | 13:11 |
| *** _amrith_ is now known as amrith | 13:11 | |
| fungi | redixin: to what repo are you considering proposing these updates, and what would trigger that? | 13:12 |
| redixin | fungi: I mean we can have whatever-pythonclient===1.1.1 (known good version) instead of whatever-pythonclient<=1.1.1 (1.1.2 may be broken) | 13:13 |
| redixin | (in requiements.txt) | 13:13 |
| fungi | redixin: that's what upper-constraints.txt is meant to achieve | 13:13 |
| fungi | redixin: how does it not fill the need you're seeing? | 13:14 |
| redixin | fungi: so we can just have upper-constraints instead of requirements.txt? | 13:15 |
| *** |-paul-| has quit IRC | 13:15 | |
| fungi | redixin: no, they're separate mechanisms | 13:16 |
| *** alaski is now known as lascii | 13:16 | |
| redixin | fungi: hmm ill try to google about using upper constraints. thanks a lot | 13:17 |
| fungi | redixin: http://git.openstack.org/cgit/openstack/requirements/tree/README.rst | 13:18 |
| fungi | it's pretty thoroughly documented there | 13:18 |
| redixin | ok thanks | 13:19 |
| *** nwkarsten has quit IRC | 13:20 | |
| *** _vs has joined #openstack-infra | 13:20 | |
| *** akshai has joined #openstack-infra | 13:20 | |
| fungi | redixin: if you're considering altering/augmenting that, i recommend talking to the requirements team in #openstack-requirements or in their weekly meeting http://eavesdrop.openstack.org/#Requirements_Team_Meeting | 13:21 |
| *** Na3iL has joined #openstack-infra | 13:21 | |
| *** ayoung has joined #openstack-infra | 13:22 | |
| *** akshai has quit IRC | 13:25 | |
| *** xyang1 has joined #openstack-infra | 13:26 | |
| *** asettle has joined #openstack-infra | 13:27 | |
| *** markusry has quit IRC | 13:27 | |
| *** ddieterly has joined #openstack-infra | 13:29 | |
| *** akshai has joined #openstack-infra | 13:30 | |
| pabelanger | infra-root: I have restarted nodepoold, jenkins02 and jenkins06 we not responding to zmq | 13:30 |
| pabelanger | in the process of shutting each down to clean up stale nodes | 13:30 |
| *** rbradf_not_found is now known as rbradfor | 13:30 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: move puppet4 jobs into check pipeline https://review.openstack.org/321837 | 13:30 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: move xenial integrations jobs into gate https://review.openstack.org/322177 | 13:30 |
| openstackgerrit | Merged openstack-infra/project-config: Enable check/gate tests for ansible-role-cloud-launcher project https://review.openstack.org/322081 | 13:31 |
| *** amitgandhinz has joined #openstack-infra | 13:31 | |
| pabelanger | #status log nodepoold restarted to address zmq issue with jenkins02 and jenkins06 | 13:32 |
| openstackstatus | pabelanger: finished logging | 13:32 |
| *** amitgandhinz has quit IRC | 13:32 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: move xenial integrations jobs into gate https://review.openstack.org/322177 | 13:32 |
| *** whoops has quit IRC | 13:33 | |
| *** amitgandhinz has joined #openstack-infra | 13:33 | |
| *** bknudson has left #openstack-infra | 13:33 | |
| *** markusry has joined #openstack-infra | 13:35 | |
| *** _vs has quit IRC | 13:35 | |
| *** bknudson has joined #openstack-infra | 13:36 | |
| *** _vs has joined #openstack-infra | 13:37 | |
| wznoinsk | it seems I'm affected by rax outage, http://intel-openstack-ci-logs.ovh/32/321932/1/check/tempest-dsvm-intel-nfv/2db9baf/logs/devstacklog.txt.gz, I can't find how the rax.openstack.org is set as pypi index-url and where... could someone have a look and try to help me? | 13:38 |
| *** whoops has joined #openstack-infra | 13:39 | |
| *** ayoung has quit IRC | 13:40 | |
| fungi | pabelanger: thanks, the outage in dfw likely dropped the zmq connections in a less-than-graceful manner | 13:40 |
| pabelanger | fungi: Ya, jenkins02 is still struggling to come backonline | 13:40 |
| pabelanger | just growing ready nodes | 13:41 |
| *** gomarivera has joined #openstack-infra | 13:41 | |
| *** asettle has quit IRC | 13:42 | |
| *** exploreshaifali has quit IRC | 13:42 | |
| *** redixin has quit IRC | 13:42 | |
| pabelanger | going to take it out of server again, give things a moment to settle before starting it again | 13:42 |
| fungi | wznoinsk: are you maybe installing an unmodified copy of http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/files/pydistutils.cfg | 13:42 |
| *** Volundr has joined #openstack-infra | 13:43 | |
| wznoinsk | fungi quite possible but I see the same problem (trying to use rax as idnex-url) on a ndoepool node that hasn't been 'built' yet, i'm logged in to one in 'ready' state | 13:44 |
| *** whoops has quit IRC | 13:44 | |
| wznoinsk | I see pydistutils installs during setup_host but I see it before ... ^, checking elements... | 13:44 |
| fungi | wznoinsk: right, it's probably getting installed by puppet during your image builds | 13:45 |
| *** nwkarsten has joined #openstack-infra | 13:45 | |
| *** javeriak has quit IRC | 13:46 | |
| *** gomarivera has quit IRC | 13:46 | |
| *** Julien-zte has joined #openstack-infra | 13:46 | |
| *** deadnull_ has quit IRC | 13:46 | |
| *** ilyashakhat has quit IRC | 13:47 | |
| *** Goneri has joined #openstack-infra | 13:47 | |
| *** zzzeek has joined #openstack-infra | 13:48 | |
| *** _vs has quit IRC | 13:48 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool: Make nodepool cmd use logfile https://review.openstack.org/322187 | 13:48 |
| *** daemontool has quit IRC | 13:48 | |
| *** zzzeek has quit IRC | 13:49 | |
| *** zzzeek has joined #openstack-infra | 13:49 | |
| *** daemontool has joined #openstack-infra | 13:49 | |
| *** dansmith is now known as superdan | 13:50 | |
| *** _vsaienko has joined #openstack-infra | 13:53 | |
| *** zz_dimtruck is now known as dimtruck | 13:53 | |
| *** markusry has quit IRC | 13:53 | |
| *** piet has quit IRC | 13:55 | |
| *** ilyashakhat has joined #openstack-infra | 13:55 | |
| *** itisha has joined #openstack-infra | 13:55 | |
| *** rbrndt has joined #openstack-infra | 13:56 | |
| *** akshai has quit IRC | 13:56 | |
| *** ilyashakhat has quit IRC | 13:56 | |
| pabelanger | okay, jenkins02.o.o back online | 13:56 |
| pabelanger | it was in some rough shape, mulitple jenkins services running | 13:56 |
| pabelanger | decided to reboot the server and bring it up fresh | 13:57 |
| clarkb | pabelanger: that can happen if you use service restart. You need to stop, check ps, kill, check ps again, start | 13:57 |
| *** eezhova has quit IRC | 13:58 | |
| pabelanger | clarkb: ack | 13:58 |
| *** banix has joined #openstack-infra | 13:58 | |
| clarkb | their init script is of not amazing quality | 13:58 |
| clarkb | in theory it shoukd do that for you | 13:59 |
| pabelanger | I think nodepool.o.o is happy again | 13:59 |
| rcarrillocruz | yeah, jenkins 'restart' is legendary... | 13:59 |
| openstackgerrit | Merged openstack-infra/project-config: Retire openstack-ansible-py_from_git repository https://review.openstack.org/319335 | 14:01 |
| fungi | right, i usually stop, wait, kill -1, wait, kill -7... after a bit longer it'll usually die though often need to do both parent and child processes | 14:02 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool: Make nodepool cmd use logfile https://review.openstack.org/322187 | 14:02 |
| *** nadya has quit IRC | 14:02 | |
| *** johnthetubaguy_ has joined #openstack-infra | 14:02 | |
| pabelanger | fungi: hopefully not for much longer. | 14:02 |
| *** daemontool has quit IRC | 14:03 | |
| *** eharney has joined #openstack-infra | 14:03 | |
| *** piet has joined #openstack-infra | 14:03 | |
| *** daemontool has joined #openstack-infra | 14:04 | |
| *** ilyashakhat has joined #openstack-infra | 14:04 | |
| *** _vsaienko has quit IRC | 14:04 | |
| *** eezhova has joined #openstack-infra | 14:04 | |
| *** johnthetubaguy has quit IRC | 14:04 | |
| *** johnthetubaguy_ is now known as johnthetubaguy | 14:05 | |
| *** pilgrimstack has joined #openstack-infra | 14:05 | |
| *** nelsnelson has quit IRC | 14:05 | |
| *** nelsnelson has joined #openstack-infra | 14:05 | |
| fungi | pabelanger: indeed! | 14:06 |
| *** xarses has quit IRC | 14:07 | |
| *** bhavik has quit IRC | 14:08 | |
| *** _vs has joined #openstack-infra | 14:08 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: zuul/layout: run puppet unit4 jobs on puppet-ceph again https://review.openstack.org/322190 | 14:08 |
| *** markusry has joined #openstack-infra | 14:08 | |
| *** jamesmcarthur has joined #openstack-infra | 14:08 | |
| *** Ravikiran_K has joined #openstack-infra | 14:09 | |
| pabelanger | this is going to sound bad, but both OSIC and bluehost look pretty good ATM | 14:09 |
| pabelanger | err | 14:09 |
| pabelanger | bluebox | 14:09 |
| *** esikachev has quit IRC | 14:09 | |
| *** akaszuba has joined #openstack-infra | 14:10 | |
| tdasilva | hello, I have a question about pushing new releases to pypi. I followed the directions here: http://docs.openstack.org/infra/manual/creators.html#give-openstack-permission-to-publish-releases and have pushed a new release tag, but I don't see the new version in pypi | 14:10 |
| tdasilva | this is the project: https://pypi.python.org/pypi/PyECLib | 14:10 |
| wznoinsk | fungi: found it and fixed it in jenkin's home dir, I'll set it to my local pypi mirror soon | 14:11 |
| wznoinsk | thanks | 14:11 |
| *** eezhova has quit IRC | 14:12 | |
| fungi | tdasilva: let's track it down... | 14:13 |
| tdasilva | fungi: thanks! | 14:13 |
| *** tonytan4ever has joined #openstack-infra | 14:13 | |
| fungi | tdasilva: this was the tag you pushed, presumably? http://git.openstack.org/cgit/openstack/pyeclib/tag/?h=v1.2.1 | 14:13 |
| tdasilva | fungi: I tried running git os-job v1.2.1 but that returned a page with "File Not Found" | 14:14 |
| tdasilva | yes | 14:14 |
| *** eezhova has joined #openstack-infra | 14:14 | |
| fungi | tag sha is fc14225584037ee76d2bc611207f00d9ec17a33b so we should have logs at http://logs.openstack.org/fc/fc14225584037ee76d2bc611207f00d9ec17a33b/ | 14:14 |
| fungi | and yes, that's a 404 | 14:14 |
| fungi | i'll check zuul's debug log to see what happened between the tag push and the logs not uploading | 14:14 |
| *** ddieterly is now known as ddieterly[away] | 14:16 | |
| fungi | this'll take a sec. zuul makes big debug logs and it's rotated and compressed since you pushed that | 14:16 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: move puppet4 jobs into check pipeline https://review.openstack.org/321837 | 14:16 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: move xenial integrations jobs into gate https://review.openstack.org/322177 | 14:17 |
| *** nwkarsten has quit IRC | 14:17 | |
| *** yamahata has joined #openstack-infra | 14:17 | |
| *** denisra has joined #openstack-infra | 14:17 | |
| fungi | tdasilva: oh! i see it | 14:17 |
| fungi | tdasilva: your tag is not a valid pep-440 version number | 14:18 |
| *** amoralej|lunch is now known as amoralej | 14:18 | |
| *** openstackgerrit has quit IRC | 14:18 | |
| *** openstackgerrit has joined #openstack-infra | 14:18 | |
| fungi | tdasilva: to enqueue into the release pipeline, you need to match this regex http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n118 | 14:18 |
| *** nwkarsten has joined #openstack-infra | 14:18 | |
| fungi | tdasilva: basically, the "v" at the beginning is the problem | 14:19 |
| *** esikachev has joined #openstack-infra | 14:19 | |
| *** _vs has quit IRC | 14:19 | |
| *** vdrok is now known as vdrok-afk | 14:19 | |
| fungi | tdasilva: so if you push a tag named "1.2.1" instead it should work | 14:20 |
| *** pilgrimstack has quit IRC | 14:21 | |
| fungi | pabelanger: when you were cleaning up nodepool, i guess you deleted held nodes too? | 14:22 |
| *** yamahata has quit IRC | 14:22 | |
| pabelanger | fungi: Oh, sorry. possible. I don't explicitly ignore them | 14:22 |
| fungi | not a big deal, just making sure we don't have something else weird going on | 14:23 |
| pabelanger | I can update my shell scripts to do that in the future | 14:23 |
| fungi | probably a good idea in case anyone is doing something with one that can take a little time | 14:23 |
| pabelanger | Yup, again apologies | 14:23 |
| *** markusry has quit IRC | 14:24 | |
| *** rossella_s has quit IRC | 14:24 | |
| *** salv-orlando has quit IRC | 14:24 | |
| fungi | no need. i only noticed in this case because i went to clean one up i'd held yesterday and it was already gone | 14:24 |
| *** rossella_s has joined #openstack-infra | 14:24 | |
| *** woodster_ has joined #openstack-infra | 14:29 | |
| *** kushal has joined #openstack-infra | 14:29 | |
| tdasilva | fungi: thank you, will try again! | 14:30 |
| *** inc0 has joined #openstack-infra | 14:31 | |
| *** zhurong has quit IRC | 14:32 | |
| *** denisra_ has joined #openstack-infra | 14:32 | |
| *** denisra has quit IRC | 14:33 | |
| *** jaosorior has quit IRC | 14:34 | |
| *** ddieterly[away] is now known as ddieterly | 14:34 | |
| *** akaszuba has quit IRC | 14:35 | |
| fungi | infra-root: heads up... according to rackspace this is the list of our volumes which may have been impacted by the network issues in dfw http://paste.openstack.org/show/505900/ | 14:35 |
| *** dmk0202 has quit IRC | 14:35 | |
| *** akaszuba has joined #openstack-infra | 14:35 | |
| *** akshai has joined #openstack-infra | 14:35 | |
| fungi | i'm going to start checking the corresponding servers for any indication of distress | 14:36 |
| sigmavirus24 | Didn't project-config/gerrit/projects.yaml used to have a launchpad key for each particular project? | 14:36 |
| *** pt_15 has joined #openstack-infra | 14:36 | |
| fungi | sigmavirus24: it's the "groups" setting | 14:36 |
| sigmavirus24 | fungi: thanks | 14:36 |
| *** esimone has joined #openstack-infra | 14:36 | |
| *** akaszuba has quit IRC | 14:36 | |
| sigmavirus24 | so the group name is the project name used by jeepyb, then, right? | 14:37 |
| fungi | sigmavirus24: for projects doing task tracking that's their corresponding lp project name if it's not the same as the name of the repo. for projects using storyboard for task tracking it's a list of names of project-groups to which they should be added | 14:37 |
| sigmavirus24 | Thanks fungi. That clears things up | 14:37 |
| fungi | yep | 14:37 |
| *** akaszuba has joined #openstack-infra | 14:37 | |
| * sigmavirus24 suspects jeepyb just failed to update some of the bugs I was working on then | 14:38 | |
| sigmavirus24 | or launchpad failed to apply the updates or whatever | 14:38 |
| fungi | jeepyb assumes the lp project name is the same as the repo's short name (the part after the /), but if it's not you can use the groups option to override it | 14:38 |
| pabelanger | fungi: thanks! Let me know if you find anything on ES02, was considering putting it into shutdown more, so I can safe remove the volume for the migration (not to repeat our hung detach on graphite.o.o) | 14:38 |
| fungi | sigmavirus24: which change was it? i can check and make sure lp permissions look correct | 14:38 |
| *** xarses has joined #openstack-infra | 14:39 | |
| sigmavirus24 | fungi: it was a change with the openstack-ansible project from one of the other projects. The change number from yesterday was 321657 | 14:39 |
| *** EricGonczer_ has quit IRC | 14:40 | |
| jeblair | fungi: that looks very close to a list of our volumes in dfw :) | 14:41 |
| *** amrith is now known as _amrith_ | 14:41 | |
| fungi | jeblair: i think it is an exact match :/ | 14:41 |
| *** nwkarsten has quit IRC | 14:41 | |
| jeblair | fungi: afs01.dfw vicepa appears to be ro | 14:42 |
| fungi | sigmavirus24: you need to add the "OpenStack Infra (hudson-openstack)" account to https://launchpad.net/~openstack-ansible-bugs/+members so that our bug update hook has adequate permission to reassign bugs in projects for which that group is a bug supervisor | 14:42 |
| jeblair | afs02 is rw | 14:42 |
| fungi | jeblair: does that mean it switched over? | 14:42 |
| sigmavirus24 | fungi: thanks I'll make sure odyssey4me sees that | 14:42 |
| *** vdrok has joined #openstack-infra | 14:43 | |
| fungi | sigmavirus24: basically the hook tried to reassign that bug to you and leave a comment on it with a link to your change, but lp denied the api call because of insufficient permission to reassign | 14:43 |
| sigmavirus24 | weird | 14:43 |
| fungi | sigmavirus24: if the bug had already been assigned to you, the script would only have attempted to leave a comment (which would have worked fine) | 14:43 |
| sigmavirus24 | before the subproject split, jeepyb used to work for osa. | 14:43 |
| * sigmavirus24 nods | 14:44 | |
| sigmavirus24 | I've pinged the appropriate people | 14:44 |
| sigmavirus24 | Thanks fungi | 14:44 |
| *** nwkarsten has joined #openstack-infra | 14:44 | |
| fungi | so either you just didn't ever notice that it has failed in the past when a bug needed reassignment, or the bug supervisor for that project changed at some point | 14:44 |
| *** _amrith_ is now known as amrith | 14:44 | |
| *** akaszuba has quit IRC | 14:45 | |
| *** akaszuba has joined #openstack-infra | 14:45 | |
| openstackgerrit | Merged openstack-dev/hacking: Updated from global requirements https://review.openstack.org/321658 | 14:45 |
| fungi | jeblair: somehow static.o.o came out of this unscathed. it had 14x the chance to get impacted of review.o.o | 14:46 |
| jeblair | wow | 14:46 |
| * fungi won't willingly roll those dice again | 14:46 | |
| fungi | i'm assuming you saw in scrollback i had to take gerrit offline late last night and fsck /home/gerrit2, then remount it rw | 14:47 |
| jeblair | i missed the fsck part | 14:47 |
| fungi | there may be some discontinuities. we also saw frequent 500 errors from it because it looked like it kept losing contact with its trove instance | 14:47 |
| *** nelsnelson has quit IRC | 14:47 | |
| fungi | jeblair: zuul.o.o's dmesg shows some segfaults from apache mod_mem_cache btw | 14:49 |
| fungi | at least a few a day going back to the last time it rebooted, presumably much longer | 14:49 |
| fungi | jeblair: the fsck for the gerrit volume was a purely prophylactic measure; it didn't report actual corruption | 14:53 |
| jeblair | Setting free inodes count to 125879692 (was 132954091) | 14:53 |
| jeblair | Setting free blocks count to 230175018 (was 466121641) | 14:53 |
| jeblair | vicepa on afs01 reported only that | 14:53 |
| fungi | that's good at least | 14:53 |
| *** ayoung has joined #openstack-infra | 14:54 | |
| fungi | so presumably only an accounting problem with incomplete deletions | 14:54 |
| *** jlanoux has quit IRC | 14:54 | |
| fungi | oh, nevermind i read that backwards | 14:54 |
| fungi | so it found unaccounted-for inodes/blocks | 14:54 |
| openstackgerrit | Merged openstack-infra/system-config: Migrate elasticsearch to ubuntu-trusty https://review.openstack.org/320642 | 14:54 |
| jeblair | afs is salvaging volumes now... | 14:55 |
| fungi | as far as the free count was concerned | 14:55 |
| jeblair | (this is automatic at startup, progress in /var/log/openafs/SalsrvLog) | 14:55 |
| wznoinsk | jeblair hi, I think you may be able to help me with my question... this morning I had a site-wide issue for my CI, all jobs failing (as I figured it out later on it was rax outage affecting me), I'm wondering what would be the best way to 'pause' running jobs in the CI... til I troubleshoot the problem and give zuul grenn light again...? (so far I figured out two ways: 1. I could point jobs to non-existent salves in jenkins once I | 14:56 |
| wznoinsk | have all jobs registered with gearman - it will hold off submitting jobs to jenkins till there are slaves to run them, 2. stop zuul-merger causing zuul no to submit a job (as it doesn't have any OVERRIDE_ZUUL_REF to pass on)... I'm wondering is there a preferred/less hacky way to follow in such situations? | 14:56 |
| openstackgerrit | Vladyslav Drok proposed openstack-infra/project-config: Remove pxe_libvirt experimental job https://review.openstack.org/322215 | 14:56 |
| jeblair | wznoinsk: you can set jenkins in 'shutdown' mode and it won't launch any new jobs | 14:56 |
| wznoinsk | jeblair yes, I've excercies that too but ideally I want to run test jobs from within jenkins to have the params same | 14:57 |
| wznoinsk | s/excercies/exercised | 14:57 |
| *** yfried has quit IRC | 14:57 | |
| openstackgerrit | Vladyslav Drok proposed openstack-infra/project-config: Remove pxe_libvirt experimental job https://review.openstack.org/322215 | 14:57 |
| fungi | wznoinsk: maybe disable the gearman plugin in jenkins temporarily? | 14:57 |
| jeblair | yeah, that's a good one | 14:58 |
| *** akaszuba has quit IRC | 14:58 | |
| wznoinsk | yeah, gearman should then have no place to send... wouldn't it unregister jobs from gearman server then? | 14:58 |
| *** jlanoux has joined #openstack-infra | 14:59 | |
| jeblair | yes it should | 14:59 |
| jeblair | (but that's fine) | 14:59 |
| fungi | no, gearman only removes job registrations if you restart it/zuul | 14:59 |
| fungi | unless i'm misunderstanding | 14:59 |
| jeblair | oh i think we're talking about different things | 14:59 |
| fungi | what it shouldn't do is cause zuul to abort the jobs with a NOT_REGISTERED result | 15:00 |
| wznoinsk | fungi that was my impression too, when I was changing project.yaml in jbb to use different labeled slave gearman server had the new job:slave and old one job:slave registered | 15:00 |
| jeblair | disabling the gearman plugin should mean that the workers do not pick up a job from the server. the server never forgets the names of jobs that have been registered, so there will be no NOT_REGISTERED errors as long as you don't restart zuul | 15:00 |
| wznoinsk | kewl, thanks guys, I'll test it next big time | 15:01 |
| *** jistr is now known as jistr|call | 15:01 | |
| *** isaacb has joined #openstack-infra | 15:01 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: jjb/puppet: fix conditional for xenial jobs https://review.openstack.org/322216 | 15:01 |
| wznoinsk | btw. I'm trying to find a script from one of the 3rdparty Cis to generate DEVSTACK_GATE_TEMPEST_REGEX based on exlusion list... would anyone have it to hand maybe? | 15:02 |
| jeblair | the openafs client on the dfw mirror seems unhappy | 15:02 |
| *** Julien-zte has quit IRC | 15:03 | |
| fungi | yep, i was just looking at the logs | 15:03 |
| jeblair | ah, i think its afs cache volume died | 15:03 |
| fungi | the logical volume on it seems not impacted though | 15:03 |
| fungi | recovered though? | 15:04 |
| fungi | mount show it still read-write | 15:04 |
| jeblair | touch: cannot touch ‘/var/cache/openafs/foo’: Read-only file system | 15:04 |
| fungi | argh | 15:04 |
| fungi | i was having trouble sifting through all the afs-related kernel errors in dmesg | 15:04 |
| fungi | but yes, now i see that buried in amongst them | 15:04 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add release announcement jobs for major OSA repos https://review.openstack.org/319234 | 15:05 |
| wznoinsk | found it: https://github.com/a10networks-ci/neutron-thirdparty-ci/blob/master/slave/v1-testcases | 15:05 |
| jeblair | looks like we're up to using 7.4G of that now... so we might be able to move that to the ephemeral volume to increase resiliency | 15:05 |
| jeblair | (it's a bit much for locating on /) | 15:05 |
| fungi | seems reasonable | 15:06 |
| fungi | there we go... [Fri May 27 02:31:50 2016] end_request: I/O error, dev xvdb, sector 100942424[Fri May 27 02:31:50 2016] end_request: I/O error, dev xvdb, sector 100942424 | 15:06 |
| jeblair | possibly due to the kernel panics, i don't think i can recover this without a reboot | 15:06 |
| fungi | the afs errors just (barely) preceded the block device errors | 15:07 |
| *** kzaitsev_mb has quit IRC | 15:07 | |
| fungi | yep, i'd expect to reboot it anyway | 15:07 |
| fungi | (and looks like you just did) | 15:07 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Rename openstack-ansible-ironic to openstack-ansible-os_ironic https://review.openstack.org/299192 | 15:07 |
| *** ifarkas_ has quit IRC | 15:08 | |
| jeblair | oh, heh, sorry. i'm pre-breakfast and did not quite connect "fungi is reading the log files" with "fungi is logged into the system" | 15:08 |
| fungi | no worries. i was "done" anyway ;) | 15:08 |
| fungi | yep, broken, needs reboot | 15:08 |
| fungi | not much else to see there. we know what caused it | 15:09 |
| notmorgan | fungi, jeblair: I think one of the git mirrors in git.openstack.org (or two) are unhappy. | 15:09 |
| fungi | notmorgan: how recently? rackspace had some terrible network outages overnight in that region | 15:10 |
| notmorgan | fungi, jeblair: was getting weird SSL errors, followed by repositories disappearing and reapearring on refreshes of cgit page | 15:10 |
| notmorgan | fungi: ... 3 minutes ago? | 15:10 |
| fungi | oh, ick | 15:10 |
| notmorgan | notably openstack-infra namespace was missing 5-10 repos on a couple refreshes of cgit (web browsing) and was getting SSL read: error:00000000:lib(0):func(0):reason(0), errno 104 | 15:11 |
| *** tesseract has quit IRC | 15:11 | |
| notmorgan | when trying to clone. | 15:11 |
| notmorgan | hit/miss | 15:11 |
| notmorgan | so sometimes it would work. | 15:11 |
| fungi | yeah, sounds like you were sometimes getting load-balanced to a broken server. i'll see if i can find it | 15:12 |
| notmorgan | wish i could make it easier to determine which one(s) were broken | 15:12 |
| clarkb | the backens are directly accessible | 15:12 |
| *** denisra_ is now known as denisra | 15:13 | |
| fungi | as for http://paste.openstack.org/show/505900/ i've gotten through eavesdrop, so at this point just need to check on all the elasticsearch servers besides 04 | 15:13 |
| fungi | also pabelanger already checked on 02 because he's in the process of replacing it | 15:13 |
| pabelanger | Yup | 15:13 |
| fungi | here are some fun kernel messages... | 15:14 |
| fungi | [Fri May 27 15:13:38 2016] xen_netfront: xennet: skb rides the rocket: 20 slots | 15:14 |
| fungi | say wha?!? | 15:14 |
| notmorgan | fungi: heh "fun" | 15:14 |
| fungi | i'm guessing this is related to connection rate limiting for haproxy | 15:15 |
| *** jlanoux has quit IRC | 15:15 | |
| *** jlanoux has joined #openstack-infra | 15:15 | |
| *** Jeffrey4l has quit IRC | 15:16 | |
| notmorgan | ls | 15:16 |
| jeblair | No such file or directory | 15:16 |
| notmorgan | jeblair: ++ | 15:16 |
| jeblair | bandersnatch just successfully updated, so i think afs rw volumes are happy now | 15:17 |
| *** amrith is now known as _amrith_ | 15:17 | |
| *** nwkarsten has quit IRC | 15:17 | |
| *** armax has joined #openstack-infra | 15:17 | |
| *** Kaiyan has joined #openstack-infra | 15:17 | |
| EmilienM | I see a lot of RAX repos timeouts | 15:18 |
| EmilienM | can't reach http://mirror.dfw.rax.openstack.org/centos/7/updates/x86_64/repodata/repomd.xml | 15:18 |
| EmilienM | lot of jobs are currently failing | 15:18 |
| jeblair | EmilienM: yeah, i'm repairing that mirror right now | 15:18 |
| *** cody-somerville_ has quit IRC | 15:18 | |
| jeblair | should be just another minute | 15:18 |
| *** _amrith_ is now known as amrith | 15:19 | |
| *** cody-somerville has joined #openstack-infra | 15:19 | |
| jeblair | done | 15:20 |
| fungi | notmorgan: clarkb: looks like git08 is unhappy. systemd is rapidly restarting the git daemon | 15:20 |
| *** nwkarsten has joined #openstack-infra | 15:20 | |
| jeblair | fungi: where do you see that? | 15:23 |
| fungi | jeblair: dmesg -T | 15:23 |
| *** salv-orlando has joined #openstack-infra | 15:23 | |
| *** rcernin has quit IRC | 15:24 | |
| fungi | i don't see any of the other 7 git servers exhibiting this logging at least | 15:24 |
| jeblair | wow. i like that's not logged anywhere. | 15:24 |
| fungi | yeah, i'm thinking it's in the systemd journal | 15:24 |
| *** jistr|call is now known as jistr | 15:25 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/project-config: Add non-voting py34 job for zuul https://review.openstack.org/322230 | 15:25 |
| jeblair | apparently the journal runs from October through mid December | 15:25 |
| EmilienM | jeblair: cool thx | 15:25 |
| fungi | i see that | 15:25 |
| notmorgan | jeblair: well at least it's not in systemctl-journal only | 15:26 |
| openstackgerrit | Isaac Beckman proposed openstack-infra/nodepool: Add log config option to nodepool cmd https://review.openstack.org/321480 | 15:26 |
| jeblair | notmorgan: i think it is? | 15:26 |
| notmorgan | jeblair: erm... systemd... | 15:26 |
| notmorgan | jeblair: iirc on my local system i don't even see most of that stuff in dmesg | 15:26 |
| jeblair | oh, you mean the crumbs in dmesg | 15:26 |
| notmorgan | jeblair: yeah. | 15:26 |
| *** Qiming has quit IRC | 15:26 | |
| notmorgan | jeblair: i am *not* a fan of the systemd journal thing. | 15:27 |
| jeblair | apparently in december it was also starting the git daemon a lot | 15:27 |
| * notmorgan kindof misses rsyslog | 15:27 | |
| notmorgan | jeblair: interesting. wonder if it's something with that host, something with the LB sending off traffic to it, etc. | 15:28 |
| clarkb | sudo journalctl -f doesnt follow a current log? | 15:28 |
| jeblair | clarkb: nope, ends December 16 | 15:28 |
| jeblair | -- Logs begin at Tue 2015-10-27 06:05:40 UTC, end at Wed 2015-12-16 09:08:31 UTC. -- | 15:28 |
| notmorgan | oh thats fun. | 15:28 |
| * notmorgan remembers to vacuum local logs. | 15:29 | |
| jeblair | NOW WHO"S STUCK IN THE PAST, SYSTEMD! | 15:29 |
| fungi | bwahahahahaha | 15:29 |
| notmorgan | jeblair: LOL | 15:29 |
| wznoinsk | +! | 15:29 |
| *** links has quit IRC | 15:29 | |
| fungi | i love how systemctl status paginates in more which refuses to render its fancy tree characters | 15:30 |
| fungi | have to |cat to see a proper rendering | 15:30 |
| *** hongbin has joined #openstack-infra | 15:31 | |
| clarkb | I want to say on debuntu at least installing rsyslog sets up journal to rsyslog stuff. But if journald isnt recording it wont write to rsyslog either | 15:31 |
| *** gomarivera has joined #openstack-infra | 15:32 | |
| jeblair | erm, does centos even write the journal to disk? | 15:32 |
| jeblair | i can't find the file to even see what the size is... | 15:32 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add md5 files to images upload https://review.openstack.org/320906 | 15:32 |
| jeblair | journalctl --disk-usage | 15:33 |
| jeblair | Archived and active journals take up 368.0M on disk. | 15:33 |
| jeblair | /run/log! | 15:33 |
| jeblair | (i straced that command to find out where the journals were :) | 15:33 |
| jeblair | so it's in a tmpfs | 15:34 |
| fungi | i still can't find where the service definition is for the git daemon | 15:34 |
| *** mixos has joined #openstack-infra | 15:34 | |
| clarkb | fungi: its in with all the others, we write our own out though because the centos one is broken | 15:34 |
| fungi | yeah, just can't *find* it | 15:35 |
| fungi | not in /etc/systemd, not in /usr/share/systemd, not in /etc/init.d... | 15:35 |
| clarkb | I think it is in /usr/share/systemd | 15:36 |
| fungi | aha, /usr/lib/systemd/system | 15:36 |
| fungi | puppet manifest ftw | 15:36 |
| clarkb | note the filename has an @ in it because systemd uses symbols in filenames to affect behavior | 15:36 |
| *** Swami has joined #openstack-infra | 15:36 | |
| *** kzaitsev_mb has joined #openstack-infra | 15:37 | |
| *** salv-orl_ has joined #openstack-infra | 15:37 | |
| fungi | for some reason `systemctl status git` and `systemctl status git-daemon` both act like those aren't defined | 15:38 |
| jeblair | apparently it's supposed to rotate automatically | 15:38 |
| fungi | even appending the @ to them | 15:38 |
| jeblair | so basically, no idea why we stopped getting journal entries | 15:39 |
| clarkb | fungi so it does socket activate them I wonder if that causes status to be weird | 15:39 |
| * jeblair thinks we should nuke 08 and rebuild | 15:39 | |
| fungi | clarkb: yeah, the units for them must be dynamically created by socket activation because they show up like git-daemon@16841920-104.239.146.131:29418-104.130.246.128:55211.service | 15:40 |
| *** salv-orlando has quit IRC | 15:40 | |
| *** vhosakot has joined #openstack-infra | 15:41 | |
| jeblair | git07 journal ends Dec 15 | 15:41 |
| *** arxcruz has quit IRC | 15:41 | |
| *** d34dh0r53 is now known as h0m3r | 15:41 | |
| *** roxanaghe has joined #openstack-infra | 15:41 | |
| *** lezbar__ has quit IRC | 15:42 | |
| *** ddieterly is now known as ddieterly[away] | 15:43 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: Python 3 Fixes: Use print() not print https://review.openstack.org/322238 | 15:43 |
| *** jordanP has quit IRC | 15:43 | |
| jeblair | the journals on all 8 servers end either dec 15 or 16, regardless of when they started | 15:43 |
| *** sigmavirus24 is now known as m3du5a | 15:43 | |
| *** h0m3r is now known as d34dh0r53 | 15:43 | |
| jeblair | (they start various times sept thru oct) | 15:44 |
| *** m3du5a is now known as sigmavirus24 | 15:44 | |
| notmorgan | jeblair: that is weird. | 15:44 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-elasticsearch: Set permissions on /var/lib/elasticsearch https://review.openstack.org/322242 | 15:44 |
| fungi | maybe we merged a change around then to adjust their logging? | 15:44 |
| pabelanger | clarkb: ^ think that should work | 15:45 |
| *** hashar is now known as hasharAway | 15:45 | |
| fungi | that was, i think, only a few days before our gerrit upgrade though i can't think of anything related to prep for that which might cause it | 15:45 |
| *** ddieterly[away] is now known as ddieterly | 15:45 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-elasticsearch: Set permissions on /var/lib/elasticsearch https://review.openstack.org/322242 | 15:45 |
| rcarrillocruz | clarkb , pabelanger : do our centos7/trusty dib images have /usr/local/bin/env or /usr/bin/env ? | 15:46 |
| fungi | jeblair: hah! run `systemctl --failed` | 15:46 |
| fungi | systemd-journald.service loaded failed failed Journal Service | 15:46 |
| rcarrillocruz | i smell i'm getting test failures on https://review.openstack.org/#/c/322189/2/tests/inventory related to that | 15:46 |
| rcarrillocruz | issue is the shade module can't be found by ansible in the tox venv | 15:46 |
| *** nwkarsten has quit IRC | 15:46 | |
| openstackgerrit | Lukas Bednar proposed openstack-infra/jenkins-job-builder: Builders: Add ansible-playbook builder https://review.openstack.org/322243 | 15:46 |
| pabelanger | rcarrillocruz: I would think /usr/bin/env | 15:47 |
| pabelanger | but would need to confirm | 15:47 |
| pabelanger | (can't actually confirm ATM) | 15:47 |
| rcarrillocruz | i'll push a change with /usr/local/bin/env , /usr/bin/env not working in the gate | 15:47 |
| rcarrillocruz | it's not urgent, you have better things to do in the sprint | 15:48 |
| *** Swami has quit IRC | 15:48 | |
| jeblair | fungi: wow. i don't. wow. | 15:48 |
| fungi | jeblair: we could check its log to see why it... no, wait | 15:49 |
| *** Swami has joined #openstack-infra | 15:49 | |
| *** nwkarsten has joined #openstack-infra | 15:49 | |
| *** lezbar has joined #openstack-infra | 15:51 | |
| fungi | i'm going to try starting it and see if it says why it's not able to start | 15:52 |
| *** vincentll has quit IRC | 15:52 | |
| fungi | of course, it fails and suggests checking `journalctl -xe` for the cause | 15:53 |
| fungi | which... no. still has nothing since december | 15:54 |
| fungi | of course, i should look in dmesg! | 15:55 |
| fungi | [Fri May 27 15:52:38 2016] systemd-journald[10568]: Failed to get machine id: Permission denied | 15:55 |
| * fungi smells selinux at work | 15:55 | |
| *** deadnull_ has joined #openstack-infra | 15:56 | |
| fungi | https://bugzilla.redhat.com/show_bug.cgi?id=1312001 | 15:57 |
| openstack | bugzilla.redhat.com bug 1312001 in systemd "systemd-journal won't start with avc: denied" [Unspecified,Closed: worksforme] - Assigned to systemd-maint | 15:57 |
| *** ddieterly is now known as ddieterly[away] | 15:57 | |
| fungi | can someone who speaks redhatese translate that for me? | 15:58 |
| jeblair | type=AVC msg=audit(1464364360.197:185776433): avc: denied { read } for pid=10582 comm="systemd-journal" name="machine-id" dev="tmpfs" ino=7471 scontext=system_u:system_r:syslogd_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file | 15:58 |
| jeblair | fungi: i think you are correct about the selinux involvement | 15:58 |
| fungi | there is a "solution" in the reply to that bug | 15:58 |
| *** liusheng has quit IRC | 15:59 | |
| * jeblair reads bug | 15:59 | |
| ttx | odyssey4me: fwiw we don't need openstack-admins as team members in Launchpad. Just as team *owners*, so we can escalate to admin role in case of need. | 15:59 |
| fungi | suggests rerunning restorecon | 15:59 |
| *** lakshmiS has joined #openstack-infra | 15:59 | |
| *** liusheng has joined #openstack-infra | 15:59 | |
| jeblair | -rw-r--r--. root root system_u:object_r:var_run_t:s0 /etc/machine-id | 15:59 |
| odyssey4me | ttx ah ok - I'm just doing some housekeeping | 16:00 |
| ttx | odyssey4me: that way we don't have rights on everything and if we escalate for admin reasons we leave a trail | 16:00 |
| jeblair | our machine-id does in fact have the wrong label | 16:00 |
| ttx | odyssey4me: I re-deactivated us | 16:00 |
| fungi | jeblair: yeah, just confirmed it myself | 16:00 |
| odyssey4me | ttx ok, thanks | 16:00 |
| *** bpokorny has joined #openstack-infra | 16:00 | |
| *** cody-somerville has quit IRC | 16:01 | |
| clarkb | cloud init at fault maybe | 16:01 |
| ttx | odyssey4me: the reason we are added in the first place is that LP automatically adds the team owner as an 'admin' member | 16:01 |
| fungi | jeblair: not a fan of the bug resolution there, as it gives us no indication of what happened in mid-december to cause this | 16:01 |
| fungi | clarkb: yeah, there's a possible explanation. i wonder if a reboot will un-fix it again | 16:01 |
| odyssey4me | ttx ah ok, makes sense to me now | 16:02 |
| jeblair | oh wow, i just noticed something -- our logs aren't from oct 27 -- dec 16 | 16:02 |
| openstackgerrit | Lukas Bednar proposed openstack-infra/jenkins-job-builder: Builders: Add ansible-playbook builder https://review.openstack.org/322243 | 16:02 |
| jeblair | they are for oct 27 *and* dec 16 | 16:02 |
| jeblair | no other days | 16:02 |
| *** Guest53789 has quit IRC | 16:02 | |
| jeblair | where do puppet logs go on this host? | 16:04 |
| *** psachin has quit IRC | 16:04 | |
| clarkb | they might go to /var/log/messages if they somehow bypass journald | 16:05 |
| fungi | seems they don't | 16:05 |
| jeblair | clarkb: that file only contains a startup line for rsyslogd | 16:05 |
| fungi | i expect they go to /var/log/messages _by way of_ journald | 16:05 |
| jeblair | yum.log says a bunch of packages were updated on dec 16 | 16:06 |
| fungi | pretty sure the chain is log socket -> journald -> rsyslog export -> logfile | 16:06 |
| jeblair | fungi: except i don't think we have an rsyslog export configured | 16:06 |
| *** bhavik has joined #openstack-infra | 16:06 | |
| fungi | ahh | 16:06 |
| fungi | i fitured it had simply rotated away if journald had been sending nothing to it since december | 16:07 |
| fungi | er, figured | 16:07 |
| jeblair | May 22 03:24:01 git08 rsyslogd: [origin software="rsyslogd" swVersion="7.4.7" x-pid="23926" x-info="http://www.rsyslog.com"] rsyslogd was HUPed | 16:07 |
| *** trown is now known as trown|lunch | 16:08 | |
| greghaynes | sounds like what happens if you logrotate under rsyslog and don't hup it | 16:08 |
| greghaynes | having done that one before | 16:09 |
| *** isaacb has quit IRC | 16:10 | |
| fungi | yeah, i'm assuming with journald sending nothing to rsyslogd, the only thing rsyslogd is going to put in the logs is its own log entries about itself | 16:10 |
| greghaynes | or that | 16:10 |
| *** esikachev has quit IRC | 16:10 | |
| jeblair | who owns /dev/log? | 16:10 |
| jeblair | does that go to systemd? | 16:10 |
| fungi | root run barter town | 16:10 |
| fungi | ouch. if i try to echo something to it, bash: /dev/log: No such device or address | 16:11 |
| fungi | but ls shows it there | 16:11 |
| jeblair | yeah, logger gets ECONNREFUSED | 16:11 |
| *** _vs has joined #openstack-infra | 16:12 | |
| greghaynes | on a systemd system it should be journald | 16:12 |
| fungi | which makes sense with journald unable to start | 16:12 |
| *** salv-orl_ has quit IRC | 16:12 | |
| *** oanson has joined #openstack-infra | 16:13 | |
| *** jlanoux has quit IRC | 16:13 | |
| jeblair | so yeah, that seems to support the socket->journald->rsyslog->file chain | 16:13 |
| jeblair | okay, so i think the only thing left to do here wrt logging is restorecon, yeah? | 16:13 |
| jeblair | i'm kind of assuming one of those package updates on dec 16 munged the file context | 16:14 |
| openstackgerrit | Caleb Boylan proposed openstack-infra/shade: [WIP] Add function to update object metadata https://review.openstack.org/321878 | 16:14 |
| fungi | that's the only working theory i have | 16:16 |
| melwitt | logstash.openstack.org appears blank. is there a known issue about it? | 16:16 |
| fungi | were you going to run it, or shall i? | 16:16 |
| jeblair | fungi: why don't you? | 16:17 |
| fungi | melwitt: pabelanger is in the middle of replacing the elasticsearch cluster members i think, so that might be having an impact on lookups | 16:17 |
| fungi | melwitt: especially if it's the one kibana is pointed as as the master | 16:17 |
| openstackgerrit | Caleb Boylan proposed openstack-infra/shade: Make it easier to give swift objects metadata https://review.openstack.org/321835 | 16:17 |
| melwitt | fungi: ah, okay. thanks | 16:17 |
| fungi | jeblair: oh, even better! | 16:18 |
| fungi | restorecon set context /etc/machine-id->system_u:object_r:machineid_t:s0 failed:'Read-only file system' | 16:18 |
| melwitt | I went there to see if many other jobs are failing to fetch packages that I saw in a recent job http://logs.openstack.org/58/317958/4/check/gate-devstack-bashate/6ad3060/console.html#_2016-05-27_15_14_03_875 | 16:18 |
| *** ddieterly[away] is now known as ddieterly | 16:19 | |
| clarkb | melwitt: pabelanger fungi the kibana instance talks to elasticsearch02.openstack.org when it proxies iirc and that was the first one replaced | 16:19 |
| *** johnny___ has joined #openstack-infra | 16:19 | |
| clarkb | so ya definitely possible that it is related | 16:19 |
| fungi | clarkb: pabelanger: maybe kibana is still trying to connect to the old ip address and needs to re-resolve it? | 16:19 |
| jeblair | melwitt: the cause of those failures should be corrected now | 16:19 |
| *** amrith is now known as _amrith_ | 16:20 | |
| melwitt | jeblair: okay. is the "/tmp/ansible/bin/ansible: No such file or directory" also related to that? http://logs.openstack.org/58/317958/4/experimental/gate-tempest-dsvm-cells/eeed3f5/console.html#_2016-05-27_15_14_24_139 | 16:20 |
| pabelanger | clarkb: fungi Oh, maybe. I thought I restarted all the firewalls | 16:20 |
| *** nwkarsten has quit IRC | 16:21 | |
| fungi | https://bugzilla.redhat.com/show_bug.cgi?id=1231869 | 16:21 |
| openstack | bugzilla.redhat.com bug 1231869 in selinux-policy "rsyslog stops working after restart if SELinux is enabled" [Unspecified,Closed: notabug] - Assigned to mgrepl | 16:21 |
| *** yamahata has joined #openstack-infra | 16:21 | |
| jeblair | melwitt: i don't know about that | 16:22 |
| fungi | tmpfs on /etc/machine-id type tmpfs (ro,relatime,seclabel,mode=755) | 16:22 |
| pabelanger | okay, I've restarted apache on logstash.o.o | 16:22 |
| *** dtantsur is now known as dtantsur|afk | 16:22 | |
| clarkb | pabelanger: ya that could be it since it goes through the proxy there | 16:23 |
| pabelanger | clarkb: Yup, it was | 16:23 |
| jeblair | wow | 16:23 |
| jeblair | that's a tmpfs | 16:23 |
| clarkb | pabelanger: confirmed working for me now. melwitt you should be able to use it now | 16:23 |
| fungi | jeblair: seems it's also unfixable without a reboot | 16:24 |
| *** nwkarsten has joined #openstack-infra | 16:24 | |
| fungi | i guess i can try to remount it rw | 16:24 |
| melwitt | clarkb: got it, thanks! already on it doing searches :) | 16:24 |
| *** oanson has quit IRC | 16:24 | |
| jeblair | fungi: ok, though i'm leaning toward reboot | 16:25 |
| fungi | jeblair: actually, remounting it rw seems to have allowed restorecon to dtrt | 16:25 |
| jeblair | alrighty then | 16:25 |
| jeblair | fungi: maybe remount ro now? | 16:25 |
| fungi | i'll remount it ro again in a sec, yep | 16:25 |
| fungi | running fixfiles -f relabel to see if there's anything else that needs fixing | 16:26 |
| fungi | nope, that seems to have been it | 16:26 |
| fungi | hah, can't remount ro now... mount: /etc/machine-id is busy | 16:26 |
| *** kzaitsev_mb has quit IRC | 16:26 | |
| *** oanson has joined #openstack-infra | 16:27 | |
| fungi | anyway, going to try to start journald up again and see if we can get some details on why git-daemon is breaking | 16:29 |
| *** cindy has joined #openstack-infra | 16:30 | |
| fungi | still getting "systemd-journald[16236]: Failed to get machine id: Permission denied" | 16:30 |
| fungi | i guess at this point we probably need to just take it out of the pools in haproxy and reboot the server? | 16:31 |
| cindy | Hi. I have a opendaylight CI build error that i don’t understand. Any ideas? https://jenkins.opendaylight.org/releng/job/docs-verify-rtd-boron/28/ | 16:31 |
| jeblair | do we need to remove it from the pool? it doesn't handle that automatically? | 16:31 |
| fungi | i thought it didn't do health checks | 16:32 |
| clarkb | it does do health checks | 16:33 |
| fungi | cindy: i'm curious why you're asking in here about problems with a job running in opendaylight's ci | 16:33 |
| clarkb | but if you remove it without telling haproxy any existing connections can have a sad | 16:33 |
| fungi | cindy: care to elaborate? | 16:33 |
| jeblair | if it's layer7 lbing, it shouldn't need to (every req is a health check, right?) | 16:33 |
| *** oanson has quit IRC | 16:33 | |
| melwitt | hits on "/tmp/ansible/bin/ansible: No such file or directory" http://goo.gl/NQu3rk are many starting today | 16:33 |
| *** salv-orlando has joined #openstack-infra | 16:33 | |
| clarkb | jeblair: its an l3 thing iirc, it just checks a 3 way handshake | 16:33 |
| cindy | @fungi sorry, i’m not sure what room to ask about opendaylight problems, i was surprised to see it | 16:34 |
| melwitt | the share url didn't keep that I used last 7 days | 16:34 |
| clarkb | melwitt: ya the kibana 3 url sharing is somewhat hacky and doesn't pass through that value | 16:34 |
| jeblair | melwitt: have you rechecked that job since the mirror was fixed? | 16:34 |
| fungi | cindy: opendaylight isn't part of openstack as far as i know, but maybe ask in the monasca channel since this is a third party ci reporting on changes for one of their repositories | 16:35 |
| *** cody-somerville has joined #openstack-infra | 16:35 | |
| *** _vs has quit IRC | 16:36 | |
| melwitt | jeblair: not yet. I didn't know if the /tmp/ansible/bin/ansible thing was related to that | 16:36 |
| *** mhickey has quit IRC | 16:36 | |
| fungi | cindy: also the error message it's leaving on that change lists contact info for their ci linked at https://wiki.openstack.org/wiki/ThirdPartySystems/OpenDaylight_CI | 16:36 |
| jeblair | melwitt: i think rechecking may help us find out the answer to that | 16:36 |
| *** mikelk has quit IRC | 16:36 | |
| melwitt | jeblair: okay, will do | 16:36 |
| clarkb | jeblair: melwitt if I had to guess the bit that installs ansible is not run with set -e, it fails then we get to a bit that is run with err exit and it errors because no such file or dir | 16:37 |
| fungi | clarkb: so what's the preference? just reboot git08 or admin down it in the haproxy pools and then reboot it? | 16:37 |
| cindy | @fungi interesting, we recently must have added it, i’ll find out why | 16:37 |
| clarkb | fungi: admin down then reboot it is always preferable as that should more gracefully handle existing connections | 16:37 |
| fungi | cindy: they only list it as reporting on neutron changes, so maybe they've misconfigured it to start reporting on monasca changes too? | 16:37 |
| *** 32NAA99AQ has joined #openstack-infra | 16:37 | |
| jeblair | fungi: hrm, i restorecon'd and it just changed the context to system_u:object_r:machineid_t:s0 | 16:38 |
| jeblair | fungi: can you try starting again? | 16:38 |
| fungi | jeblair: that seems to have worked | 16:38 |
| jeblair | fungi: i did restorecon /etc/machine-id | 16:39 |
| *** 32NAA99AQ has quit IRC | 16:39 | |
| jeblair | (i have restarted haproxy-statsd so we get data to graphite/grafana) | 16:40 |
| *** _ari_ is now known as _ari_|afk | 16:40 | |
| fungi | jeblair: strange... here is is out of my console history http://paste.openstack.org/show/505927/ | 16:40 |
| jeblair | fungi: huh, did something set it back? | 16:41 |
| fungi | jeblair: oh! fixfiles -f relabel looks like it set it back again | 16:41 |
| jeblair | wow | 16:41 |
| jeblair | what is fixfiles? never used it | 16:41 |
| cindy | @fungi the monasca team doesn’t know why opendaylight is reporting on us. Should we go to a neutron room you think to change this? | 16:41 |
| *** thorst_ has quit IRC | 16:42 | |
| clarkb | cindy: you should contact the people running the CI and talk to them | 16:42 |
| clarkb | cindy: https://wiki.openstack.org/wiki/ThirdPartySystems/OpenDaylight_CI includes contact info | 16:42 |
| fungi | jeblair: yep http://paste.openstack.org/show/505928/ | 16:42 |
| *** _vs has joined #openstack-infra | 16:42 | |
| fungi | jeblair: fixfiles is _supposed_ to relabel according to configured policy | 16:43 |
| cindy | @clarkb thanks! I just saw the contact link above. Thanks fungi too! | 16:44 |
| *** thorst_ has joined #openstack-infra | 16:44 | |
| *** asettle has joined #openstack-infra | 16:44 | |
| *** pt_15 has quit IRC | 16:46 | |
| *** sarob has joined #openstack-infra | 16:46 | |
| fungi | jeblair: anyway, journalctl -xe has some details for us about git-daemon now | 16:47 |
| *** nwkarsten has quit IRC | 16:47 | |
| *** kzaitsev_mb has joined #openstack-infra | 16:47 | |
| *** dizquierdo has joined #openstack-infra | 16:48 | |
| *** thorst_ has quit IRC | 16:48 | |
| fungi | jeblair: i'm now thinking the service failures for git-daemon are misleading | 16:48 |
| pabelanger | jeblair: fungi: +1 for fixfiles. Recently started using it over restorecon | 16:49 |
| fungi | pabelanger: well, in this case fixfiles seems to set an incorrect label for /etc/machine-id | 16:49 |
| fungi | pabelanger: while restorecon sets a working one | 16:49 |
| *** nwkarsten has joined #openstack-infra | 16:49 | |
| *** asettle has quit IRC | 16:50 | |
| pabelanger | fungi: Odd, haven't had that issue before | 16:50 |
| *** lucasagomes is now known as lucas-dinner | 16:50 | |
| pabelanger | my issues with restorecon were running it in a chroot | 16:50 |
| pabelanger | it checks the host system for SELinux, and if not found it silently fails, and return success | 16:50 |
| *** _vs has quit IRC | 16:50 | |
| clarkb | mtreinish: you around? before I delete the old logstash.o.o can you double check that the subunit2sql things are all working as you expect? you are getting new data into mysql and the mysql proxy is functional | 16:51 |
| *** ilyashakhat has quit IRC | 16:52 | |
| *** sdake has joined #openstack-infra | 16:52 | |
| fungi | also, even with journald running again we're still not getting anything in /var/log/messages | 16:52 |
| clarkb | fungi: I think jeblair is right on centos7 (unlike debuntu) we must not get that configured when we install rsyslog | 16:52 |
| *** burgerk has quit IRC | 16:53 | |
| sarob | i have a small problem with https://review.openstack.org/#/c/320645/ not creating a new irc meeting | 16:53 |
| fungi | though the journal for the systemd-journal service indicates that it's being flooded by messages from /system.slice/system-git\x2ddaemon.slice | 16:53 |
| tdasilva | fungi: I tried to create a new tag for pyeclib '1.2.1' but 'git os-job 1.2.1' still returns with a 404, any ideas? | 16:54 |
| *** Apoorva has joined #openstack-infra | 16:54 | |
| *** javeriak has joined #openstack-infra | 16:54 | |
| *** csomerville has joined #openstack-infra | 16:55 | |
| jeblair | sarob: i'll check on it | 16:55 |
| *** cloudtrainme has joined #openstack-infra | 16:56 | |
| *** cody-somerville has quit IRC | 16:56 | |
| fungi | tdasilva: zuul's debug log says there are no jobs configured for it. do you have any release pipeline jobs set up at all? | 16:56 |
| *** liusheng has quit IRC | 16:57 | |
| fungi | tdasilva: looks like no... http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n9996 | 16:57 |
| *** liusheng has joined #openstack-infra | 16:58 | |
| *** javeriak_ has joined #openstack-infra | 16:58 | |
| openstackgerrit | Lukas Bednar proposed openstack-infra/jenkins-job-builder: Builders: Add ansible-playbook builder https://review.openstack.org/322243 | 16:58 |
| tdasilva | fungi: ah ok, so I need to add publish-to-pypi and pypy-jobs ? | 16:58 |
| *** derekh has quit IRC | 16:58 | |
| fungi | tdasilva: yeah, see http://docs.openstack.org/infra/manual/creators.html#configure-zuul-to-run-jobs | 16:59 |
| fungi | tdasilva: also python-jobs or at least something that'll run a tarball job for you | 16:59 |
| *** javeriak has quit IRC | 16:59 | |
| *** asettle has joined #openstack-infra | 16:59 | |
| *** haypo has left #openstack-infra | 16:59 | |
| fungi | tdasilva: pypy-jobs is for running your unit tests under the "pypy" interpreter, so that's probably not one you care to add | 17:00 |
| tdasilva | fungi: yeah, i have this patch up for review: https://review.openstack.org/#/c/317672/ | 17:00 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/release-tools: Added lp-tag.py tool that helps adding tags to bugs https://review.openstack.org/322270 | 17:00 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/release-tools: docs: added missing .py extension to annotate-lp-bugs example https://review.openstack.org/322271 | 17:00 |
| tdasilva | fungi: so I will just update that | 17:00 |
| clarkb | unless you want to make usre you support pypy | 17:00 |
| fungi | right | 17:00 |
| *** kzaitsev_mb has quit IRC | 17:00 | |
| tdasilva | fungi: ok, thanks for the heads up on pypy-jobs | 17:00 |
| jeblair | sarob: the publishing job raced a second one that didn't have that meeting in it. it should show up the next time a change lands to that repo | 17:01 |
| fungi | (though it's also globally nonvoting at the moment, and the version on trusty doesn't work with some projects' dependencies such as cryptography>=1.0) | 17:01 |
| tdasilva | clarkb, fungi. I read that too quickly as pypi-jobs | 17:01 |
| *** jamesmcarthur has quit IRC | 17:01 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add magnum services call to shade https://review.openstack.org/313583 | 17:01 |
| *** asettle has quit IRC | 17:02 | |
| clarkb | fungi: I was thinking about that a bit more. We can switch pypy to xenial across the board, any of the jobs that fail we delete, the rest we restrict to newer than mitaka, then email dev list and say "hey we did this if pypy is important to you let us know and we can add it back in but please get it working" | 17:03 |
| *** eezhova has quit IRC | 17:04 | |
| clarkb | that should hopeflly put us in a good position to have meaningful pypy testing going forwward because right no wI think its just a bunch tests we run that never work | 17:04 |
| fungi | clarkb: sure, seems a fine solution to me | 17:04 |
| *** jamesmcarthur has joined #openstack-infra | 17:05 | |
| *** gyee has joined #openstack-infra | 17:05 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/puppet-phabricator: Development tools for puppet-phabricator https://review.openstack.org/310320 | 17:06 |
| *** oanson has joined #openstack-infra | 17:06 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/puppet-phabricator: De-Montyfy Puppet-phabricator https://review.openstack.org/310319 | 17:06 |
| openstackgerrit | Thiago da Silva proposed openstack-infra/project-config: add python jobs to pyeclib project https://review.openstack.org/317672 | 17:06 |
| *** trown|lunch is now known as trown | 17:07 | |
| tdasilva | fungi, clarkb ^^^ | 17:07 |
| *** _sarob has joined #openstack-infra | 17:07 | |
| clarkb | and then when 2020 comes around we drop py27 and say use pypy :) | 17:07 |
| *** e0ne has quit IRC | 17:07 | |
| *** shashank_hegde has joined #openstack-infra | 17:09 | |
| *** e0ne has joined #openstack-infra | 17:09 | |
| *** sarob has quit IRC | 17:09 | |
| *** nwkarsten has quit IRC | 17:10 | |
| *** e0ne has quit IRC | 17:10 | |
| *** tonytan4ever has quit IRC | 17:10 | |
| *** akshai has quit IRC | 17:10 | |
| *** cloudtrainme has quit IRC | 17:10 | |
| *** nwkarsten has joined #openstack-infra | 17:11 | |
| *** oanson has quit IRC | 17:11 | |
| *** Goneri has quit IRC | 17:12 | |
| *** jheroux has joined #openstack-infra | 17:12 | |
| *** eezhova has joined #openstack-infra | 17:13 | |
| *** dguitarbite_ has joined #openstack-infra | 17:14 | |
| fungi | i'm not getting anywhere on the restarting git processes on git08 | 17:14 |
| fungi | the journal doesn't include anything useful, just startup messages | 17:14 |
| clarkb | are we sure those git daemons are non functional? they are being inetd'd basically by systemd using socket activation | 17:15 |
| fungi | actually, the flood of them in dmesg ceased at 16:39 utc | 17:16 |
| clarkb | git clone git://git08.openstack.org:29418/openstack/neutron seems to work from here | 17:16 |
| fungi | which is exactly when i started systemd-journald again | 17:16 |
| fungi | spooky | 17:17 |
| *** csomerville has quit IRC | 17:17 | |
| *** abregman has joined #openstack-infra | 17:17 | |
| *** gyee has quit IRC | 17:18 | |
| notmorgan | fungi: weird. | 17:18 |
| *** ilyashakhat has joined #openstack-infra | 17:18 | |
| fungi | ahh, because it's going into the journal and not being reported back to dmesg i guess? i see some similar starting/started messages looking at journalctl -xe | 17:19 |
| clarkb | did we change anything on our end to make osic happier? we seem to be using a proper amount of quota there now | 17:19 |
| *** ayoung has quit IRC | 17:20 | |
| jeblair | fungi: is the git daemon an inet daemon, and so perhaps those messages are normal? | 17:20 |
| *** kzaitsev_mb has joined #openstack-infra | 17:20 | |
| *** gyee has joined #openstack-infra | 17:20 | |
| *** kushal has quit IRC | 17:20 | |
| notmorgan | jeblair: that seems... odd to make it inet. | 17:20 |
| clarkb | jeblair: basically yes, it uses systemd's similar functionality (this is what the @ in the name means) | 17:20 |
| notmorgan | jeblair: i mean, i don't knpow the best practices today but that seems weird to me. | 17:21 |
| clarkb | notmorgan: its actually how you are supposed to do things with systemd now didn'y you know? | 17:21 |
| * notmorgan remembers a big push to move away from inet-like things. | 17:21 | |
| jeblair | notmorgan: i don't think there is another option with the git protocol | 17:21 |
| notmorgan | jeblair: ah. | 17:21 |
| clarkb | jeblair: that too | 17:21 |
| notmorgan | jeblair: that makes more sense | 17:21 |
| fungi | jeblair: possibly, though we've got 4 git-daemon processes on our other servers and only one on git08. also systemctl --failed reports failed git-daemon processes on 08 not not on others | 17:21 |
| *** mixos has quit IRC | 17:21 | |
| notmorgan | clarkb: let me just get angry and find a soapbox for more ... reasons about systemd now. :P | 17:21 |
| *** ociuhandu has quit IRC | 17:22 | |
| fungi | no, i take that back, we have a varying number. i just got lucky on some spot checks | 17:22 |
| clarkb | notmorgan: basically you put an @ in the unit file name (says allow me to run many instances of this) then set up the socket activation stuff and you have an inet | 17:22 |
| fungi | however, systemctl --failed is only reporting failed git-daemon services on 08 | 17:22 |
| jeblair | fungi: perhaps the failed ones are just individual spawn instances that have failed for some reason or other? | 17:23 |
| *** cindy has left #openstack-infra | 17:23 | |
| jeblair | the unit name "git-daemon@16841920-104.239.146.131:29418-104.130.246.128:55211.service" looks very specific ... | 17:23 |
| fungi | jeblair: seems probable, but would be good to know what that reason is | 17:23 |
| notmorgan | clarkb: i *still* don't like that. it makes my skin crawl [then again i don't like piling tons of things on the same system, so dynamic up/down scaling is less predictable]-- i might be stuck in the past of systemsengineering/admin though :P | 17:23 |
| *** nwkarsten has quit IRC | 17:24 | |
| clarkb | notmorgan: the argument for it is it allows your system to only do the work necessary rather than having a gazillion daemons all hanging out waiting for connections. | 17:24 |
| *** salv-orlando has quit IRC | 17:24 | |
| *** nwkarsten has joined #openstack-infra | 17:24 | |
| mtreinish | clarkb: I'm around now, what do I need to check? | 17:24 |
| *** kushal has joined #openstack-infra | 17:25 | |
| fungi | jeblair: if you `sudo systemctl status -l git-daemon@16841920-104.239.146.131:29418-104.130.246.128:55211` for example, you'll see they're recent | 17:25 |
| clarkb | mtreinish: just double check that the db is being updated properly and the mysql proxy works | 17:25 |
| clarkb | mtreinish: we replaced the instance so before I delete the old one want to make sure the new one is happy | 17:25 |
| *** shashank_hegde has quit IRC | 17:25 | |
| mtreinish | clarkb: nope, can't connect | 17:25 |
| notmorgan | clarkb: sure. i also have historically been dealing with environments where it's spike-y enough to justify hanving daemons lingering around (video games), and spinning up new servers/vms on demand with fixed utilization to handle larger/smaller loads (since you can't share the system resources cleanly) | 17:25 |
| mtreinish | clarkb: http://paste.openstack.org/show/505935/ | 17:25 |
| notmorgan | clarkb: so different backgrounds ;) | 17:26 |
| *** ilyashakhat has quit IRC | 17:26 | |
| *** HeOS has quit IRC | 17:27 | |
| *** Goneri has joined #openstack-infra | 17:27 | |
| *** tqtran has joined #openstack-infra | 17:27 | |
| clarkb | mtreinish: start-stop-daemon: user 'logstash' not found | 17:28 |
| clarkb | mtreinish: we are using a user that isn't on that host beacuse we don't install logstash there | 17:28 |
| *** ddieterly is now known as ddieterly[away] | 17:28 | |
| *** nwkarsten has quit IRC | 17:28 | |
| *** gomarivera has quit IRC | 17:29 | |
| mtreinish | clarkb: hmm, ok. I guess we should update the puppet setting up simpleproxy | 17:29 |
| clarkb | mtreinish: this is a simple fix, will have a patch soon | 17:29 |
| mtreinish | clarkb: ok | 17:29 |
| mtreinish | clarkb: also looking at openstack-health it doesn't look like there is any data in the db since 2:00am (I think it's utc, but I'm not sure) | 17:30 |
| *** maestro has joined #openstack-infra | 17:32 | |
| clarkb | mtreinish: I have kicked the subunit gearman worker it was hanging out waiting for gearman jobs and not getting any | 17:32 |
| *** yamamoto has quit IRC | 17:33 | |
| *** yamamoto has joined #openstack-infra | 17:34 | |
| pabelanger | fungi: I've manually promoted 315894,3 in the gate queue, the tox-db-legacy_drivers job was hung at 4+ hours and I couldn't see any nodes it was actually using | 17:34 |
| pabelanger | that should help clear out the integrated queue | 17:34 |
| *** pfallenop has quit IRC | 17:35 | |
| pabelanger | same problem is happening for a few jobs in check | 17:35 |
| *** ayoung has joined #openstack-infra | 17:35 | |
| *** yamamoto has quit IRC | 17:37 | |
| *** Kaiyan has quit IRC | 17:39 | |
| *** Na3iL has quit IRC | 17:40 | |
| *** ihrachys has quit IRC | 17:40 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-simpleproxy: Create a simpleproxy user https://review.openstack.org/322284 | 17:40 |
| mtreinish | clarkb: on o-h it looks like subunit2sql just started getting data again | 17:40 |
| clarkb | mtreinish: pabelanger ^ I think that is the fix for the proxy | 17:41 |
| pabelanger | clarkb: where does that run? | 17:43 |
| pabelanger | never see simpleproxy before | 17:43 |
| *** vdrok has quit IRC | 17:43 | |
| pabelanger | seen* | 17:43 |
| *** vdrok has joined #openstack-infra | 17:44 | |
| clarkb | pabelanger: on logstash.openstack.org, it provides read only access to the trove subunit2sql db | 17:44 |
| pabelanger | thanks | 17:44 |
| clarkb | otherwise you have to be on the rax network and know what the instance name/ip is | 17:44 |
| *** vdrok has quit IRC | 17:45 | |
| *** vdrok has joined #openstack-infra | 17:45 | |
| *** nadya has joined #openstack-infra | 17:45 | |
| *** vdrok has quit IRC | 17:46 | |
| *** roxanaghe has quit IRC | 17:46 | |
| *** pfallenop has joined #openstack-infra | 17:47 | |
| *** ociuhandu has joined #openstack-infra | 17:47 | |
| *** roxanaghe has joined #openstack-infra | 17:48 | |
| mtreinish | pabelanger: oh, you missed way back when I was working on getting a proxy setup. Had a lot of fun trying to get mysql proxy to work | 17:49 |
| *** thorst_ has joined #openstack-infra | 17:49 | |
| mtreinish | it turns out you could DOS mysql proxy with telnet | 17:49 |
| mtreinish | so we just went with a tcp proxy | 17:49 |
| *** links has joined #openstack-infra | 17:49 | |
| *** thorst_ has quit IRC | 17:50 | |
| *** thorst_ has joined #openstack-infra | 17:50 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci: Change DLRN promote method https://review.openstack.org/321801 | 17:51 |
| *** sdague has quit IRC | 17:51 | |
| pabelanger | mtreinish: Oh, neat | 17:51 |
| clarkb | infra-root crinkle https://review.openstack.org/322284 will get logstash.o.o sorted and we can finish its trusty upgrade | 17:51 |
| *** _vs has joined #openstack-infra | 17:51 | |
| *** shashank_hegde has joined #openstack-infra | 17:52 | |
| *** twm2016 has joined #openstack-infra | 17:52 | |
| pabelanger | infra-root: jenkins05 looks offline, going to start the recovery process | 17:52 |
| *** sdake_ has joined #openstack-infra | 17:52 | |
| twm2016 | Is this channel the place to ask questions related, to gerrit? | 17:52 |
| mtreinish | pabelanger: https://bugs.launchpad.net/ubuntu/+source/mysql-proxy/+bug/1402011 | 17:53 |
| openstack | Launchpad bug 1402011 in mysql-proxy (Ubuntu) "telnet crashes mysql-proxy" [Undecided,New] | 17:53 |
| clarkb | twm2016: if you are looking for review.openstack.org help then yes, but general gerrit questions may be better directed at #gerrit or their google group (we can still try to help though) | 17:53 |
| crinkle | clarkb: i don't think the user resource creates the homedir automatically, is that okay? | 17:53 |
| mtreinish | it doesn't seem to have moved at all, I guess it wasn't a high priority | 17:53 |
| clarkb | crinkle: yup the package creates that dir | 17:53 |
| clarkb | crinkle: which is why I chose it, you get dumped into the help docs dir if you ever su to that user | 17:54 |
| twm2016 | clarkb: thanks, just asking for a friend :) | 17:54 |
| crinkle | ah fungi got it | 17:54 |
| fungi | clarkb: crinkle: my only concern is that it might introduce a bootstrapping ordering issue since the user isn't created until after the service is installed/configured | 17:55 |
| *** sdake has quit IRC | 17:55 | |
| *** twm2016 has left #openstack-infra | 17:55 | |
| *** vhosakot has quit IRC | 17:55 | |
| clarkb | fungi: oh does the service.pp not depend on the init.pp as a whole? | 17:55 |
| fungi | if the simpleproxy package tries to start the service at installation, it will likely fail | 17:55 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: tempest: move puppet jobs from exp to check pipeline https://review.openstack.org/321174 | 17:55 |
| clarkb | if so then yes | 17:55 |
| EmilienM | oomichi: ^ | 17:55 |
| clarkb | (I sort of operated under the assumption it did but that wasn't double checked by me) | 17:55 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: move puppet4 jobs into check pipeline https://review.openstack.org/321837 | 17:56 |
| *** julim has quit IRC | 17:56 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: move xenial integrations jobs into gate https://review.openstack.org/322177 | 17:56 |
| *** vhosakot has joined #openstack-infra | 17:57 | |
| mtreinish | crinkle: any ideas on what I did wrong here: https://review.openstack.org/#/c/321147/ I'm not sure how that change broke the beaker tests | 17:57 |
| *** bpokorny_ has joined #openstack-infra | 17:57 | |
| fungi | clarkb: there's likely a bit of a dependency loop there if you rely on the package to create the homedir, but configure the package to start as a user that depends on the package being installed first | 17:57 |
| pabelanger | #status log jenkins05.o.o back online | 17:58 |
| openstackstatus | pabelanger: finished logging | 17:58 |
| *** deadnull_ has quit IRC | 17:58 | |
| fungi | clarkb: er, configure the service to start, i meant | 17:58 |
| clarkb | fungi: it should be package <- user <- service | 17:58 |
| crinkle | mtreinish: looks like its 500ing :( | 17:58 |
| clarkb | with package and user happening first because they are in the init manifest and service happening after because it is in th service manifest but I don't know that this bit is enforced inthe puppet | 17:59 |
| *** eezhova has quit IRC | 17:59 | |
| pabelanger | and cleaning up jenkins06 now | 17:59 |
| *** _vs has quit IRC | 17:59 | |
| mtreinish | crinkle: oh, I know what it is thanks | 18:00 |
| mtreinish | crinkle: http://logs.openstack.org/47/321147/2/check/gate-puppet-openstack_health-puppet-beaker-rspec-ubuntu-trusty/b4a3b2e/logs/apache/openstack-health-api-error.txt.gz | 18:00 |
| crinkle | mtreinish: neat | 18:00 |
| clarkb | ya it only requires the package not the entirety of init.pp | 18:00 |
| *** bpokorny has quit IRC | 18:01 | |
| EmilienM | hey infra folks, I know most of you are busy in sprint but if someone has time, I have some project-config changes to improve our Puppet OpenStack CI, https://goo.gl/Sa8cSx - thanks | 18:01 |
| mtreinish | crinkle: https://review.openstack.org/#/c/321202/4 should fix it (well by accident I fixed it in there without even thinking about it) | 18:01 |
| *** EricGonczer_ has joined #openstack-infra | 18:02 | |
| *** Goneri has quit IRC | 18:02 | |
| *** cloudtrainme has joined #openstack-infra | 18:02 | |
| fungi | clarkb: oh, i see the service definition is elsewhere | 18:02 |
| crinkle | mtreinish: accidentally fixing things is the best | 18:02 |
| fungi | clarkb: so probably not an issue as long as the simpleproxy package doesn't try to start the service automatically at installation (which is somewhat typical for debian packages, but maybe not this one as it's a lot more generic and configuration-dependent( | 18:03 |
| mtreinish | fungi: luckily simpleproxy isn't actually a daemon so that shouldn't be an issue | 18:04 |
| fungi | clarkb: and also, we're feedint it our own initscript, so almost certainly not | 18:04 |
| mtreinish | fungi: we wrote our own initscript for it | 18:04 |
| fungi | yeah, just realized that | 18:04 |
| clarkb | fungi: right | 18:04 |
| fungi | so anyway, lgtm | 18:04 |
| *** dizquierdo has quit IRC | 18:05 | |
| pabelanger | #status log jenkins06.o.o back online | 18:06 |
| openstackstatus | pabelanger: finished logging | 18:06 |
| *** gomarivera has joined #openstack-infra | 18:06 | |
| *** sdake_ is now known as sdake | 18:06 | |
| *** flepied has quit IRC | 18:06 | |
| *** cloudtrainme has quit IRC | 18:07 | |
| *** Sukhdev_ has joined #openstack-infra | 18:07 | |
| *** EricGonc_ has joined #openstack-infra | 18:08 | |
| notmorgan | jeblair: so in doing the py3 things for nodepool (easy way to look at all the code), i think I'm going to marshal b'' -> str vs force everything to b''. it seems like it would be more fragile/harder to maintain | 18:09 |
| notmorgan | jeblair: unless i am misunderstanding something about gear and a requirement for things to be in b'' form (i know ZK is coming, but i'm getting familiar with the codebase) | 18:10 |
| *** links has quit IRC | 18:10 | |
| clarkb | notmorgan: gear shuffles bytes around not python strings | 18:11 |
| notmorgan | clarkb: ok, so i'll need to be aware that when it drops into gear it needs to be marshalled back to bytes | 18:11 |
| clarkb | so at least at the edges where you submit and receive jobs you will need to encode/decode | 18:11 |
| *** EricGonczer_ has quit IRC | 18:11 | |
| fungi | yeah, dealing with data at the protocol level there | 18:12 |
| *** jheroux has quit IRC | 18:12 | |
| notmorgan | clarkb: yeah. thats fine -- easier to not need to remember to make everything b'' in the codebase though | 18:12 |
| fungi | if you do have wrapper functions which are handling the protocol layer communication and do all the encoding/decoding within them, then the rest of the program can just assume strings and not need to care | 18:13 |
| *** jamesmcarthur has quit IRC | 18:13 | |
| *** degorenko is now known as _degorenko|afk | 18:15 | |
| fungi | which i guess is another way to describe marshalling | 18:16 |
| *** mtanino has joined #openstack-infra | 18:16 | |
| *** mtanino has quit IRC | 18:16 | |
| *** kzaitsev_mb has quit IRC | 18:16 | |
| *** piet has quit IRC | 18:20 | |
| *** piet has joined #openstack-infra | 18:21 | |
| fungi | okay, i've gotten systemd-journald successfully started again on all the git servers | 18:21 |
| notmorgan | fungi: basically thats the plan I'm going with | 18:23 |
| fungi | strangely, the rngd service is reported as failed on all the git servers except git08 | 18:23 |
| *** yamahata has quit IRC | 18:23 | |
| notmorgan | fungi: woo ^5, glad my "hey these servers are b0rked" comment helped discover a separate issue. | 18:23 |
| fungi | well, i still haven't gotten to the bottom of the git issues you were seeing | 18:24 |
| *** yamahata has joined #openstack-infra | 18:24 | |
| notmorgan | fungi: but hey, not having systemd-journal running is bad, so... | 18:25 |
| *** nwkarsten has joined #openstack-infra | 18:25 | |
| *** Goneri has joined #openstack-infra | 18:25 | |
| fungi | Starting Hardware RNG Entropy Gatherer Daemon... Unable to open file: /dev/tpm0... can't open any entropy source... Maybe RNG device modules are not loaded... rngd.service: main process exited, code=exited, status=1/FAILURE | 18:26 |
| notmorgan | fungi: maybe the physical hosts under those VMs aren't exposing it? | 18:26 |
| fungi | possible | 18:27 |
| *** ddieterly[away] has quit IRC | 18:28 | |
| fungi | well, on git08 i still get "Unable to open file: /dev/tpm0" during startup, but the service is up and running | 18:29 |
| *** pvaneck has joined #openstack-infra | 18:29 | |
| fungi | so presumably it found a different entropy source. maybe lsmod will enlighten me | 18:29 |
| clarkb | hrm we may not have the entropy package thing installed on centos | 18:30 |
| clarkb | we do on ubuntu | 18:30 |
| clarkb | haveged? | 18:30 |
| *** jerryz has joined #openstack-infra | 18:30 | |
| *** kushal has quit IRC | 18:33 | |
| fungi | the only kernel module difference between 01 and 08 is that 01 has intel_rapl loaded, so i doubt that's related | 18:33 |
| fungi | haveged isn't installed on either of them | 18:34 |
| fungi | though we should probably do that | 18:34 |
| *** nwkarsten has quit IRC | 18:35 | |
| *** nwkarsten has joined #openstack-infra | 18:35 | |
| openstackgerrit | Merged openstack-infra/puppet-simpleproxy: Create a simpleproxy user https://review.openstack.org/322284 | 18:36 |
| fungi | cpu flags ftw! | 18:36 |
| fungi | on git08, /proc/cpuinfo indicates rdrand is present, while on git01 it is not | 18:36 |
| *** yamamoto has joined #openstack-infra | 18:38 | |
| *** piet has quit IRC | 18:38 | |
| jeblair | clarkb, fungi, mtreinish, (where is sdague?): http://www.fedmsg.com/en/latest/ | 18:38 |
| *** sdague has joined #openstack-infra | 18:39 | |
| mtreinish | jeblair: I think sdague is at home depot | 18:39 |
| *** piet has joined #openstack-infra | 18:39 | |
| jeblair | maybe i will see him there this weekend | 18:39 |
| fungi | heh | 18:39 |
| clarkb | I should be at home depot | 18:39 |
| sdague | I just got back | 18:39 |
| notmorgan | fungi: yep. ok | 18:39 |
| jeblair | (i prefer lowes when possible; it's not always possible) | 18:39 |
| *** nwkarsten has quit IRC | 18:39 | |
| jeblair | sdague: http://www.fedmsg.com/en/latest/ | 18:40 |
| notmorgan | fungi: so it's a VM / host issue *shrug* | 18:40 |
| clarkb | jeblair: that looks like ti uses zmq | 18:40 |
| jeblair | bummer | 18:40 |
| clarkb | which automatically sort of puts it in the bin of probably not a good idea for me | 18:40 |
| jeblair | it's in my don't touch it with a 10ft pole bin | 18:40 |
| *** e0ne has joined #openstack-infra | 18:40 | |
| sdague | yeh, fedmsg seems neat in concept, I just wish they used a proper bus | 18:40 |
| sdague | I conceptually want the same thing as fedmsg | 18:41 |
| jeblair | i'll ask em | 18:41 |
| clarkb | but with proper error handling | 18:41 |
| *** maestro has quit IRC | 18:41 | |
| fungi | (but with more working!) | 18:41 |
| *** EricGonc_ has quit IRC | 18:42 | |
| sdague | right, the nice thing about mosquitto is there is a ton of stuff to talk mqtt, including arduino code :) | 18:42 |
| *** ilyashakhat has joined #openstack-infra | 18:43 | |
| *** banix has quit IRC | 18:43 | |
| sdague | https://github.com/mqtt/mqtt.github.io/wiki/libraries | 18:44 |
| *** roxanaghe has quit IRC | 18:44 | |
| clarkb | 0mq also has a really volatile community | 18:44 |
| clarkb | there are 2 or 3 forks now that have decided backward compat is impossible and even they have then had similar issues with development | 18:45 |
| sdague | right, it's also much lower level and you have to build the semantics yourself. vs. a semantic pub / sub | 18:45 |
| jeblair | yeah, i like what i've seen of mqtt | 18:46 |
| sdague | the retain and will concepts also let you build proactive status reporting, where you can make a part of you subtree the pub status, so easy to know that a publisher went goofy | 18:46 |
| *** yamamoto has quit IRC | 18:48 | |
| *** ilyashakhat has quit IRC | 18:49 | |
| mtreinish | ooh, http://status.openstack.org/openstack-health/#/ finally is showing a elastic-recheck hit (well 3 of them) | 18:50 |
| clarkb | mtreinish: did we do a mass cleanup of the bug list yet? | 18:50 |
| mtreinish | clarkb: yeah I did that before we landed the o-h change | 18:50 |
| mtreinish | clarkb: https://review.openstack.org/#/c/315765/ | 18:51 |
| *** _sarob has quit IRC | 18:51 | |
| clarkb | nice | 18:51 |
| *** amoralej is now known as amoralej|off | 18:52 | |
| *** markusry has joined #openstack-infra | 18:52 | |
| *** gomarivera has quit IRC | 18:53 | |
| *** javeriak_ has quit IRC | 18:55 | |
| *** javeriak has joined #openstack-infra | 18:55 | |
| clarkb | mtreinish: ok try the mysql proxy now | 18:56 |
| mtreinish | clarkb: it works! | 18:57 |
| clarkb | yay, so from your end we are good ya? | 18:57 |
| mtreinish | clarkb: I think so | 18:57 |
| *** csomerville has joined #openstack-infra | 18:57 | |
| clarkb | great any objectsion to deleting the old logstash.o.o to complete the trusty update? | 18:57 |
| mtreinish | clarkb: go for it | 18:58 |
| *** bpokorny_ has quit IRC | 18:58 | |
| *** flepied has joined #openstack-infra | 18:58 | |
| *** Sukhdev_ has quit IRC | 18:59 | |
| *** bpokorny has joined #openstack-infra | 18:59 | |
| *** somerville32 has joined #openstack-infra | 18:59 | |
| *** markusry has quit IRC | 19:01 | |
| *** csomerville has quit IRC | 19:02 | |
| mtreinish | clarkb, fungi, jeblair, nibalizer: I'm getting 500s on: http://health.openstack.org/tests/recent/fail if you get a sec can you pull out the stacktrace from the log | 19:02 |
| nibalizer | oh i got it | 19:02 |
| nibalizer | i have my script and everything | 19:02 |
| nibalizer | http://paste.openstack.org/show/505946/ boom | 19:03 |
| *** ayoung has quit IRC | 19:03 | |
| *** ayoung has joined #openstack-infra | 19:03 | |
| mtreinish | nibalizer: heh, nice | 19:03 |
| mtreinish | oh, that's more than I was expecting | 19:03 |
| mtreinish | hmm, half of that is elasticsearch errors | 19:04 |
| *** mtanino has joined #openstack-infra | 19:04 | |
| mtreinish | the other stuff looks like it's related to the change which just landed | 19:04 |
| clarkb | mtreinish: pabelanger is in the process of upgrading the entire cluster right now | 19:04 |
| clarkb | which could slow queries and or make hosts unavailable | 19:04 |
| mtreinish | clarkb: yeah I figured that's what the es errors were related too | 19:05 |
| mtreinish | s/too/to | 19:05 |
| *** sdake_ has joined #openstack-infra | 19:09 | |
| clarkb | nibalizer: any chance yo uare going to be able to work on puppetdb today? I see your name on it | 19:10 |
| *** sdake has quit IRC | 19:10 | |
| *** inc0 has quit IRC | 19:11 | |
| *** _ari_|afk has quit IRC | 19:11 | |
| nibalizer | yes my name is on it | 19:13 |
| *** e0ne has quit IRC | 19:13 | |
| nibalizer | i have an errand to run but can kick it this afternoon | 19:14 |
| nibalizer | if someone else really wants it they can go for it | 19:14 |
| nibalizer | there is a puppetdb01 that got created but it doesn't work :( | 19:14 |
| mtreinish | nibalizer: https://review.openstack.org/#/c/322304/ should fix it | 19:14 |
| openstackgerrit | Antoine Musso proposed openstack/diskimage-builder: dpkg: fake initctl version now parseable by puppet https://review.openstack.org/322305 | 19:14 |
| *** nadya has quit IRC | 19:15 | |
| *** ddieterly has joined #openstack-infra | 19:17 | |
| *** e0ne has joined #openstack-infra | 19:18 | |
| *** e0ne has quit IRC | 19:18 | |
| *** e0ne has joined #openstack-infra | 19:18 | |
| *** sdake has joined #openstack-infra | 19:18 | |
| *** daemontool has quit IRC | 19:18 | |
| *** eezhova has joined #openstack-infra | 19:19 | |
| *** rbrndt has quit IRC | 19:19 | |
| *** sdake_ has quit IRC | 19:19 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-bandersnatch: Fix acceptance tests https://review.openstack.org/320068 | 19:20 |
| *** e0ne has quit IRC | 19:20 | |
| *** javeriak_ has joined #openstack-infra | 19:20 | |
| *** ilyashakhat has joined #openstack-infra | 19:20 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Upgrade Laravel and ORM https://review.openstack.org/322307 | 19:21 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-bandersnatch: Fix acceptance tests https://review.openstack.org/320068 | 19:22 |
| *** javeriak has quit IRC | 19:24 | |
| *** e0ne has joined #openstack-infra | 19:25 | |
| *** e0ne has quit IRC | 19:27 | |
| *** gomarivera has joined #openstack-infra | 19:27 | |
| *** burgerk has joined #openstack-infra | 19:28 | |
| *** dimtruck is now known as zz_dimtruck | 19:28 | |
| *** eezhova has quit IRC | 19:29 | |
| *** ddieterly has quit IRC | 19:31 | |
| *** e0ne has joined #openstack-infra | 19:31 | |
| *** chem`` has quit IRC | 19:31 | |
| *** chem`` has joined #openstack-infra | 19:31 | |
| *** burgerk has quit IRC | 19:33 | |
| *** e0ne has quit IRC | 19:34 | |
| *** markusry has joined #openstack-infra | 19:37 | |
| fungi | clarkb: it looks like we only install haveged on job nodes and on servers where we install kerberos. i wonder if we should expand it to be installed in our template class or something? | 19:39 |
| *** salv-orlando has joined #openstack-infra | 19:39 | |
| clarkb | probably a good idea | 19:39 |
| fungi | at least that's what i'm interpreting http://codesearch.openstack.org/?q=haveged to indicate | 19:40 |
| clarkb | It doesnt reduce our security tremendously right? | 19:41 |
| *** mixos has joined #openstack-infra | 19:41 | |
| fungi | shouldn't reduce it at all | 19:41 |
| *** gomarivera has quit IRC | 19:41 | |
| fungi | mixing more sources into an entropy pool, as long as the mixing algorithm is cryptographically sound, should never reduce the amount of entropy | 19:42 |
| fungi | even if the additional sources are completely non-entropic | 19:42 |
| *** openstack has joined #openstack-infra | 21:43 | |
| *** lascii is now known as alaski | 21:43 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/nodepool: Python 3 Fix: cmp -> key function https://review.openstack.org/321919 | 21:46 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/nodepool: Python 3 fix: Use new-style raise syntax https://review.openstack.org/321926 | 21:46 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/nodepool: Python 3 Fixes: Encode config write in tests https://review.openstack.org/321927 | 21:46 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/nodepool: Python 3 fixes: dict.iteritems https://review.openstack.org/321928 | 21:46 |
| *** gordc has quit IRC | 21:46 | |
| *** esker has quit IRC | 21:47 | |
| *** tlian has quit IRC | 21:47 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: support static workers https://review.openstack.org/321569 | 21:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: some ansible fixes https://review.openstack.org/322331 | 21:49 |
| *** jamesmcarthur has quit IRC | 21:50 | |
| *** rbradfor is now known as rbradf_not_found | 21:51 | |
| *** amrith is now known as _amrith_ | 21:53 | |
| *** johnny___ has quit IRC | 21:53 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: handle JJB with no macros https://review.openstack.org/322332 | 21:54 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!