| *** haaj_ has quit IRC | 00:00 | |
| *** gouthamr has joined #openstack-infra | 00:01 | |
| *** amitgandhinz has joined #openstack-infra | 00:02 | |
| *** gouthamr_ has quit IRC | 00:03 | |
| *** sflanigan has joined #openstack-infra | 00:03 | |
| *** amitgandhinz has quit IRC | 00:06 | |
| *** ffrank has quit IRC | 00:06 | |
| *** ffrank has joined #openstack-infra | 00:07 | |
| *** thorst has joined #openstack-infra | 00:15 | |
| *** thorst has quit IRC | 00:20 | |
| *** thorst has joined #openstack-infra | 00:24 | |
| *** thorst has quit IRC | 00:28 | |
| *** Nakato has quit IRC | 00:30 | |
| *** Nakato has joined #openstack-infra | 00:31 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Enable pipelining for ansible-playbook https://review.openstack.org/331499 | 00:37 |
|---|---|---|
| *** sdake has joined #openstack-infra | 00:43 | |
| *** lvdongbing has joined #openstack-infra | 00:43 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Enable pipelining for ansible-playbook https://review.openstack.org/331499 | 00:44 |
| *** thorst has joined #openstack-infra | 00:45 | |
| *** thorst has quit IRC | 00:47 | |
| *** thorst has joined #openstack-infra | 00:48 | |
| *** x00350071__ is now known as xiangxinyong | 00:48 | |
| *** nwkarsten has quit IRC | 00:52 | |
| *** nwkarsten has joined #openstack-infra | 00:52 | |
| *** roxanaghe has joined #openstack-infra | 00:54 | |
| *** thorst has quit IRC | 00:56 | |
| *** nwkarsten has quit IRC | 00:57 | |
| *** roxanaghe has quit IRC | 00:59 | |
| *** yuanying has quit IRC | 01:00 | |
| *** amitgandhinz has joined #openstack-infra | 01:02 | |
| *** amitgandhinz has quit IRC | 01:07 | |
| *** zhurong has joined #openstack-infra | 01:10 | |
| *** bhavik has joined #openstack-infra | 01:23 | |
| *** tphummel has quit IRC | 01:24 | |
| *** kushal has quit IRC | 01:26 | |
| *** ffrank has quit IRC | 01:27 | |
| *** PollyZ has joined #openstack-infra | 01:29 | |
| *** sdake has quit IRC | 01:29 | |
| *** ajmiller_ has quit IRC | 01:29 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: Revert "No-op Change to force a re-publish" https://review.openstack.org/331010 | 01:35 |
| *** mriedem has quit IRC | 01:37 | |
| ianw | jhesketh: do you know about our ingestion of logs into logstash. istm breaking devstack log files up by line is not really very useful | 01:44 |
| jhesketh | ianw: sorry, don't know much about that area | 01:45 |
| jhesketh | ianw: why wouldn't it be useful? | 01:45 |
| *** Na3iL has joined #openstack-infra | 01:46 | |
| *** gus_ is now known as gus | 01:46 | |
| ianw | jhesketh: mostly, you get a bunch of rather irrelevant tracing lines that really only make sense when put together. but also, it seems the chopping happens somewhat haphazardly, so there's some messages that get a bunch of stuff concatenated. i'm having a look at the scripts now | 01:47 |
| *** kushal has joined #openstack-infra | 01:49 | |
| *** banix has quit IRC | 01:52 | |
| *** thorst has joined #openstack-infra | 01:54 | |
| *** thorst has quit IRC | 02:01 | |
| *** amitgandhinz has joined #openstack-infra | 02:03 | |
| openstackgerrit | liusheng proposed openstack-infra/project-config: Disable Tempest plugins gate for Aodh in stable/liberty https://review.openstack.org/331501 | 02:04 |
| *** amitgandhinz has quit IRC | 02:08 | |
| *** zhurong has quit IRC | 02:09 | |
| *** banix has joined #openstack-infra | 02:10 | |
| *** nwkarsten has joined #openstack-infra | 02:14 | |
| *** bhavik has quit IRC | 02:14 | |
| *** nwkarsten has quit IRC | 02:19 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Publish eavesdrop to yaml2ical dir https://review.openstack.org/331502 | 02:21 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Rename the eavesdrop site to be clearer https://review.openstack.org/331503 | 02:27 |
| *** gouthamr has quit IRC | 02:27 | |
| *** yarkot1 has joined #openstack-infra | 02:30 | |
| *** robot has quit IRC | 02:32 | |
| *** zhurong has joined #openstack-infra | 02:33 | |
| *** nwkarsten has joined #openstack-infra | 02:34 | |
| openstackgerrit | liusheng proposed openstack-infra/project-config: Disable Tempest plugins gate for Aodh in stable/liberty https://review.openstack.org/331501 | 02:34 |
| *** PollyZ has quit IRC | 02:36 | |
| *** robot has joined #openstack-infra | 02:39 | |
| *** jyuso1 has joined #openstack-infra | 02:45 | |
| *** sree has joined #openstack-infra | 02:50 | |
| *** roxanaghe has joined #openstack-infra | 02:56 | |
| *** zhurong has quit IRC | 02:57 | |
| *** banix has quit IRC | 02:58 | |
| *** thorst has joined #openstack-infra | 03:00 | |
| *** PollyZ has joined #openstack-infra | 03:00 | |
| *** roxanaghe has quit IRC | 03:00 | |
| *** banix has joined #openstack-infra | 03:01 | |
| *** nwkarsten has quit IRC | 03:03 | |
| *** amitgandhinz has joined #openstack-infra | 03:04 | |
| *** zhurong has joined #openstack-infra | 03:05 | |
| *** banix has quit IRC | 03:06 | |
| *** thorst has quit IRC | 03:07 | |
| *** nwkarsten has joined #openstack-infra | 03:07 | |
| *** amitgandhinz has quit IRC | 03:08 | |
| jhesketh | #status notice static.openstack.org (which hosts logs.openstack.org) is currently migrating due to a hardware failure. It should be back up shortly. | 03:09 |
| openstackstatus | jhesketh: sending notice | 03:09 |
| ianw | ha, was just about to ask if that was down :) | 03:09 |
| *** zhurong has quit IRC | 03:10 | |
| *** PollyZ has quit IRC | 03:10 | |
| -openstackstatus- NOTICE: static.openstack.org (which hosts logs.openstack.org) is currently migrating due to a hardware failure. It should be back up shortly. | 03:10 | |
| openstackstatus | jhesketh: finished sending notice | 03:11 |
| *** ramishra has joined #openstack-infra | 03:17 | |
| *** zhurong has joined #openstack-infra | 03:17 | |
| *** rwsu has joined #openstack-infra | 03:19 | |
| *** PollyZ has joined #openstack-infra | 03:21 | |
| *** PollyZ_ has joined #openstack-infra | 03:29 | |
| *** PollyZ has quit IRC | 03:30 | |
| *** yamamot__ has joined #openstack-infra | 03:32 | |
| *** hongbin has quit IRC | 03:33 | |
| *** nwkarsten has quit IRC | 03:36 | |
| *** nwkarsten has joined #openstack-infra | 03:37 | |
| *** roxanaghe has joined #openstack-infra | 03:43 | |
| *** zhurong has quit IRC | 03:52 | |
| *** nwkarsten has quit IRC | 03:54 | |
| *** PollyZ_ has quit IRC | 03:54 | |
| *** links has joined #openstack-infra | 03:56 | |
| *** dingyichen has quit IRC | 03:57 | |
| *** PollyZ has joined #openstack-infra | 04:00 | |
| *** thorst has joined #openstack-infra | 04:05 | |
| *** amitgandhinz has joined #openstack-infra | 04:05 | |
| *** yuanying has joined #openstack-infra | 04:08 | |
| *** yamamot__ has quit IRC | 04:09 | |
| *** amitgandhinz has quit IRC | 04:09 | |
| *** shahid__ has joined #openstack-infra | 04:11 | |
| *** thorst has quit IRC | 04:12 | |
| *** jaosorior has joined #openstack-infra | 04:13 | |
| *** zhurong has joined #openstack-infra | 04:13 | |
| *** nwkarsten has joined #openstack-infra | 04:16 | |
| *** savihou has joined #openstack-infra | 04:16 | |
| zaro | jhesketh: noticed that i cannot load tarballs.o.o either. are you working on that? | 04:17 |
| jhesketh | zaro: same thing... that's on the static site which is currently down unfortunately | 04:17 |
| *** markvoelker has joined #openstack-infra | 04:18 | |
| *** markvoelker has quit IRC | 04:22 | |
| *** savihou has quit IRC | 04:24 | |
| nibalizer | jhesketh: ohai | 04:26 |
| nibalizer | need a hand? | 04:26 |
| *** nwkarsten has quit IRC | 04:26 | |
| jhesketh | nibalizer: thanks, but I don't think there is anything we can do while the host is migrated | 04:27 |
| *** nwkarsten has joined #openstack-infra | 04:27 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/os-loganalyze: Match ansible output for console https://review.openstack.org/331514 | 04:29 |
| ianw | jhesketh: ^ for when the current fire is out | 04:29 |
| *** nwkarsten has quit IRC | 04:31 | |
| *** jcoufal has quit IRC | 04:36 | |
| *** banix has joined #openstack-infra | 04:38 | |
| *** ramishra has quit IRC | 04:38 | |
| *** ramishra has joined #openstack-infra | 04:39 | |
| *** ramishra has quit IRC | 04:41 | |
| *** zhurong has quit IRC | 04:42 | |
| *** hongbin has joined #openstack-infra | 04:44 | |
| winggundamth | hi. any problem with http://logs.openstack.org ? I got the connection was reset | 04:45 |
| winggundamth | I can not ping to logs.openstack.org as well | 04:45 |
| *** hongbin has quit IRC | 04:46 | |
| *** PollyZ has quit IRC | 04:46 | |
| *** ramishra has joined #openstack-infra | 04:47 | |
| winggundamth | This is traceroute http://paste.openstack.org/show/520550/ | 04:47 |
| *** zhurong has joined #openstack-infra | 04:49 | |
| *** nwkarsten has joined #openstack-infra | 04:51 | |
| *** zhurong has quit IRC | 04:52 | |
| *** abregman has joined #openstack-infra | 04:52 | |
| *** banix has quit IRC | 04:52 | |
| ianw | winggundamth: yes, is down and migrating due to hardware failure | 04:53 |
| *** nwkarsten has quit IRC | 04:55 | |
| *** zhurong has joined #openstack-infra | 04:57 | |
| *** savihou has joined #openstack-infra | 04:59 | |
| *** javeriak has joined #openstack-infra | 05:01 | |
| *** amitgandhinz has joined #openstack-infra | 05:05 | |
| *** yamamot__ has joined #openstack-infra | 05:09 | |
| *** links has quit IRC | 05:10 | |
| *** amitgandhinz has quit IRC | 05:10 | |
| *** thorst has joined #openstack-infra | 05:10 | |
| winggundamth | ianw: do you have etc? | 05:12 |
| winggundamth | *eta? | 05:12 |
| *** nmagnezi has joined #openstack-infra | 05:13 | |
| ianw | winggundamth: i think for the next few hours jhesketh is the one who would be able to see, so not sure | 05:13 |
| winggundamth | ianw: ok no problem. thanks for update :) | 05:14 |
| winggundamth | ianw: just having problem with my code that got jenkins -1 so need to see it | 05:14 |
| jhesketh | sadly no.. it's still "in progress" | 05:16 |
| jhesketh | winggundamth: right, jobs returning POST_FAILURE have failed because it couldn't upload the logs... the job itself hasn't failed | 05:17 |
| jhesketh | I might set an alert | 05:17 |
| *** nmagnezi has quit IRC | 05:17 | |
| *** thorst has quit IRC | 05:17 | |
| jhesketh | #status alert static.openstack.org (which hosts logs.openstack.org and tarballs.openstack.org among others) is currently being rebuilt. As jobs can not upload logs they are failing with POST_FAILURE. This should be resolved soon. Please do not recheck until then. | 05:18 |
| openstackstatus | jhesketh: sending alert | 05:18 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Move element-info a standard entry-point https://review.openstack.org/330893 | 05:19 |
| *** hanchao has joined #openstack-infra | 05:19 | |
| winggundamth | jhesketh: got it. thanks for update :) | 05:20 |
| -openstackstatus- NOTICE: static.openstack.org (which hosts logs.openstack.org and tarballs.openstack.org among others) is currently being rebuilt. As jobs can not upload logs they are failing with POST_FAILURE. This should be resolved soon. Please do not recheck until then. | 05:21 | |
| *** ChanServ changes topic to "static.openstack.org (which hosts logs.openstack.org and tarballs.openstack.org among others) is currently being rebuilt. As jobs can not upload logs they are failing with POST_FAILURE. This should be resolved soon. Please do not recheck until then." | 05:21 | |
| *** zhurong has quit IRC | 05:21 | |
| *** savihou has quit IRC | 05:22 | |
| *** savihou has joined #openstack-infra | 05:24 | |
| openstackstatus | jhesketh: finished sending alert | 05:24 |
| *** links has joined #openstack-infra | 05:26 | |
| *** _nadya_ has joined #openstack-infra | 05:28 | |
| *** x00350071_ has joined #openstack-infra | 05:28 | |
| *** xiangxinyong has quit IRC | 05:31 | |
| *** nwkarsten has joined #openstack-infra | 05:34 | |
| *** zhurong has joined #openstack-infra | 05:36 | |
| *** nwkarsten has quit IRC | 05:38 | |
| *** _nadya_ has quit IRC | 05:40 | |
| *** maishsk_ has joined #openstack-infra | 05:43 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Configure its-storyboard plugin https://review.openstack.org/330925 | 05:45 |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: Get java_home from parent modules https://review.openstack.org/331523 | 05:45 |
| *** maishsk has quit IRC | 05:46 | |
| *** maishsk_ is now known as maishsk | 05:46 | |
| *** mohankumar has joined #openstack-infra | 05:49 | |
| *** sorantis has joined #openstack-infra | 05:53 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Configure its-storyboard plugin https://review.openstack.org/330925 | 05:55 |
| *** Na3iL has quit IRC | 05:55 | |
| *** Na3iL has joined #openstack-infra | 05:55 | |
| *** roxanaghe has quit IRC | 06:00 | |
| *** roxanaghe has joined #openstack-infra | 06:01 | |
| *** x00350071_ is now known as xiangxinyong | 06:02 | |
| *** ociuhandu has joined #openstack-infra | 06:05 | |
| *** amitgandhinz has joined #openstack-infra | 06:06 | |
| *** roxanaghe has quit IRC | 06:06 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config: Logstash matching for anisble console output https://review.openstack.org/331526 | 06:10 |
| *** amitgandhinz has quit IRC | 06:11 | |
| *** ramishra has quit IRC | 06:11 | |
| *** zhurong has quit IRC | 06:12 | |
| *** rcernin has joined #openstack-infra | 06:13 | |
| Jeffrey4l_ | http://tarballs.openstack.org/ this site is down? | 06:16 |
| winggundamth | Jeffrey4l_: yes. Please see irc topic :) | 06:16 |
| *** _nadya_ has joined #openstack-infra | 06:16 | |
| *** linuxmaniac_ has quit IRC | 06:16 | |
| *** thorst has joined #openstack-infra | 06:16 | |
| Jeffrey4l_ | Got. thanks. | 06:17 |
| *** _nadya_ has quit IRC | 06:17 | |
| *** zhurong has joined #openstack-infra | 06:17 | |
| *** markvoelker has joined #openstack-infra | 06:18 | |
| *** linuxmaniac has joined #openstack-infra | 06:20 | |
| *** thorst has quit IRC | 06:22 | |
| *** ociuhandu has quit IRC | 06:22 | |
| *** zhurong has quit IRC | 06:22 | |
| *** markvoelker has quit IRC | 06:23 | |
| *** pcaruana has joined #openstack-infra | 06:24 | |
| *** sorantis has quit IRC | 06:26 | |
| *** e0ne has joined #openstack-infra | 06:27 | |
| *** andreas_s has joined #openstack-infra | 06:27 | |
| *** ramishra has joined #openstack-infra | 06:28 | |
| *** AJaeger_ has joined #openstack-infra | 06:32 | |
| *** garyk has joined #openstack-infra | 06:32 | |
| *** Wei_Liu has quit IRC | 06:35 | |
| *** AJaeger has quit IRC | 06:35 | |
| *** chem has joined #openstack-infra | 06:38 | |
| *** zhurong has joined #openstack-infra | 06:44 | |
| *** isaacb has joined #openstack-infra | 06:45 | |
| *** ajmiller has joined #openstack-infra | 06:47 | |
| *** zhurong has quit IRC | 06:48 | |
| *** ihrachys has joined #openstack-infra | 06:49 | |
| *** _nadya_ has joined #openstack-infra | 06:49 | |
| *** zhurong has joined #openstack-infra | 06:49 | |
| *** _nadya_ has quit IRC | 06:51 | |
| *** zhurong has quit IRC | 06:52 | |
| *** zhurong has joined #openstack-infra | 06:53 | |
| *** Na3iL has quit IRC | 06:56 | |
| *** flepied has joined #openstack-infra | 06:57 | |
| *** zhurong has quit IRC | 06:57 | |
| *** sorantis has joined #openstack-infra | 06:59 | |
| *** coreyob_ has quit IRC | 06:59 | |
| *** flepied1 has quit IRC | 07:00 | |
| *** roxanaghe has joined #openstack-infra | 07:03 | |
| AJaeger_ | odyssey4me: please see topic - rechecking right now will not help | 07:03 |
| *** pgadiya has joined #openstack-infra | 07:05 | |
| *** zhurong has joined #openstack-infra | 07:05 | |
| *** ajmiller has quit IRC | 07:06 | |
| *** amitgandhinz has joined #openstack-infra | 07:07 | |
| *** roxanaghe has quit IRC | 07:07 | |
| *** hanchao has quit IRC | 07:07 | |
| *** links has quit IRC | 07:08 | |
| *** sorantis has quit IRC | 07:10 | |
| *** matrohon has joined #openstack-infra | 07:10 | |
| *** sorantis has joined #openstack-infra | 07:11 | |
| *** jtomasek has joined #openstack-infra | 07:11 | |
| *** javeriak has quit IRC | 07:11 | |
| *** amitgandhinz has quit IRC | 07:11 | |
| *** dtardivel has joined #openstack-infra | 07:14 | |
| *** Jeffrey4l_ has quit IRC | 07:16 | |
| *** arxcruz has joined #openstack-infra | 07:16 | |
| *** jed56 has joined #openstack-infra | 07:18 | |
| *** _nadya_ has joined #openstack-infra | 07:18 | |
| *** vincentll has joined #openstack-infra | 07:19 | |
| *** thorst has joined #openstack-infra | 07:19 | |
| *** _nadya_ has quit IRC | 07:19 | |
| *** links has joined #openstack-infra | 07:21 | |
| *** Na3iL has joined #openstack-infra | 07:23 | |
| *** thorst has quit IRC | 07:26 | |
| *** kushal has quit IRC | 07:27 | |
| *** jlanoux has joined #openstack-infra | 07:28 | |
| *** ifarkas has joined #openstack-infra | 07:30 | |
| openstackgerrit | Hisashi Osanai proposed openstack-infra/project-config: Fix Monasca Analytics python27 job https://review.openstack.org/331535 | 07:30 |
| *** mikelk has joined #openstack-infra | 07:30 | |
| *** samuelBartel has joined #openstack-infra | 07:30 | |
| *** rkukura has quit IRC | 07:30 | |
| *** amoralej|off is now known as amoralej | 07:31 | |
| *** abregman has quit IRC | 07:31 | |
| *** e0ne has quit IRC | 07:32 | |
| *** ociuhandu has joined #openstack-infra | 07:33 | |
| *** AJaeger_ is now known as AJaeger | 07:33 | |
| *** sree has quit IRC | 07:33 | |
| *** abregman has joined #openstack-infra | 07:34 | |
| *** rkukura has joined #openstack-infra | 07:35 | |
| *** sree has joined #openstack-infra | 07:37 | |
| *** zeih has joined #openstack-infra | 07:41 | |
| *** jaosorior is now known as jaosorior_lunch | 07:41 | |
| *** Sukhdev has quit IRC | 07:41 | |
| *** javeriak has joined #openstack-infra | 07:42 | |
| *** HeOS has joined #openstack-infra | 07:42 | |
| *** tmorin has joined #openstack-infra | 07:42 | |
| *** Jeffrey4l has joined #openstack-infra | 07:42 | |
| *** yaume has joined #openstack-infra | 07:45 | |
| *** flip214 has quit IRC | 07:46 | |
| *** lezbar has joined #openstack-infra | 07:47 | |
| *** e0ne has joined #openstack-infra | 07:48 | |
| *** yaume_ has joined #openstack-infra | 07:48 | |
| *** hosanai has joined #openstack-infra | 07:48 | |
| *** yaume has quit IRC | 07:50 | |
| *** zeih has quit IRC | 07:51 | |
| *** sarob has quit IRC | 07:52 | |
| hosanai | hello, andreas. i have a question about patch 331535. i checked bindep but it only works for packages which provide as OS packages. but Spark doen't have that package. So I try to install devstack env for python27 | 07:53 |
| *** sarob has joined #openstack-infra | 07:53 | |
| odyssey4me | @AJaeger yeah, I saw the results come in and realised that there's a problem - the rechecks were done from my phone before I got onto IRC :/ | 07:53 |
| *** Ravikiran_K has joined #openstack-infra | 07:54 | |
| odyssey4me | AJaeger can I help in any way? | 07:54 |
| *** jpich has joined #openstack-infra | 07:54 | |
| *** savihou has quit IRC | 07:56 | |
| AJaeger | odyssey4me: I doubt that there's anything you can do - jhesketh would know more | 07:57 |
| AJaeger | hosanai: why would a devstack environment help in that case? It also needs OS packages, doesn't it? | 07:57 |
| *** ociuhandu has quit IRC | 07:58 | |
| AJaeger | hosanai: with the log server down, I cannot check logs right now, would love to see the failures... | 07:59 |
| jhesketh | not much to do at the moment.. it's still rsyncing across apparently | 07:59 |
| hosanai | AJaeger: in the devstack plugin, the package will be downloaded from Spark site | 07:59 |
| hosanai | AJaeger: thanks for your review & help | 07:59 |
| *** zzzeek_ has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** ramishra has quit IRC | 08:01 | |
| AJaeger | hosanai: I see. | 08:01 |
| *** ramishra has joined #openstack-infra | 08:02 | |
| *** ramishra has quit IRC | 08:02 | |
| odyssey4me | jhesketh what went wrong? | 08:03 |
| jhesketh | odyssey4me: hardware failure on the host | 08:03 |
| odyssey4me | Ah, I see an earlier notice regarding hardware failure. | 08:03 |
| jhesketh | it's being migrated | 08:03 |
| jhesketh | right, rebuilt probably wasn't the best term to put in the status update, but that was the state when I wrote it | 08:04 |
| *** bauzas has joined #openstack-infra | 08:05 | |
| AJaeger | hosanai: python27 is unit tests. Do you really need spark for *unit* tests? | 08:05 |
| *** sorantis has quit IRC | 08:05 | |
| *** Nikolay_St has joined #openstack-infra | 08:05 | |
| *** samP has joined #openstack-infra | 08:07 | |
| *** Fdaisuke has joined #openstack-infra | 08:07 | |
| ihrachys | AJaeger: gate broken completely? any bug to track? | 08:07 |
| *** amitgandhinz has joined #openstack-infra | 08:08 | |
| aspiers | meetbot just became unresponsive on #openstack-meeting, any ideas? | 08:08 |
| hosanai | AJaeger: yeah, Monasca-Analytics uses pyspark (Spark i/f for python) so i would like to have it. | 08:08 |
| *** ramishra has joined #openstack-infra | 08:08 | |
| AJaeger | ihrachys: it's post jobs, not aware of a bug. | 08:08 |
| aspiers | meetbot worked for #startmeeting but then nothing | 08:08 |
| aspiers | AJaeger: don't suppose you have ideas why? | 08:09 |
| ihrachys | AJaeger: I now spotted a notice is in all topics. I guess that's smth to track for projects. | 08:09 |
| AJaeger | aspiers, could be related to #topic? jhesketh? | 08:09 |
| aspiers | ahh | 08:09 |
| jhesketh | no meetbot should be unrelated... I'll take a look | 08:09 |
| AJaeger | thanks, jhesketh ! | 08:09 |
| *** samuelBartel has quit IRC | 08:09 | |
| *** abregman has quit IRC | 08:10 | |
| aspiers | jhesketh: thanks a lot! | 08:10 |
| AJaeger | hosanai: sorry, don't have time to discuss this further with you - but this looks really odd. | 08:10 |
| *** amitgandhinz has quit IRC | 08:12 | |
| AJaeger | hosanai: I suggest you ask on #openstack-qa on how to setup your tests in the best way or come back here during US time and ask for help. | 08:12 |
| odyssey4me | jhesketh when you have a moment I'd appreciate a review of https://review.openstack.org/#/q/status:open+project:openstack-infra/project-config+branch:master+topic:openstack-ansible please | 08:12 |
| hosanai | AJaeger: thanks for the advice. I will do it. | 08:13 |
| *** rossella_s has joined #openstack-infra | 08:13 | |
| rakhmerov | yep, seems like Jenkins is down | 08:13 |
| rakhmerov | oops, wrong window.. | 08:14 |
| AJaeger | rakhmerov: did you see topic? | 08:14 |
| rakhmerov | yes, wrong window ) | 08:14 |
| AJaeger | rakhmerov: and jenkins is buried for good - see http://lists.openstack.org/pipermail/openstack-dev/2016-June/097584.html | 08:14 |
| rakhmerov | sorry | 08:14 |
| *** nmagnezi has joined #openstack-infra | 08:15 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: aodh: disable tempest plugin tests in Liberty https://review.openstack.org/331550 | 08:15 |
| rakhmerov | AJaeger: didn't see yet, reading.. | 08:15 |
| odyssey4me | hosanai AJaeger commented on that review for a possible alternative | 08:17 |
| *** markvoelker has joined #openstack-infra | 08:19 | |
| *** masayukig has quit IRC | 08:20 | |
| *** lucas-afk is now known as lucasagomes | 08:20 | |
| *** _nadya_ has joined #openstack-infra | 08:22 | |
| *** abregman has joined #openstack-infra | 08:22 | |
| jhesketh | hmm, meetbot is still logging the channels... I don't really want to restart it as we'll lose history while it reconnects | 08:23 |
| hosanai | odyssey4me: thanks for your review. I checked bindep but it doesn't help :( | 08:23 |
| *** markvoelker has quit IRC | 08:24 | |
| *** rossella_s has quit IRC | 08:24 | |
| *** rossella_s has joined #openstack-infra | 08:25 | |
| *** thorst has joined #openstack-infra | 08:26 | |
| odyssey4me | hosanai it could, if you install the package that provides pip - for example 'python-pip' on ubuntu | 08:28 |
| odyssey4me | hosanai *but* afaik all images provided by infra already have pip installed on them | 08:29 |
| AJaeger | odyssey4me: hosanai means: No package on pypi to install with pip. | 08:29 |
| odyssey4me | AJaeger ah | 08:29 |
| odyssey4me | hosanai which is the package that needs installing? | 08:30 |
| AJaeger | odyssey4me: pyspark | 08:30 |
| hosanai | odyssey4me, AJaeger: yep. | 08:30 |
| *** masayukig has joined #openstack-infra | 08:30 | |
| *** masayukig has quit IRC | 08:30 | |
| *** yuanying has quit IRC | 08:31 | |
| *** thorst has quit IRC | 08:31 | |
| odyssey4me | hosanai ok, so why not include 'git+git://git.apache.org/spark.git@master#egg=pyspark' in your test-requirements.txt? | 08:32 |
| *** eezhova has joined #openstack-infra | 08:32 | |
| *** ramishra has quit IRC | 08:33 | |
| *** permalac has joined #openstack-infra | 08:33 | |
| odyssey4me | hosanai also, I'd suggest discussing publishing the package to pypi with the dev team for it as that is a better long term solution | 08:34 |
| *** hashar has joined #openstack-infra | 08:38 | |
| *** ramishra has joined #openstack-infra | 08:39 | |
| jd__ | the world is broken | 08:39 |
| AJaeger | jd__: did you see the topic? | 08:39 |
| AJaeger | jd__: or even worse? | 08:39 |
| jd__ | nah I was checking the backlog indeed :] | 08:39 |
| *** ramishra has quit IRC | 08:39 | |
| odyssey4me | happy monday? ;) | 08:40 |
| jd__ | hehe | 08:40 |
| * jd__ goes back to bed | 08:40 | |
| *** Hal has joined #openstack-infra | 08:42 | |
| *** derekh has joined #openstack-infra | 08:42 | |
| *** Hal is now known as Guest32736 | 08:42 | |
| *** hashar is now known as hasharAway | 08:43 | |
| *** ajmiller has joined #openstack-infra | 08:43 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Remove stray ";" from purge env command https://review.openstack.org/328933 | 08:44 |
| hosanai | odyssey4me: thanks for the info. I will use that approach (now i'm checking the behavior) | 08:46 |
| *** ociuhandu has joined #openstack-infra | 08:46 | |
| *** ramishra has joined #openstack-infra | 08:47 | |
| *** sorantis has joined #openstack-infra | 08:51 | |
| *** dtantsur|afk is now known as dtantsur | 08:52 | |
| hosanai | odyssey4me: it's not working... there is no setup.py for pyspark... | 08:55 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: The meetbot is openstack https://review.openstack.org/331563 | 08:57 |
| jhesketh | aspiers: so looking at ChanServ's output, open_stack does have access, but not openstack. ^ should fix it, but I'm not sure why it is the way it is | 08:58 |
| jhesketh | so should get some more eyes on it firsts | 08:58 |
| jhesketh | odyssey4me: I'll take a look at those reviews you sent me when static.o.o has recovered | 08:58 |
| *** samP has quit IRC | 08:58 | |
| aspiers | jhesketh: thanks a lot! weird that it worked before | 08:59 |
| aspiers | jhesketh: maybe someone changed manually and forgot to update channels.yaml | 09:00 |
| AJaeger | jhesketh: that change is strange - this always worked... Did we change something recently? | 09:00 |
| *** roxanaghe has joined #openstack-infra | 09:00 | |
| AJaeger | aspiers: there shouldn't be manual changes ;) | 09:00 |
| jhesketh | AJaeger: hence the uncertainty in the commit message ;-) | 09:00 |
| aspiers | AJaeger: of course not | 09:01 |
| *** kdas__ has joined #openstack-infra | 09:01 | |
| jhesketh | I guess it's possible openstack was added to the operators manually and now its gone | 09:01 |
| *** acoles_ is now known as acoles | 09:01 | |
| jhesketh | but why either of those things happened I'm not sure | 09:01 |
| AJaeger | jhesketh: yeah - let's see whether any infra-root can shed some light on the meetbot situation | 09:01 |
| jhesketh | yep.. I suspect that's the right fix, but I like knowing why first | 09:01 |
| AJaeger | jhesketh: do we need accessbot for a channel to be part of meetbot? | 09:04 |
| *** markusry has joined #openstack-infra | 09:05 | |
| jhesketh | AJaeger: I'm not sure what you mean sorry? | 09:05 |
| *** kdas__ is now known as kushal | 09:05 | |
| *** roxanaghe has quit IRC | 09:05 | |
| *** kushal has quit IRC | 09:05 | |
| *** kushal has joined #openstack-infra | 09:05 | |
| AJaeger | I mean: statusbot - I ask since #openstack-charms is part of statusbot in system-config but the project-config change has not merged | 09:05 |
| AJaeger | So, this looks unrelated to meetbot, thus ignore me ;) | 09:06 |
| odyssey4me | thanks jhesketh | 09:06 |
| jhesketh | AJaeger: statusbot needs access to post alerts, and meetbot needs access as it also does the logging to eavesdrop | 09:08 |
| *** amitgandhinz has joined #openstack-infra | 09:09 | |
| AJaeger | jhesketh: so, should https://review.openstack.org/329159 get reverted? | 09:09 |
| *** masayukig has joined #openstack-infra | 09:09 | |
| *** masayukig has quit IRC | 09:09 | |
| AJaeger | jhesketh: see https://review.openstack.org/#/c/329129/ | 09:09 |
| derekh | AJaeger: can you remove your -2 from https://review.openstack.org/#/c/330514/ , the patch doesn't switch anything, the idea will be to run both clouds for a time until rh2 is proved to be working | 09:09 |
| *** masayukig has joined #openstack-infra | 09:09 | |
| AJaeger | derekh: I don't have a -2, just a -W - but can remove that. I was waiting for you to say "we're ready to merge this" ;) | 09:10 |
| AJaeger | derekh: done | 09:10 |
| *** samuelBartel has joined #openstack-infra | 09:11 | |
| AJaeger | derekh: we discussed last week on irc with EmilienM - and couldn't get in touch with you... | 09:11 |
| derekh | AJaeger: ack, thanks, it should be good to go, the follow on patch adds an experimental job to test out the new cloud https://review.openstack.org/#/c/330515/3 | 09:11 |
| AJaeger | jhesketh: I'll revert 329159 now... | 09:11 |
| derekh | AJaeger: yup, I looked through the logs to see what happened, thanks again | 09:12 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/system-config: Revert "Add #openstack-charms to statusbot" https://review.openstack.org/331566 | 09:12 |
| *** amitgandhinz has quit IRC | 09:13 | |
| jhesketh | AJaeger: WHY REVERT IT? | 09:14 |
| jhesketh | woah, caps, sorry | 09:14 |
| AJaeger | jhesketh: what else do you propose? WE cannot merge https://review.openstack.org/#/c/329129/ - since we're limited to 120 gerritbot channels... | 09:14 |
| *** masayukig has quit IRC | 09:15 | |
| AJaeger | Not sure whether accessbot has a limit as well, so probably this could be split up... | 09:15 |
| *** masayukig has joined #openstack-infra | 09:15 | |
| jhesketh | oh, now I see the problem | 09:15 |
| AJaeger | Should we just add charms to accessbot instead? | 09:16 |
| *** masayukig has quit IRC | 09:16 | |
| *** samuelBartel has quit IRC | 09:17 | |
| jhesketh | AJaeger: is the accessbot actually the limit? doesn't it just set the permissions on the channels, it doesn't hang around | 09:17 |
| AJaeger | jhesketh: gerritbot is the limit, that's why I think splitting the change up should do the trick - instead of the revert | 09:17 |
| jhesketh | AJaeger: right, yes, agreed | 09:17 |
| jhesketh | AJaeger: we should look at how close to the limit our bots are and figure out if we can remove cruft or if we need more bots | 09:18 |
| jhesketh | FYI the static node has been rebuilt but is currently doing a fsck... hopefully not long now | 09:18 |
| *** masayukig has joined #openstack-infra | 09:19 | |
| eliqiao | hi, I see that nova 's CI get "POST_FAILURE" on my patch, https://review.openstack.org/#/c/320897/ | 09:19 |
| *** savihou has joined #openstack-infra | 09:19 | |
| AJaeger | jhesketh: we already removed two channels to be at the limit - and there was some discussion to raise limit. But I guess nobody took action on that | 09:19 |
| AJaeger | eliqiao: see topic | 09:19 |
| *** electrofelix has joined #openstack-infra | 09:20 | |
| eliqiao | AJaeger: oh yep, I see. thanks :) | 09:20 |
| *** infra-red has joined #openstack-infra | 09:20 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add #openstack-charms and associated projects https://review.openstack.org/329129 | 09:21 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Add #openstack-charms to accessbot https://review.openstack.org/331572 | 09:21 |
| AJaeger | jhesketh: split up the change ^ | 09:21 |
| jhesketh | AJaeger: I must have missed that discussion | 09:22 |
| jhesketh | the split lgtm | 09:22 |
| AJaeger | jhesketh: that was a brief one... | 09:23 |
| AJaeger | bbl | 09:23 |
| *** infra-red has quit IRC | 09:24 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Update OSA job skip for all jobs except docs/releasenotes https://review.openstack.org/331574 | 09:24 |
| *** masayukig has quit IRC | 09:24 | |
| *** jaosorior_lunch is now known as jaosorior | 09:27 | |
| *** thorst has joined #openstack-infra | 09:29 | |
| *** ajmiller has quit IRC | 09:30 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:35 | |
| *** ramishra has quit IRC | 09:35 | |
| *** ramishra has joined #openstack-infra | 09:36 | |
| hosanai | AJaeger, odyssey4me: I asked my issue in #openstack-qa now. just FYI. I already asked this here on last Friday (US time). The advise was use monasca-analytics devstack plugin. | 09:36 |
| *** thorst has quit IRC | 09:37 | |
| *** aderyugin has joined #openstack-infra | 09:37 | |
| *** ramishra has quit IRC | 09:38 | |
| *** guardianJ has joined #openstack-infra | 09:39 | |
| *** haaj has joined #openstack-infra | 09:42 | |
| *** maishsk_ has joined #openstack-infra | 09:43 | |
| *** ramishra has joined #openstack-infra | 09:43 | |
| *** masayukig has joined #openstack-infra | 09:43 | |
| *** lvdongbing has quit IRC | 09:44 | |
| *** maishsk has quit IRC | 09:46 | |
| *** maishsk_ is now known as maishsk | 09:46 | |
| bkero | Hmm, anyone else having trouble accessing specs.o.o? | 09:46 |
| bkero | [bkero@Whitefall ~]$ nc -v -z specs.openstack.org 80 | 09:46 |
| bkero | nc: connect to specs.openstack.org port 80 (tcp) failed: Connection refused | 09:46 |
| bkero | nc: connect to specs.openstack.org port 80 (tcp) failed: Network is unreachable | 09:46 |
| aspiers | bkero: I think that's $TOPIC | 09:47 |
| jhesketh | bkero: yeah, that's also on static.o.o sorry | 09:47 |
| bkero | ah, 'among others' | 09:47 |
| bkero | got it | 09:47 |
| *** yuanying has joined #openstack-infra | 09:48 | |
| *** guardianJ has left #openstack-infra | 09:51 | |
| *** javeriak has quit IRC | 09:51 | |
| *** sambetts|afk is now known as sambetts | 09:55 | |
| *** milan has joined #openstack-infra | 09:57 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Add infrstucture scripts to prepare rh2 https://review.openstack.org/295243 | 09:58 |
| *** kzaitsev_mb has quit IRC | 09:59 | |
| *** masayukig has quit IRC | 10:00 | |
| *** masayukig has joined #openstack-infra | 10:00 | |
| *** javeriak has joined #openstack-infra | 10:00 | |
| *** eezhova has quit IRC | 10:01 | |
| *** sree has quit IRC | 10:02 | |
| *** abregman has quit IRC | 10:02 | |
| *** sree has joined #openstack-infra | 10:03 | |
| *** ociuhandu has quit IRC | 10:04 | |
| *** maishsk has quit IRC | 10:04 | |
| *** maishsk has joined #openstack-infra | 10:04 | |
| *** sorantis has quit IRC | 10:05 | |
| *** maishsk_ has joined #openstack-infra | 10:05 | |
| *** zhurong has quit IRC | 10:05 | |
| *** haaj_ has joined #openstack-infra | 10:06 | |
| *** masayukig has quit IRC | 10:06 | |
| *** haaj has quit IRC | 10:07 | |
| *** jokke__ is now known as jokke_ | 10:07 | |
| *** sree has quit IRC | 10:08 | |
| *** eezhova has joined #openstack-infra | 10:08 | |
| dteselkin | Hi, are there any issues with logs.openstack.org? | 10:08 |
| *** masayukig has joined #openstack-infra | 10:08 | |
| AJaeger | dteselkin: see topic | 10:09 |
| *** maishsk has quit IRC | 10:09 | |
| *** maishsk_ is now known as maishsk | 10:09 | |
| *** amitgandhinz has joined #openstack-infra | 10:09 | |
| *** hosanai has quit IRC | 10:12 | |
| dteselkin | Thanks :) | 10:13 |
| *** amitgandhinz has quit IRC | 10:14 | |
| *** dizquierdo has joined #openstack-infra | 10:15 | |
| *** flip214 has joined #openstack-infra | 10:16 | |
| *** yuanying has quit IRC | 10:17 | |
| *** abregman has joined #openstack-infra | 10:18 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Check git tag matches xstatic version https://review.openstack.org/330321 | 10:19 |
| jhesketh | AJaeger: addressed your feedback ^ | 10:19 |
| *** markvoelker has joined #openstack-infra | 10:20 | |
| *** eyalb has joined #openstack-infra | 10:22 | |
| *** masayukig has quit IRC | 10:22 | |
| *** haaj has joined #openstack-infra | 10:22 | |
| *** masayukig has joined #openstack-infra | 10:23 | |
| *** pblaho has joined #openstack-infra | 10:24 | |
| *** markvoelker has quit IRC | 10:24 | |
| *** haaj_ has quit IRC | 10:27 | |
| jhesketh | okay static.o.o is online... I'm going to check jobs are uploading before removing the status | 10:27 |
| *** ams__ has joined #openstack-infra | 10:31 | |
| *** thorst has joined #openstack-infra | 10:34 | |
| jhesketh | #status ok static.openstack.org is back up. If you have POST_FAILURE and are missing logs from your CI jobs, please leave a 'recheck'. | 10:35 |
| openstackstatus | jhesketh: sending ok | 10:35 |
| yamamot__ | jhesketh: thank you | 10:36 |
| *** rcernin is now known as rcernin|lunch | 10:37 | |
| *** _oanson has joined #openstack-infra | 10:37 | |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 10:37 | |
| -openstackstatus- NOTICE: static.openstack.org is back up. If you have POST_FAILURE and are missing logs from your CI jobs, please leave a 'recheck'. | 10:37 | |
| *** hferenc has joined #openstack-infra | 10:39 | |
| openstackstatus | jhesketh: finished sending ok | 10:40 |
| *** thorst has quit IRC | 10:42 | |
| hferenc | hi all! | 10:42 |
| hferenc | ianw: can we talk about this change: https://review.openstack.org/#/c/329832/ ? | 10:43 |
| AJaeger | thanks, jhesketh | 10:43 |
| *** yuanying has joined #openstack-infra | 10:44 | |
| isaacb | AJaeger: Hi, Can I ask you please to look in https://review.openstack.org/#/c/322509/ | 10:45 |
| *** yuanying has quit IRC | 10:45 | |
| ianw | hferenc: sure | 10:47 |
| *** reduxio has quit IRC | 10:47 | |
| *** mhickey has joined #openstack-infra | 10:47 | |
| AJaeger | isaacb: we delete that file in https://review.openstack.org/#/c/331494 ... | 10:48 |
| AJaeger | isaacb: if https://review.openstack.org/#/c/331494 gets merged, no need to take yours | 10:48 |
| hferenc | ianw: ok, so i totally agree with you, all the things should be under the openstack umbrella - i was just not sure how to do it without docker | 10:50 |
| snarwade | how to enable nightly build in jenkins | 10:51 |
| *** yuanying has joined #openstack-infra | 10:51 | |
| hferenc | my biggest problem is that i don't now how to test this job | 10:52 |
| AJaeger | snarwade: jenkins is gone for good - see http://lists.openstack.org/pipermail/openstack-dev/2016-June/097584.html | 10:52 |
| *** eezhova has quit IRC | 10:52 | |
| *** tosky has joined #openstack-infra | 10:52 | |
| ianw | hferenc: my first thought would be to checkin that script that makes the images to nova, maybe tempest, and then you can create this periodic job to run that script and produce the output images | 10:53 |
| isaacb | AJaeger: see comments in https://review.openstack.org/#/c/325339/ | 10:53 |
| AJaeger | snarwade: project-config/zuul/layout.yaml is the file to configure - use the periodic pipeline | 10:53 |
| ianw | hferenc: that script should run basically unmodified on a ubuntu-trusty node and produce the right thing | 10:53 |
| AJaeger | isaacb: please sort it out with pabelanger. | 10:54 |
| isaacb | AJaeger: Don't you think we better wait with this removal. This will break some downstrream CI systems | 10:54 |
| AJaeger | isaacb: shouldn't those be using project-config-example instead? | 10:54 |
| snarwade | AJaeger, that doesnt have nightly build | 10:55 |
| *** yuanying has quit IRC | 10:55 | |
| ianw | hferenc: to be clear, the periodic job would use zuul-cloner to clone nova/tempest (wherever the script lives) and then simply runs it | 10:55 |
| AJaeger | snarwade: periodic jobs are nightly builds basically | 10:55 |
| *** eezhova has joined #openstack-infra | 10:55 | |
| snarwade | AJaeger, thanks | 10:55 |
| *** eyalb has quit IRC | 10:57 | |
| AJaeger | snarwade: what do you want to build? | 10:57 |
| snarwade | AJaeger, I have a project that needs to have nightly build | 10:58 |
| *** Ravikiran_K has quit IRC | 10:58 | |
| AJaeger | snarwade: which project and for what purpose? | 10:59 |
| *** yuanying has joined #openstack-infra | 10:59 | |
| *** amotoki has joined #openstack-infra | 10:59 | |
| snarwade | AJaeger, even I donr know | 11:00 |
| hferenc | ianw: so the nicest way to do this would be to create the appropriate script in nova/tempest, then create a periodic job which runs that script (directly) on the ubuntu-trusty node | 11:00 |
| snarwade | dont know* | 11:00 |
| *** yuanying has quit IRC | 11:01 | |
| ianw | hferenc: yep! that sounds right. probably tempest i'd say, seems like they'd the the most logical reviewers | 11:01 |
| *** roxanaghe has joined #openstack-infra | 11:02 | |
| hferenc | ianw: ok, it'll be record long chain of "depends-on:" tags, but it is more elegant this way :) | 11:04 |
| *** kzaitsev_mb has joined #openstack-infra | 11:05 | |
| *** _nadya_ has quit IRC | 11:05 | |
| *** roxanaghe has quit IRC | 11:08 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Reduce the tenants stack create timeout https://review.openstack.org/314195 | 11:08 |
| *** csatari has joined #openstack-infra | 11:10 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Stop destroying VM's that don't exist https://review.openstack.org/312863 | 11:10 |
| *** amitgandhinz has joined #openstack-infra | 11:10 | |
| *** cargonza is now known as cargonza-holiday | 11:10 | |
| *** ramishra has quit IRC | 11:11 | |
| *** ramishra has joined #openstack-infra | 11:12 | |
| *** yuanying has joined #openstack-infra | 11:13 | |
| hferenc | ianw: two last questions for now: 1.) is there an incubator for new jobs where one could test before submitting it to gerrit? 2.) where can i find some info about what configuration does a specific node uses exactly? e.g. if i specify a job with "node: ubuntu-trusty" will it be a desktop or server version without any further hacks | 11:14 |
| *** amitgandhinz has quit IRC | 11:14 | |
| *** dprince has joined #openstack-infra | 11:16 | |
| *** icey has joined #openstack-infra | 11:17 | |
| openstackgerrit | Merged openstack-infra/project-config: Add #openstack-charms to accessbot https://review.openstack.org/331572 | 11:18 |
| ianw | hferenc: 1) not really, but if the script mostly works it would be a matter of iterating to get it just right. 2) it is more-or-less a lightly customised version of the cloud-image you can download. you can actually build it (see project-config:tools/build-image.sh) but if you're working on the cloud image, you'd be basically there | 11:18 |
| *** sdague has joined #openstack-infra | 11:19 | |
| *** yuanying has quit IRC | 11:19 | |
| *** ramishra has quit IRC | 11:19 | |
| *** yuanying has joined #openstack-infra | 11:20 | |
| *** ramishra has joined #openstack-infra | 11:20 | |
| *** eezhova has quit IRC | 11:21 | |
| *** dprince has quit IRC | 11:21 | |
| *** yuanying has quit IRC | 11:22 | |
| *** daemontool has joined #openstack-infra | 11:22 | |
| *** rhallisey has joined #openstack-infra | 11:24 | |
| hferenc | ok, thank you! | 11:25 |
| *** sorantis has joined #openstack-infra | 11:25 | |
| AJaeger | hferenc: best to run the job as experimental first - that way you can trigger running it manually while fixing it... | 11:26 |
| *** yolanda has joined #openstack-infra | 11:29 | |
| *** rodrigods has quit IRC | 11:30 | |
| *** rodrigods has joined #openstack-infra | 11:30 | |
| *** lucasagomes is now known as lucas-hungry | 11:32 | |
| *** wfoster is now known as wfoster_afk | 11:32 | |
| *** weshay has joined #openstack-infra | 11:33 | |
| *** kdas__ has joined #openstack-infra | 11:33 | |
| *** kushal has quit IRC | 11:34 | |
| *** erlon has joined #openstack-infra | 11:34 | |
| *** ajmiller has joined #openstack-infra | 11:34 | |
| *** julim has joined #openstack-infra | 11:35 | |
| *** sorantis has quit IRC | 11:35 | |
| *** yuanying has joined #openstack-infra | 11:35 | |
| *** yuanying has quit IRC | 11:37 | |
| *** abregman has quit IRC | 11:38 | |
| *** yuanying has joined #openstack-infra | 11:39 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Add the ability to permalink to a comment https://review.openstack.org/289078 | 11:39 |
| *** dgonzalez has quit IRC | 11:40 | |
| *** baoli has joined #openstack-infra | 11:42 | |
| *** sree has joined #openstack-infra | 11:42 | |
| *** yuanying has quit IRC | 11:44 | |
| *** dgonzalez has joined #openstack-infra | 11:44 | |
| *** dtantsur is now known as dtantsur|bbl | 11:45 | |
| *** ccamacho is now known as ccamacho|lunch | 11:45 | |
| *** sree has quit IRC | 11:46 | |
| *** thorst has joined #openstack-infra | 11:49 | |
| *** thorst_ has joined #openstack-infra | 11:50 | |
| mkoderer___ | hi folks, all the links for the jobs on http://zuul.openstack.org/ have a "telnet" prefix | 11:50 |
| *** kdas__ is now known as kushal | 11:51 | |
| *** kushal has quit IRC | 11:51 | |
| *** kushal has joined #openstack-infra | 11:51 | |
| mkoderer___ | like gate-cinder-pep8 links to telnet://23.253.205.11:8088 | 11:51 |
| mkoderer___ | is this a bug or a feature? ;) | 11:51 |
| *** ldnunes has joined #openstack-infra | 11:51 | |
| *** abregman has joined #openstack-infra | 11:51 | |
| rcarrillocruz | it's a feature | 11:52 |
| rcarrillocruz | with jenkins gone | 11:52 |
| *** rcernin|lunch is now known as rcernin | 11:53 | |
| rcarrillocruz | now you can see the jobs progress by telneting on the link given | 11:53 |
| AJaeger | mkoderer___: http://lists.openstack.org/pipermail/openstack-dev/2016-June/097595.html | 11:53 |
| AJaeger | mkoderer___: did you telnet to that URL? Give it a try! | 11:53 |
| *** eezhova has joined #openstack-infra | 11:54 | |
| *** thorst has quit IRC | 11:54 | |
| *** tpsilva has joined #openstack-infra | 11:54 | |
| AJaeger | rcarrillocruz: seems mkoderer___ is shocked that Jenkins is gone ;) | 11:54 |
| rcarrillocruz | hah | 11:54 |
| *** raildo-afk is now known as raildo | 11:55 | |
| rcarrillocruz | we've been hacked! | 11:55 |
| AJaeger | That evil zuul hacker... | 11:55 |
| *** dprince has joined #openstack-infra | 11:56 | |
| *** ramishra has quit IRC | 11:56 | |
| mkoderer___ | rcarrillocruz: AJaeger: yeah - I really thought somebody hacked infra :) ok nice! | 11:58 |
| *** aysyd has joined #openstack-infra | 11:59 | |
| mkoderer___ | rcarrillocruz: bahh telnet doesn't work with our proxy :( | 11:59 |
| *** daemontool_ has joined #openstack-infra | 11:59 | |
| rcarrillocruz | lol | 11:59 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Stop allowing duplicate users to be added to private stories https://review.openstack.org/320895 | 11:59 |
| *** baoli_ has joined #openstack-infra | 12:00 | |
| *** daemontool has quit IRC | 12:02 | |
| *** ramishra has joined #openstack-infra | 12:02 | |
| *** baoli has quit IRC | 12:03 | |
| *** roxanaghe has joined #openstack-infra | 12:03 | |
| *** kzaitsev_mb has quit IRC | 12:03 | |
| *** aysyd has quit IRC | 12:04 | |
| *** aysyd has joined #openstack-infra | 12:04 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Remove pagination for timeline events https://review.openstack.org/327682 | 12:05 |
| *** e0ne has quit IRC | 12:06 | |
| hferenc | AJaeger: does the "check experimental" magic work even if the job is just under review? | 12:06 |
| AJaeger | hferenc: jobs always need to merge first, there'S no testing of them anyway | 12:07 |
| mordred | mkoderer___: websockets-based web version of that is coming | 12:07 |
| AJaeger | hferenc: I speak about the project-config part... | 12:07 |
| mordred | jhesketh: morning! looks like you had fun on your Monday :) | 12:07 |
| jhesketh | mordred: howdy | 12:07 |
| AJaeger | mordred: yeah, he has been busy shuffling bits and bytes around ;) | 12:08 |
| jhesketh | other than a long fsck it came up okay so no harm in the end | 12:08 |
| mordred | \o/ | 12:08 |
| mordred | jhesketh: what failed? | 12:08 |
| *** roxanaghe has quit IRC | 12:08 | |
| *** vincentll has quit IRC | 12:08 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Add TAGS_ADDED and TAGS_DELETED to event_types.ALL https://review.openstack.org/331641 | 12:08 |
| *** yuanying has joined #openstack-infra | 12:09 | |
| jhesketh | mordred: hardware failure on the host for static.o.o | 12:09 |
| jhesketh | mordred: Rackspace migrated us, but it took a long time to sync and then come back up | 12:09 |
| *** HeOS has quit IRC | 12:10 | |
| *** HeOS has joined #openstack-infra | 12:11 | |
| *** vincentll has joined #openstack-infra | 12:11 | |
| *** amitgandhinz has joined #openstack-infra | 12:11 | |
| *** coreyob has joined #openstack-infra | 12:11 | |
| *** xenogear has joined #openstack-infra | 12:12 | |
| *** nmagnezi has quit IRC | 12:12 | |
| *** yuanying has quit IRC | 12:12 | |
| *** markvoelker has joined #openstack-infra | 12:12 | |
| *** furlongm has quit IRC | 12:13 | |
| *** furlongm has joined #openstack-infra | 12:13 | |
| *** daemontool has joined #openstack-infra | 12:13 | |
| *** daemontool_ has quit IRC | 12:14 | |
| *** Liuqing has joined #openstack-infra | 12:14 | |
| *** amoralej is now known as amoralej|lunch | 12:14 | |
| *** nibalizer has quit IRC | 12:15 | |
| *** DinaBelova has quit IRC | 12:15 | |
| *** ekarlso has quit IRC | 12:15 | |
| *** shahid__ has quit IRC | 12:15 | |
| *** amitgandhinz has quit IRC | 12:15 | |
| *** links has quit IRC | 12:16 | |
| *** rodrigods has quit IRC | 12:16 | |
| *** coreycb has quit IRC | 12:16 | |
| *** spiffxp has quit IRC | 12:16 | |
| *** ccamacho|lunch has quit IRC | 12:16 | |
| *** spiffxp has joined #openstack-infra | 12:16 | |
| *** jpeeler has quit IRC | 12:16 | |
| *** rodrigods has joined #openstack-infra | 12:16 | |
| *** gmann has quit IRC | 12:17 | |
| *** lindsayk_ has quit IRC | 12:17 | |
| *** jcoufal has joined #openstack-infra | 12:17 | |
| *** flip214 has quit IRC | 12:17 | |
| *** jlvillal has quit IRC | 12:18 | |
| *** wfoster_afk is now known as wfoster | 12:18 | |
| *** coreycb has joined #openstack-infra | 12:18 | |
| *** gmann has joined #openstack-infra | 12:18 | |
| *** DinaBelova has joined #openstack-infra | 12:18 | |
| *** gouthamr has joined #openstack-infra | 12:18 | |
| *** sirushti has quit IRC | 12:18 | |
| *** bunting has quit IRC | 12:19 | |
| *** sdague has quit IRC | 12:19 | |
| *** psilvad has joined #openstack-infra | 12:19 | |
| *** flip214 has joined #openstack-infra | 12:19 | |
| *** masayukig has quit IRC | 12:19 | |
| *** bunting has joined #openstack-infra | 12:19 | |
| *** jlvillal has joined #openstack-infra | 12:19 | |
| *** sc68cal has quit IRC | 12:19 | |
| *** pradk has joined #openstack-infra | 12:19 | |
| openstackgerrit | Merged openstack-infra/storyboard: Support subscription to worklists in the API https://review.openstack.org/324493 | 12:20 |
| *** Jeffrey4l has quit IRC | 12:20 | |
| *** julim has quit IRC | 12:20 | |
| *** sirushti has joined #openstack-infra | 12:21 | |
| AJaeger | project-config reviewers, could I get another +2 on https://review.openstack.org/#/c/330725/ , please? | 12:21 |
| *** garyk has quit IRC | 12:21 | |
| *** ccamacho has joined #openstack-infra | 12:22 | |
| hferenc | AJaeger: sorry, but i don't get it | 12:22 |
| *** jaosorior has quit IRC | 12:22 | |
| *** jaosorior has joined #openstack-infra | 12:22 | |
| *** lezbar has quit IRC | 12:23 | |
| *** jpeeler has joined #openstack-infra | 12:23 | |
| *** lezbar has joined #openstack-infra | 12:23 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Only request desired timeline events https://review.openstack.org/331646 | 12:24 |
| *** sc68cal has joined #openstack-infra | 12:26 | |
| *** bswartz has quit IRC | 12:26 | |
| *** gordc has joined #openstack-infra | 12:26 | |
| *** dosaboy has quit IRC | 12:26 | |
| *** links has joined #openstack-infra | 12:26 | |
| AJaeger | hferenc: rephrase your check experimental question for me, please | 12:27 |
| *** Jeffrey4l has joined #openstack-infra | 12:27 | |
| *** kzaitsev_mb has joined #openstack-infra | 12:28 | |
| *** yuanying has joined #openstack-infra | 12:28 | |
| *** dosaboy has joined #openstack-infra | 12:28 | |
| *** shahid__ has joined #openstack-infra | 12:28 | |
| *** russellb has joined #openstack-infra | 12:29 | |
| *** rfolco has joined #openstack-infra | 12:30 | |
| *** yuanying has quit IRC | 12:31 | |
| *** mohankumar has quit IRC | 12:31 | |
| *** amotoki has quit IRC | 12:32 | |
| *** yuanying has joined #openstack-infra | 12:32 | |
| kambiz | is there a problem with openid? I can't seem to login to review.openstack.org currently. | 12:33 |
| *** mohankumar has joined #openstack-infra | 12:33 | |
| *** amotoki has joined #openstack-infra | 12:33 | |
| *** leifmadsen has joined #openstack-infra | 12:34 | |
| *** garyk has joined #openstack-infra | 12:34 | |
| jamespage | hey - are publications under docs.openstack.org/developer reserved for official projects? or can I add an openstack-charms guide under that URL? | 12:34 |
| *** rlandy has joined #openstack-infra | 12:34 | |
| AJaeger | jamespage: docs.o.o is for official projects | 12:34 |
| *** cbader has quit IRC | 12:34 | |
| kambiz | openstackgerrit: help | 12:34 |
| kambiz | openstackstatus: help | 12:34 |
| AJaeger | http://docs.openstack.org/infra/manual/creators.html#decide-status-of-your-project | 12:35 |
| *** e0ne has joined #openstack-infra | 12:35 | |
| *** cbader has joined #openstack-infra | 12:35 | |
| *** leifmadsen has quit IRC | 12:35 | |
| *** leifmadsen has joined #openstack-infra | 12:35 | |
| AJaeger | kambiz: our bots won't be able to help - and openid comes for launchpad. You're the first to ask here, no idea. | 12:35 |
| jamespage | AJaeger, yah - that's what I thought - we have a review up for official project status, but we're currently re-licensing which is taking a little longer than I anticipated... | 12:36 |
| *** e0ne has quit IRC | 12:36 | |
| hferenc | AJaeger: so if i add the script to tempest and a simple job which runs it to project-config, then i can fine tune the script (submitting a new patchset + commenting "check experimental"), but the job has to be merged to project-config first, right? | 12:37 |
| *** yamamot__ has quit IRC | 12:37 | |
| AJaeger | kambiz: yeah, looks like ubuntu-one is broken ;( perhaps jamespage can help you on that... | 12:37 |
| *** infra-red has joined #openstack-infra | 12:37 | |
| jamespage | AJaeger, I'd be delighted to | 12:37 |
| AJaeger | hferenc: correct. Once the project-config has merged, you can run check experimental on the tempest one | 12:37 |
| *** qb_ has quit IRC | 12:38 | |
| *** icey has quit IRC | 12:38 | |
| *** tkelsey has joined #openstack-infra | 12:38 | |
| AJaeger | jamespage: review.o.o does not seem to go to ubuntu-one, seem it's down ;( | 12:38 |
| AJaeger | thanks, jamespage | 12:38 |
| EmilienM | I was about to say "good morning, what's up with openid?" | 12:39 |
| *** shahid__ has quit IRC | 12:39 | |
| *** david-lyle has quit IRC | 12:39 | |
| jamespage | AJaeger, do review.o.o use login.ubuntu.com? | 12:40 |
| *** jcoufal has quit IRC | 12:40 | |
| jokke_ | anyone else having issues review.o.o timing out? | 12:41 |
| *** sree has joined #openstack-infra | 12:41 | |
| *** kgiusti has joined #openstack-infra | 12:41 | |
| jokke_ | haven't been able to reach for ~20-30 min now | 12:41 |
| AJaeger | jamespage: AFAIK: Yes | 12:42 |
| *** david-lyle has joined #openstack-infra | 12:42 | |
| *** yamamoto has joined #openstack-infra | 12:42 | |
| *** e0ne has joined #openstack-infra | 12:42 | |
| jamespage | AJaeger, ack - checking a few other things that use the same service - no reported problems with Canonical IS atm | 12:42 |
| *** javeriak has quit IRC | 12:43 | |
| AJaeger | jamespage: let's see whether any infra-root is around that knows more details than I do... | 12:43 |
| *** yuanying has quit IRC | 12:43 | |
| AJaeger | thanks for checking, jamespage | 12:43 |
| *** masayukig has joined #openstack-infra | 12:44 | |
| AJaeger | jamespage: https://login.launchpad.net/+openid is used | 12:44 |
| *** e0ne has quit IRC | 12:45 | |
| *** jaosorior has quit IRC | 12:45 | |
| *** Ravikiran_K has joined #openstack-infra | 12:45 | |
| jamespage | AJaeger, where is r.o.o hosted? | 12:45 |
| AJaeger | jamespage: at rackspace public cloud AFAIK | 12:45 |
| jamespage | AJaeger, where geographically? | 12:46 |
| *** yamamoto has quit IRC | 12:46 | |
| AJaeger | jamespage: no idea;( WE might need to wait for some other infra-roots to wake up and answer those questions... | 12:47 |
| *** iurygregory has joined #openstack-infra | 12:47 | |
| AJaeger | traceroute gives me dfw2.rackspace.net as hop before it... | 12:48 |
| jamespage | AJaeger, ok - there have been some reported trans-atlantic performance issues US -> Europe which might account for the problems... | 12:48 |
| jamespage | ok us then | 12:48 |
| *** milan has quit IRC | 12:48 | |
| zigo | I could login from here. | 12:48 |
| hferenc | AJaeger: seems like i get it after all, but... 1.) shouldn't the job's change depend on the script's change? 2.) the job might need to be moved to an other pipeline from experimental | 12:49 |
| *** M-docaedo_vector has quit IRC | 12:49 | |
| EmilienM | jamespage: San Antonio | 12:50 |
| jamespage | EmilienM, ok thanks | 12:50 |
| EmilienM | https://db-ip.com/104.130.246.91 | 12:50 |
| EmilienM | and http://puppetdb.openstack.org/node/afb0cdb2-4800-4268-af83-0b2d5f68a635 <-- for infos about the node | 12:50 |
| mordred | jamespage, EmilienM: dallas actually - that ip thing is wrong - but for network reasons there should be no real difference | 12:51 |
| mordred | I have logged out of gerrit and am logging back in | 12:51 |
| mordred | it's definitely taking its sweet time | 12:52 |
| *** jaosorior has joined #openstack-infra | 12:52 | |
| AJaeger | hferenc: the job change needs to merge. A check experimental checks out the repo and runs configured experimental jobs on it. | 12:52 |
| mordred | jamespage: oh wow. trying to log in for me I get: "Provider is not supported, or was incorrectly entered." | 12:52 |
| mordred | which is a gerrit error | 12:52 |
| mordred | but it clearly did not like login.lp.net :) | 12:52 |
| jamespage | "<axino> jamespage: well I experience high packet loss to that IP" | 12:52 |
| pabelanger | isaacb: AJaeger: I've added a comment to 331494. | 12:53 |
| AJaeger | hferenc: but the job itself will not be different, we only check the sources out. | 12:53 |
| EmilienM | mordred: using Gertty works ~fine | 12:53 |
| mordred | jamespage: \o/ for packet loss | 12:53 |
| mordred | EmilienM: yup. :) | 12:53 |
| EmilienM | mordred: but still super slow | 12:53 |
| jamespage | digging on the actual problem | 12:53 |
| *** burgerk has joined #openstack-infra | 12:53 | |
| AJaeger | thanks, pabelanger. Btw. if you review later today, please look also at https://review.openstack.org/#/c/330725/ | 12:54 |
| AJaeger | EmilienM: gertty uses https password, not ubuntu one | 12:54 |
| jamespage | ok so the current sitrep is that we're seeing poor transatlantic connectivity - hence the high packet loss | 12:54 |
| jamespage | outside of the infra hosting login.launchpad.net/login.ubuntu.com | 12:55 |
| EmilienM | AJaeger: gerrit is still super slow, and also was this week end (502 errors) | 12:55 |
| AJaeger | EmilienM: gerrit has been restarted on the weekend | 12:55 |
| mordred | #status OpenID login from review.o.o is experiencing difficulties, possibly due to transatlantic network performance issues. Things are being investigated | 12:55 |
| openstackstatus | mordred: unknown command | 12:55 |
| mordred | gah | 12:55 |
| AJaeger | EmilienM: it was fast for me this morning | 12:55 |
| mordred | #status alert OpenID login from review.o.o is experiencing difficulties, possibly due to transatlantic network performance issues. Things are being investigated | 12:55 |
| openstackstatus | mordred: sending alert | 12:55 |
| *** ramishra has quit IRC | 12:55 | |
| *** sorantis has joined #openstack-infra | 12:56 | |
| *** slaweq has quit IRC | 12:56 | |
| jamespage | I've asked Canonical IS to ping me if they get any updates - I'll relay here | 12:57 |
| jamespage | AJaeger, is there a *right* location to use for docs for as-yet not official openstack projects? | 12:58 |
| *** tlian has joined #openstack-infra | 12:58 | |
| -openstackstatus- NOTICE: OpenID login from review.o.o is experiencing difficulties, possibly due to transatlantic network performance issues. Things are being investigated | 12:58 | |
| *** ChanServ changes topic to "OpenID login from review.o.o is experiencing difficulties, possibly due to transatlantic network performance issues. Things are being investigated" | 12:58 | |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run cinderclient functional tests in Identity v3-only https://review.openstack.org/313562 | 12:58 |
| *** ekarlso has joined #openstack-infra | 12:59 | |
| *** permalac_ has joined #openstack-infra | 12:59 | |
| pabelanger | AJaeger: LGTM | 12:59 |
| *** permalac has quit IRC | 12:59 | |
| *** matt-borland has joined #openstack-infra | 12:59 | |
| *** trown|outtypewww is now known as trown | 12:59 | |
| *** links has quit IRC | 13:00 | |
| *** icey has joined #openstack-infra | 13:01 | |
| openstackstatus | mordred: finished sending alert | 13:01 |
| mordred | jamespage: most of them publish to rtfd, but no. there isn't an answer answer | 13:01 |
| mordred | jamespage: and thanks! | 13:01 |
| *** M-docaedo_vector has joined #openstack-infra | 13:01 | |
| matt-borland | good morning all! | 13:04 |
| *** roxanaghe has joined #openstack-infra | 13:04 | |
| EmilienM | jamespage, mordred: it works now | 13:04 |
| EmilienM | I'm logged-in | 13:04 |
| *** burgerk has quit IRC | 13:04 | |
| AJaeger | jamespage: not really - some projects use rtfm.org | 13:05 |
| jamespage | EmilienM, confirmed yes ok for me | 13:05 |
| AJaeger | jamespage: rftd probably as mordred said;) | 13:05 |
| *** jzerebec1i is now known as jzerebecki | 13:05 | |
| AJaeger | thanks, pabelanger | 13:05 |
| *** yuanying has joined #openstack-infra | 13:05 | |
| *** sdague has joined #openstack-infra | 13:06 | |
| *** YorikSar has quit IRC | 13:06 | |
| *** mdrabe has joined #openstack-infra | 13:07 | |
| *** YorikSar has joined #openstack-infra | 13:08 | |
| hferenc | AJaeger: ok, i mean either i use this "check experimental" trick or the "Depends-On: the script's change id" tag for the job's commit, because the job itself won't be merged until it has not merged dependencies | 13:08 |
| openstackgerrit | Merged openstack-infra/project-config: Add install-guide jobs for trove https://review.openstack.org/330725 | 13:08 |
| *** esberglu has joined #openstack-infra | 13:08 | |
| AJaeger | hferenc: http://docs.openstack.org/infra/manual/developers.html#limitations-and-caveats | 13:09 |
| *** roxanaghe has quit IRC | 13:09 | |
| *** yuanying has quit IRC | 13:09 | |
| mordred | #status ok OpenID logins are back to normal | 13:09 |
| openstackstatus | mordred: sending ok | 13:09 |
| mordred | jamespage: thanks a ton! | 13:09 |
| matt-borland | :) | 13:09 |
| AJaeger | hferenc: for depends-on: We checkout the source code for running. But our launchers use the current configuration for the jobs and don't get updated. So, depends-on with project-config is just a "merge constraint" | 13:10 |
| hferenc | AJaeger: i see | 13:11 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 13:11 | |
| -openstackstatus- NOTICE: OpenID logins are back to normal | 13:11 | |
| *** amitgandhinz has joined #openstack-infra | 13:11 | |
| *** rook has joined #openstack-infra | 13:12 | |
| *** eezhova has quit IRC | 13:12 | |
| *** savihou has quit IRC | 13:13 | |
| * jamespage enjoys being the message boy | 13:14 | |
| jamespage | mordred, yw | 13:14 |
| *** savihou has joined #openstack-infra | 13:14 | |
| *** ociuhandu has joined #openstack-infra | 13:14 | |
| openstackstatus | mordred: finished sending ok | 13:14 |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run glanceclient functional tests in Identity v3-only https://review.openstack.org/313572 | 13:15 |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run Swift functional tests in Identity v3-only https://review.openstack.org/313659 | 13:15 |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run Heat functional tests in Identity v3-only https://review.openstack.org/314643 | 13:15 |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run Ceilometer functional tests in Identity v3-only https://review.openstack.org/315077 | 13:15 |
| openstackgerrit | Raildo Mascena proposed openstack-infra/project-config: Run Magnum functional tests in Identity v3-only https://review.openstack.org/315118 | 13:15 |
| *** edmondsw has joined #openstack-infra | 13:15 | |
| *** amitgandhinz has quit IRC | 13:16 | |
| njohnston | Hi! Could I get a second +2 for this minor change in system-config please? https://review.openstack.org/329570 Thanks much! | 13:17 |
| *** amoralej|lunch is now known as amoralej | 13:18 | |
| *** infra-red has quit IRC | 13:19 | |
| mordred | AJaeger, jhesketh: what's our situation with bots and channels? | 13:19 |
| mordred | there was something about a limit earlier, yeah? | 13:19 |
| *** savihou has quit IRC | 13:20 | |
| *** savihou has joined #openstack-infra | 13:20 | |
| *** eezhova has joined #openstack-infra | 13:20 | |
| AJaeger | mordred: we noticed last week that gerritbot does not show up on all channels - it had 122 configured with a limit of 120. | 13:21 |
| *** burgerk has joined #openstack-infra | 13:21 | |
| AJaeger | So, we either need a second bot or the limit raised. No idea whether anybody is following up on that. fungi? | 13:21 |
| *** jcoufal has joined #openstack-infra | 13:23 | |
| *** mriedem has joined #openstack-infra | 13:23 | |
| AJaeger | mordred: no idea whether statusbot or meetbot have similiar problems | 13:24 |
| mordred | k. I'm going to hold off on landing any bot related channels until we know more | 13:25 |
| *** dtantsur|bbl is now known as dtantsur | 13:25 | |
| *** sorantis has quit IRC | 13:25 | |
| *** burgerk has quit IRC | 13:26 | |
| AJaeger | have to leave now, perhaps until later... | 13:26 |
| *** bswartz has joined #openstack-infra | 13:26 | |
| *** julim has joined #openstack-infra | 13:29 | |
| *** javeriak has joined #openstack-infra | 13:29 | |
| *** julim has quit IRC | 13:29 | |
| *** permalac__ has joined #openstack-infra | 13:30 | |
| *** permalac_ has quit IRC | 13:30 | |
| jroll | AJaeger: dtantsur: did you see my comment with the link to the other patch here? would you prefer I rebase this on top? https://review.openstack.org/#/c/331422/ | 13:31 |
| jroll | I put that comment there because i knew someone would mention it :) | 13:31 |
| *** vgridnev has quit IRC | 13:33 | |
| *** javeriak has quit IRC | 13:35 | |
| *** rdx_matt_ has joined #openstack-infra | 13:38 | |
| *** flepied has quit IRC | 13:38 | |
| *** vgridnev has joined #openstack-infra | 13:38 | |
| dtantsur | jroll, lol, and I still missed it ^_^ | 13:39 |
| *** eezhova has quit IRC | 13:39 | |
| *** _ari_ has joined #openstack-infra | 13:39 | |
| dtantsur | jroll, I think it needs rebasing | 13:40 |
| *** eezhova has joined #openstack-infra | 13:40 | |
| rdx_matt_ | Hi, I'm unable to login to my openstack wiki account | 13:41 |
| rdx_matt_ | When I'm clicking "Yes, log me in" I'm getting "The wiki.openstack.org page isn’t working wiki.openstack.org is currently unable to handle this request." | 13:42 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: zuul/layout: fix rule for beaker-xenial jobs https://review.openstack.org/331674 | 13:42 |
| mordred | rdx_matt_: hi! sorry for the inconvience. We've disabled new account signups for the moment while dealing with a lovely wiki spam attack | 13:43 |
| *** amitgandhinz has joined #openstack-infra | 13:43 | |
| rdx_matt_ | mordred: Can you enable it for a short while so I can log in? | 13:44 |
| jroll | dtantsur: yeah, can do | 13:44 |
| openstackgerrit | Jim Rollenhagen proposed openstack-infra/project-config: Make ironic grenade voting https://review.openstack.org/331422 | 13:44 |
| jroll | there it is | 13:44 |
| njohnston | thanks mordred for taking a look at my change | 13:45 |
| mordred | jeblair: my hunch on the launcher timeout thing from friday seems to be inaccurate. or, at least, it's inaccurate in as much as I can prove anything one way or the other | 13:45 |
| mordred | njohnston: sure! I'll try to get that landed once I know what we're doing with the bots | 13:46 |
| *** nwkarsten has joined #openstack-infra | 13:46 | |
| mordred | rdx_matt_: I know it can be done, but I'm not sure of the process (I've mostly been out for a month) - but I do believe that humans exist who can help | 13:46 |
| rdx_matt_ | mordrer_: Thanks :) Can anyone here assist me logging in to my account? | 13:47 |
| *** ddieterly has joined #openstack-infra | 13:48 | |
| *** ddieterly is now known as ddieterly[away] | 13:48 | |
| *** yolanda has quit IRC | 13:49 | |
| *** inc0 has joined #openstack-infra | 13:49 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for delete_subnet ObjectActionError bug 1594376 https://review.openstack.org/331677 | 13:49 |
| openstack | bug 1594376 in neutron "Delete subnet fails with "ObjectDeletedError: Instance '<IPAllocation at 0x7f6abc3abe50>' has been deleted, or its row is otherwise not present."" [Undecided,New] https://launchpad.net/bugs/1594376 | 13:49 |
| *** isaacb has quit IRC | 13:50 | |
| *** ociuhandu has quit IRC | 13:50 | |
| wznoinsk | hi all | 13:51 |
| wznoinsk | would anyone know if keystone and glance be in ci-watch in the future? | 13:52 |
| *** roxanaghe has joined #openstack-infra | 13:54 | |
| wznoinsk | or it's as simple as no 3rdparty CIs = no point to have it in ci-watch? | 13:54 |
| fungi | mordred: AJaeger: yeah, tomaw is sometimes in here and i was going to ask to have the openstackgerrit account's chanmax raised | 13:54 |
| *** cody-somerville has joined #openstack-infra | 13:54 | |
| *** ddieterly[away] is now known as ddieterly | 13:55 | |
| fungi | according to pleia2 it can be increased on a per-account basis (because she had it done for hers at one point) | 13:55 |
| rdx_matt_ | Ping - Can anyone here assist me logging in to my account? Right now it is not working. | 13:55 |
| fungi | mordred: AJaeger: i would much rather do that than run additional gerritbot accounts (and so would the freenode staff probably?) | 13:55 |
| fungi | rdx_matt_: your gerrit account? i saw something about connectivity issues for login.launchpad.net earlier | 13:56 |
| rdx_matt_ | fungi: Nope, my wiki/launchpad account. | 13:56 |
| *** tosky has quit IRC | 13:57 | |
| rdx_matt_ | fungi: Oh, yeah, same account. | 13:57 |
| *** mriedem has quit IRC | 13:57 | |
| fungi | rdx_matt_: have you signed into the wiki before? if not, i need to adjust our settings to allow it to create an account for you. we've been blocking new account creation to keep spammers at bay for a few months | 13:57 |
| *** mriedem has joined #openstack-infra | 13:58 | |
| rdx_matt_ | fungi: I have, after I try to sign in I'm redirected to "Personal Data Request" and when I click ,"Yes, Log me in" I get browser error: ""The wiki.openstack.org page isn’t working wiki.openstack.org is currently unable to handle this request."" | 13:58 |
| fungi | rdx_matt_: i've enabled new account creation temporarily, can you try to log in again? | 13:58 |
| *** andreas_s has quit IRC | 13:58 | |
| fungi | rdx_matt_: looks like it created an account for "mattan" just now | 13:59 |
| rdx_matt_ | fungi: Great, works. Thanks!! | 13:59 |
| *** hongbin has joined #openstack-infra | 13:59 | |
| fungi | probably you had a different account on it before or something | 13:59 |
| *** andreww has quit IRC | 14:00 | |
| *** rajinir has joined #openstack-infra | 14:00 | |
| *** _sigmavirus24 is now known as sigmavirus24 | 14:00 | |
| *** sigmavirus24 has joined #openstack-infra | 14:00 | |
| *** yolanda has joined #openstack-infra | 14:01 | |
| *** woodster_ has joined #openstack-infra | 14:02 | |
| *** rdx_matt has joined #openstack-infra | 14:02 | |
| *** PollyZ has joined #openstack-infra | 14:02 | |
| *** burgerk has joined #openstack-infra | 14:03 | |
| *** tosky has joined #openstack-infra | 14:03 | |
| zigo | fungi: https://review.openstack.org/#/c/331008/ <--- Could you have a quick look? I'm trivially using a known folder instead of {project} for the upload. | 14:04 |
| *** eharney has joined #openstack-infra | 14:04 | |
| *** scottynomad has quit IRC | 14:04 | |
| *** scottynomad has joined #openstack-infra | 14:04 | |
| mmedvede | wznoinsk: I am not aware why every existing project is not tracked by ci-watch. My guess is it was easier to have a preset number of projects initially. | 14:05 |
| *** milan has joined #openstack-infra | 14:05 | |
| *** sorantis has joined #openstack-infra | 14:06 | |
| *** zhurong has joined #openstack-infra | 14:06 | |
| openstackgerrit | Jim Rollenhagen proposed openstack-infra/project-config: Use ironic-core for ironic-python-agent ACLs https://review.openstack.org/331425 | 14:07 |
| *** rbrndt has joined #openstack-infra | 14:07 | |
| *** trown is now known as trown|brb | 14:09 | |
| fungi | jhesketh: AJaeger: open_stack is the account name the meetbot authenticates as, openstack is the nick it uses http://git.openstack.org/cgit/openstack-infra/project-config/tree/accessbot/channels.yaml#n143 (so the access is right, chances are it just lost chanops during a netsplit/reconnect when chanserv wasn't available to re-op it) | 14:10 |
| *** daemontool has quit IRC | 14:10 | |
| fungi | aspiers: ^ | 14:11 |
| fungi | i'll explicitly ask chanserv to re-op it now | 14:11 |
| fungi | fixed on all our meeting channels now | 14:14 |
| *** kien-ha has joined #openstack-infra | 14:14 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Do timeout calculation in jinja https://review.openstack.org/331686 | 14:14 |
| mordred | jeblair: ^^ that works better for me - used this: http://paste.openstack.org/show/520603/ to test it | 14:14 |
| *** yamamoto has joined #openstack-infra | 14:15 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for delete_subnet ObjectActionError bug 1594376 https://review.openstack.org/331677 | 14:15 |
| openstack | bug 1594376 in neutron "Delete subnet fails with "ObjectDeletedError: Instance '<IPAllocation at 0x7f6abc3abe50>' has been deleted, or its row is otherwise not present."" [High,New] https://launchpad.net/bugs/1594376 | 14:15 |
| wznoinsk | mmedvede: I wasn't sure what's exactly the purpose of the ci-watch hence the question, if that's to eyeball 3rdparty CIs vs. jenkins then it makes sense to have there only projects that have any sort of 3rdparty testing at all, if ci-watch becoming more openstack infra tooling then it's a different story altogether | 14:16 |
| jeblair | mordred: excellent, thank you! that explanation makes sense -- well mostly -- | 14:16 |
| jeblair | mordred: the bit about it not being set until the task completes -- that's by design | 14:16 |
| jeblair | mordred: what i mean is that the callback plugin was intended to update the timeout after each task completed | 14:17 |
| mordred | yah - so when a task completes, in theory you'd be setting the timeout for the next task | 14:17 |
| jeblair | right, that was the idea | 14:17 |
| mordred | yup. makes sense - oh, so for the commit message, what I'm leaving out | 14:17 |
| mordred | is that I also tried splitting timeout into two variables so that you set the extra_vars one with one variable name, and then had the callback set a different value | 14:18 |
| *** trown|brb is now known as trown | 14:18 | |
| mordred | and that doesn't work because it runs at task end | 14:18 |
| fungi | zigo: the reason the {project} parameter wasn't set to anything in the deb-pkgs macro is because the pkgdeb-publish-openstack-pkg-tools job didn't pass in a value for it | 14:18 |
| *** andreww has joined #openstack-infra | 14:18 | |
| mordred | jeblair: actually, come to think of it, that should still potentially work ... | 14:18 |
| jeblair | mordred: but isn't that where the elapsed time is getting set? | 14:18 |
| mordred | yah. I'll go back through and make sure I didn't mis-test something | 14:19 |
| zigo | fungi: Even if we can fix that, I don't think it helps in anyway for the pickup job. | 14:19 |
| mordred | but in any case, setting elapsed time at the end and does seem to always make it available to subsequent task processing | 14:19 |
| mordred | so that's good at least | 14:19 |
| fungi | zigo: i don't know what you mean by "the pickup job: | 14:19 |
| jeblair | mordred: okay. however, if we have to use 2 variables, this way (elapsed time) probably makes the most sense anyway, so i'd suggest we go with what you have | 14:19 |
| mordred | ++ | 14:19 |
| zigo | fungi: The job that will take the produced .deb files and push them in a repository. | 14:20 |
| jeblair | mordred: (but doing the experiment might be good for understanding this stuff, cause, it's weird, right? :) | 14:20 |
| mordred | it is weird | 14:20 |
| mordred | for sure | 14:20 |
| zigo | fungi: Just having a single <known-hard-coded-folder-name>/<sha256> is IMO enough. | 14:20 |
| fungi | zigo: is the job you have there meant to be the one which builds packages, or the one which updates the package repository to include them? | 14:20 |
| *** rhochmuth has joined #openstack-infra | 14:20 | |
| mmedvede | wznoinsk: main usecase for ci-watch is comparing stability of third-party CI s between each other and infra CI | 14:20 |
| *** nibalizer has joined #openstack-infra | 14:21 | |
| zigo | fungi: What we discussed is that, after a merge, we push files to tarballs.o.o, then a job for maintaining the repo (what I call the pick-up job) takes the files from there. | 14:21 |
| zigo | fungi: Then we can regularly clean whats pushed in tarballs.o.o as it's only a temp folder. | 14:22 |
| *** cody-somerville has quit IRC | 14:22 | |
| zigo | fungi: The pickup job could even clean it if it had enough rights. | 14:22 |
| mordred | jeblair: oh! so what failed in my testing of the two-timeout-variables is that we're subtracting the int() of the elapsed_time - which in my synthetic testing is never high enough to be 1 | 14:22 |
| zigo | Does it makes sense now? | 14:22 |
| mordred | jeblair: so I just see the timeout set in that var | 14:22 |
| mordred | so that makes sense | 14:22 |
| jeblair | mordred: okay cool | 14:23 |
| jeblair | mordred: so the main issue is "we can't override the extra var" | 14:23 |
| *** zz_dimtruck is now known as dimtruck | 14:24 | |
| fungi | zigo: it's not a pattern we've used in the past.; we certainly don't expect tarballs.openstack.org to be temporary storage (it may be redundant copies of some files which we also upload to other places, but at most we replace files on it with updated versions we don't really ever delete or "clean up" files there) | 14:24 |
| *** mdrabe has quit IRC | 14:24 | |
| zigo | fungi: That's what mordred and pabelanger advised me to do, so I did that. | 14:25 |
| openstackgerrit | Neil Jerram proposed openstack-infra/irc-meetings: Add networking-calico meeting https://review.openstack.org/331689 | 14:25 |
| *** mdrabe has joined #openstack-infra | 14:25 | |
| *** lucas-hungry is now known as lucasagomes | 14:25 | |
| zigo | fungi: If you have another idea for a design, I may implement it. | 14:25 |
| *** cody-somerville has joined #openstack-infra | 14:25 | |
| *** cody-somerville has joined #openstack-infra | 14:25 | |
| zigo | fungi: The idea was to have a temp storage, so that the machine holding keys for the repo wouldn't be the one pushing files at merge time. | 14:25 |
| zigo | fungi: If you have another place where to scp, I guess it wouldn't be hard to change the destination host... | 14:26 |
| zigo | fungi: Your thoughts? | 14:26 |
| *** kushal has quit IRC | 14:26 | |
| mordred | jeblair: yah | 14:28 |
| mordred | zigo: the above design is correct except for the temp folder part- it's a two step publication process - artifacts get pushed to tarballs.o.o, then downloaded by the repo worker and added to repo | 14:29 |
| *** isaacb has joined #openstack-infra | 14:29 | |
| *** mdrabe_ has joined #openstack-infra | 14:29 | |
| mordred | zigo: also, sorry I havne't been more help recently - I've been basically out of office for the last month | 14:29 |
| aspiers | fungi_lounge: thanks! (re meetbot issue) I wonder if there is a way to make that automatically self-correcting for the next time there is a netsplit or whatever | 14:29 |
| EmilienM | AJaeger, pabelanger: if you can review it today (small patch) https://review.openstack.org/#/c/331674/ - thanks | 14:30 |
| *** mdrabe has quit IRC | 14:30 | |
| zigo | mordred: I'm not sure if you followed the last evolution, but now, the TC wants to remove packaging-deb from the big tent, because of no visible progress. | 14:30 |
| zigo | :( | 14:30 |
| mordred | yah - I saw that. will respond soon | 14:31 |
| fungi | aspiers: it doesn't usually happen, it seems to require a very specific sequence of events to end up with the meetbot in channels but no longer a channel operator. and that doesn't actually break anything except its ability to manipulate channel topics, so it's a pretty low priority to track down and solve | 14:32 |
| fungi | meeting commands and logging all still work fine even if the meetbot isn't a chanop | 14:33 |
| derekh | Is there anybody around to take a look at adding a new cloud rh2 for tripleo ci https://review.openstack.org/#/c/330514/3 | 14:33 |
| derekh | and a experimental job to test it out https://review.openstack.org/#/c/330515/3 | 14:33 |
| aspiers | fungi: OK, sounds like not worth worrying too much about it then | 14:33 |
| fungi | zigo: mordred: yeah, it's the introduction of tarballs.o.o as "temporary storage space" and discussion of cleanup that concerned me | 14:34 |
| aspiers | fungi: actually, worth making the bot say something like "I don't have permissions to change the topic"? | 14:34 |
| aspiers | fungi: it could say that once per meeting | 14:34 |
| *** tonytan4ever has joined #openstack-infra | 14:34 | |
| *** Liuqing has quit IRC | 14:34 | |
| fungi | aspiers: that may be doable. have a look at http://git.openstack.org/cgit/openstack-infra/meetbot for the plugin we're using | 14:34 |
| *** julim has joined #openstack-infra | 14:34 | |
| *** burgerk has quit IRC | 14:35 | |
| *** _nadya_ has joined #openstack-infra | 14:36 | |
| *** sorantis has quit IRC | 14:36 | |
| mordred | fungi: right. we do not need to do that, it's a miscommunication | 14:37 |
| *** rossella_s has quit IRC | 14:37 | |
| *** abregman has quit IRC | 14:37 | |
| *** rossella_s has joined #openstack-infra | 14:37 | |
| mordred | fungi: same as the other things, we need to copy the artifacts somewhere before we can let trusted workers operate on them. same situation here | 14:37 |
| *** amitgandhinz has quit IRC | 14:38 | |
| mordred | zigo: I have replied to the change | 14:38 |
| fungi | mordred: yep, that's what i had thought the original plan was as well | 14:38 |
| Shrews | mordred: i have learned things from your timeout patch | 14:38 |
| *** amitgandhinz has joined #openstack-infra | 14:38 | |
| mordred | Shrews: me too! | 14:38 |
| fungi | anyway, i'm going to spend more time today hacking on key management for artifact signing (if no other major emergencies crop up), which may be germane to this as well | 14:39 |
| mordred | fungi: woot! | 14:39 |
| mordred | fungi: I would like key management | 14:39 |
| *** ramishra has joined #openstack-infra | 14:40 | |
| fungi | mostly the puppet bits to install a keychain containing only the signing subkey | 14:40 |
| *** PollyZ has quit IRC | 14:40 | |
| *** zul_ is now known as zul | 14:40 | |
| *** burgerk has joined #openstack-infra | 14:43 | |
| *** jamesmcarthur has joined #openstack-infra | 14:43 | |
| *** ramishra has quit IRC | 14:44 | |
| *** ddieterly is now known as ddieterly[away] | 14:45 | |
| mordred | jeblair, Shrews: it's also worth noting that I _think_ that timeout is only valid for a given play and if a second play were added to the playbook the timeout value would reset | 14:45 |
| mordred | this is not a problem right now, since we only have one play in the playbook - but we might need to dig in once we start supporting arbitrary playbooks with different sequences of plays across host groupings in v3 | 14:45 |
| *** jlanoux has quit IRC | 14:47 | |
| Shrews | mordred: you mean elapsed_time would reset? or timeout? timeout isn't changing with your patch | 14:47 |
| mordred | jeblair, Shrews: nevermind. please disregard everything I just said. I'm clearly lying | 14:47 |
| mordred | I meant elapsed_time - but I'm just 100% wrong | 14:47 |
| *** ddieterly[away] is now known as ddieterly | 14:47 | |
| Shrews | k k | 14:47 |
| *** ayoung has quit IRC | 14:48 | |
| *** ramishra has joined #openstack-infra | 14:49 | |
| *** inc0_ has joined #openstack-infra | 14:51 | |
| *** mriedem1 has joined #openstack-infra | 14:51 | |
| *** mdrabe_ has quit IRC | 14:52 | |
| *** mriedem has quit IRC | 14:52 | |
| *** rdx_matt_ has quit IRC | 14:52 | |
| *** ramishra has quit IRC | 14:52 | |
| *** inc0_ has quit IRC | 14:53 | |
| *** inc0 has quit IRC | 14:53 | |
| *** mriedem has joined #openstack-infra | 14:53 | |
| *** yolanda has quit IRC | 14:54 | |
| *** matrohon has quit IRC | 14:54 | |
| *** PollyZ has joined #openstack-infra | 14:55 | |
| *** Nikolay_St has quit IRC | 14:55 | |
| *** mriedem1 has quit IRC | 14:56 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update FindBugs doc options to have default value https://review.openstack.org/331711 | 14:56 |
| openstackgerrit | Hisashi Osanai proposed openstack-infra/project-config: Fix Monasca Analytics python27 job https://review.openstack.org/331535 | 14:56 |
| *** hosanai has joined #openstack-infra | 14:57 | |
| *** sorantis has joined #openstack-infra | 14:58 | |
| *** cody-somerville has quit IRC | 14:59 | |
| *** mdrabe has joined #openstack-infra | 14:59 | |
| *** pgadiya has quit IRC | 14:59 | |
| apuimedo | could somebody please approve and merge https://review.openstack.org/#/c/331333/ ? | 15:00 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Only request desired timeline events https://review.openstack.org/331646 | 15:01 |
| *** haaj has quit IRC | 15:01 | |
| *** Na3iL has quit IRC | 15:01 | |
| *** sorantis has quit IRC | 15:04 | |
| *** trown is now known as trown|lunch | 15:05 | |
| openstackgerrit | Hisashi Osanai proposed openstack-infra/project-config: Fix Monasca Analytics python27 job https://review.openstack.org/331535 | 15:05 |
| *** trown|lunch is now known as trown|mtg | 15:05 | |
| *** pots has joined #openstack-infra | 15:07 | |
| *** sc68cal has quit IRC | 15:07 | |
| *** zhurong has quit IRC | 15:07 | |
| apuimedo | dims: tonyb lifeless dirk | 15:07 |
| apuimedo | ^^ | 15:07 |
| *** ramishra has joined #openstack-infra | 15:09 | |
| *** garyk has quit IRC | 15:11 | |
| *** amitgandhinz has quit IRC | 15:11 | |
| zigo | fungi: mordred: If we shouldn't use tarballs.o.o as temp, then what should be used? | 15:12 |
| *** mriedem is now known as mriedem_away | 15:12 | |
| *** amitgandhinz has joined #openstack-infra | 15:12 | |
| fungi | zigo: i think the idea is to use tarballs.o.o but don't treat it as "temporary storage" (plan for the things you upload there to exist indefinitely) | 15:12 |
| fungi | it may be "redundant" storage but not "temporary" | 15:13 |
| zigo | fungi: I don't see what's the point of storing each commit as built packages. | 15:13 |
| *** dprince has quit IRC | 15:13 | |
| zigo | fungi: Only tagged version could be kept, and for that, there's already snapshot.debian.org. | 15:14 |
| *** mriedem_away has quit IRC | 15:14 | |
| fungi | zigo: why not just replace the old files with new copies (e.g., built packages of the "latest" commit) | 15:14 |
| *** vincentll has quit IRC | 15:14 | |
| zigo | We just need the latest version of the package for a given distro + openstack release ... | 15:14 |
| fungi | we do that with our "branch tarballs" already | 15:14 |
| fungi | we don't overwrite them, we simply upload updated versions over top the originals | 15:15 |
| zigo | fungi: So, basically, just generate a /debian/pool structure then? | 15:15 |
| *** ramishra has quit IRC | 15:15 | |
| zigo | I'm really not sure what you want me to do here. | 15:15 |
| zigo | Can I even do the delete on merge of a patch? | 15:16 |
| *** arxcruz has quit IRC | 15:16 | |
| *** jlanoux has joined #openstack-infra | 15:16 | |
| fungi | zigo: upload files to tarballs.openstack.org in your build job, then leave them there (you can upload the files to the same path/name and replace the old ones if you want), just don't delete them | 15:17 |
| *** nmagnezi has joined #openstack-infra | 15:17 | |
| fungi | zigo: then in your "pickup" job, retrieve those files and use them to build your package repository | 15:17 |
| *** hrubi has quit IRC | 15:17 | |
| zigo | fungi: On each merge, the name of the package will change, so it wont overwrite anything, | 15:17 |
| zigo | s/name of/version of/ | 15:17 |
| *** sree has quit IRC | 15:18 | |
| fungi | but really, the number of commits to each packaging repo is likely to be small (a few dozen per cycle? how often do you update the packaging files?) so keeping them all around indefinitely on tarballs.openstack.org shouldn't pose a problem | 15:18 |
| mordred | yah | 15:18 |
| mordred | let's not get fixated on solving the problem of pushing too many artifacts to tarballs.o.o | 15:19 |
| mordred | if we start running low on disk, we can solve deleting unneeded things | 15:19 |
| fungi | right, that's my thought on the matter as well | 15:19 |
| mordred | but for now, I think uploading to tarballs, downloading and adding to repo is more important than cleaning up | 15:19 |
| *** tlian2 has joined #openstack-infra | 15:19 | |
| *** isaacb has quit IRC | 15:20 | |
| zigo | Allright then. | 15:20 |
| zigo | mordred: fungi: Is https://review.openstack.org/#/c/331008/ still a problem? | 15:20 |
| zigo | Or is that fine? | 15:20 |
| dims | apuimedo : +2 | 15:20 |
| *** hrubi has joined #openstack-infra | 15:20 | |
| *** ramishra has joined #openstack-infra | 15:20 | |
| zigo | Do we want to revert and fix {project} ? | 15:21 |
| apuimedo | thanks dims | 15:21 |
| fungi | zigo: depends on whether you want to upload to a different directory for each project whose jobs use that publisher macro | 15:21 |
| zigo | fungi: I really don't care. :) | 15:21 |
| zigo | What I'll soon end-up doing is uploading a full useable package repository, so that we can test using it. | 15:22 |
| zigo | Though it doesn't mater where I upload. | 15:22 |
| *** _nadya_ has quit IRC | 15:22 | |
| zigo | Hum... | 15:22 |
| *** _nadya_ has joined #openstack-infra | 15:23 | |
| fungi | zigo: then seems fine to me, but also odyssey4me makes a point i also suggested a while back which is there's no point in having that publisher in macros.yaml, just add it to the job definition directly | 15:23 |
| * zigo thinks that probably, everything in a single folder could be a mess, there will be too many stuff. | 15:23 | |
| *** tlian has quit IRC | 15:23 | |
| zigo | fungi: It's going to be re-used as soon as we get this Debian repo up and running. | 15:23 |
| *** vhosakot has joined #openstack-infra | 15:24 | |
| zigo | fungi: So, what's the way to fix the {project} stuff then? | 15:24 |
| fungi | easier to keep the job definition all in one place until it's working the way you want. when you get to the point where you want to start running it for multiple projects, then go to the additional effort of modularizing it and parameterizing it for use by additional jobs and projects | 15:24 |
| *** tonytan4ever has quit IRC | 15:24 | |
| fungi | let me approach this from a different angle... will the " | 15:25 |
| fungi | argh, stray newline | 15:25 |
| zigo | fungi: I don't think it's a good idea to ask me to do something I will revert in a week of time. | 15:25 |
| fungi | zigo: will the "pkgdeb-publish-openstack-pkg-tools" job every be used for other repositories besides "openstack-pkg-tools"? | 15:26 |
| fungi | er, ever | 15:26 |
| zigo | No, but the publisher will. | 15:26 |
| zigo | Or am I missing something? | 15:26 |
| *** dizquierdo has quit IRC | 15:27 | |
| *** cody-somerville has joined #openstack-infra | 15:27 | |
| *** tmorin has left #openstack-infra | 15:27 | |
| fungi | okay, so you will have a different job which also runs the "deb-pkgs" publisher. you want them all to upload into https://tarballs.openstack.org/packaging-deb/ or into separate directories? | 15:27 |
| *** locust has joined #openstack-infra | 15:27 | |
| locust | hello | 15:28 |
| *** harlowja_at_home has joined #openstack-infra | 15:28 | |
| locust | I am settuping my own CI, should i use as example /etc/puppet/manifests/site.pp | 15:28 |
| locust | or it will be better to use http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/manifests/single_node_ci.pp | 15:29 |
| zigo | fungi: Into something like /packaging-deb/<NAME-OF-SOURCE-PACKAGE>/<SHA256> is probably the thing we should do. | 15:29 |
| *** banix has joined #openstack-infra | 15:29 | |
| asselin | locust, for 3rd party ci, that link is recommended and what the docs reference | 15:29 |
| *** ramishra has quit IRC | 15:29 | |
| asselin | locust, http://docs.openstack.org/infra/openstackci/ | 15:30 |
| zigo | If it is /packaging-deb/deb-<NAME-OF-SOURCE-PACKAGE>/<SHA256> that's not a big deal though. | 15:30 |
| zigo | (since git repo names will be prefixed with "deb-") | 15:30 |
| *** sree has joined #openstack-infra | 15:30 | |
| *** bapalm has joined #openstack-infra | 15:31 | |
| locust | asselin: but it looks like doc file is different from the one I get from git repo | 15:31 |
| AJaeger | fungi, do we hit limits on accessbot, statusbot or meetbot as well? | 15:31 |
| fungi | zigo: okay, so the pkgdeb-publish-openstack-pkg-tools job currently creates files at uploads/deb-openstack-pkg-tools/<SHA256>/... ? | 15:31 |
| fungi | AJaeger: we haven't yet, but i think a lot more projects use gerritbot than the others | 15:32 |
| fungi | AJaeger: also accessbot doesn't join channels | 15:32 |
| zigo | fungi: No, at http://tarballs.openstack.org/%7Bproject%7D/uploads/220dead31374edabced0d890bfaf2f900da4b66c/ | 15:32 |
| zigo | Which should be fixed... | 15:32 |
| zigo | fungi: All the bikeshedding is about fixing this uglyness. | 15:32 |
| fungi | zigo: well, right, i mean where do you want it uploading? | 15:32 |
| fungi | yes, i'm trying to end the bikeshedding and help you fix it, though there are lots of other things i need to be spending time on right now | 15:33 |
| zigo | In an ideal world, at: http://tarballs.openstack.org/packaging-deb/openstack-pkg-tools/220dead31374edabced0d890bfaf2f900da4b66c/ | 15:33 |
| *** sree_ has joined #openstack-infra | 15:33 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: zuul/layout: fix rule for beaker-xenial jobs https://review.openstack.org/331674 | 15:34 |
| EmilienM | AJaeger: done ^ | 15:34 |
| *** sc68cal has joined #openstack-infra | 15:34 | |
| *** sree_ is now known as Guest52727 | 15:34 | |
| locust | asselin: for instance from git: project_config_repo => hiera('project_config_repo'), | 15:34 |
| zigo | fungi: My bad, sorry for taking too much time to explain correctly... :( | 15:34 |
| locust | asselin: from web $project_config_repo = undef, | 15:35 |
| *** ramishra has joined #openstack-infra | 15:35 | |
| *** sree has quit IRC | 15:35 | |
| fungi | zigo: getting rid of the "uploads" in there is going to be complicated. i'm hunting down an example of where we've implemented a similar workaround | 15:37 |
| zigo | fungi: Leave the "uploads" there then. | 15:37 |
| AJaeger | fungi, good - thanks. Please tell me once the limit for gerritbot is changed so that we can approve changes again. | 15:37 |
| AJaeger | EmilienM: Let me review quickly... | 15:37 |
| anteaya | zaro: any objection to me fixing rocket alignment => in https://review.openstack.org/#/c/330922/3 ? | 15:37 |
| zigo | fungi: The important bit is having {project} expanded correctly, so we don't end up with thousands of files in a single repo. | 15:38 |
| *** Guest52727 has quit IRC | 15:38 | |
| EmilienM | AJaeger: thanks | 15:38 |
| zigo | s/files/folders/ | 15:38 |
| fungi | zigo: here's how we would probably need to go about getting rid of the "upload" part: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n736 | 15:38 |
| fungi | zigo: for the {project} parameter, you just need to add "project: openstack-pkg-tools" in the pkgdeb-publish-openstack-pkg-tools job definition indented immediately below the "publisher: deb-pkgs" line | 15:39 |
| odyssey4me | jeblair pabelanger jhesketh The async timeout is a plague. Are other projects not seeing the same thing? https://etherpad.openstack.org/p/zuul-timeouts | 15:39 |
| *** mikelk has quit IRC | 15:40 | |
| *** claudiub has joined #openstack-infra | 15:42 | |
| fungi | odyssey4me: is it still happening today? i thought we merged at least a workaround for it over the weekend | 15:42 |
| hosanai | hello, one question about patch 331535. the layout job failed with "Job gate-monasca-analytics-dsvm-python27 not defined" but a template for the job is in this patch (monasca.yaml). do i need to define it in other places? | 15:42 |
| jeblair | odyssey4me: i'm just setting down to look. i'm reviewing some patches to fix up and get debug info for timeouts which we should be able to get into production today. | 15:42 |
| jeblair | fungi: did we? | 15:42 |
| anteaya | AJaeger: what is the limit we are hitting with gerritbot, I can't seem to find the beginning of the conversation | 15:42 |
| *** burgerk_ has joined #openstack-infra | 15:42 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Fix upload path to tarballs.o.o for packaging https://review.openstack.org/331008 | 15:43 |
| AJaeger | anteaya: 120 channels | 15:43 |
| zigo | fungi: Like this? | 15:43 |
| jeblair | odyssey4me: i can try to check the logs and see if it's happening elsewhere | 15:43 |
| anteaya | AJaeger: oh thanks, I guess we found that limit by crossing it, or trying to | 15:43 |
| *** sorantis has joined #openstack-infra | 15:43 | |
| jeblair | mordred: this change should be compatible with yours, right? https://review.openstack.org/331426 | 15:44 |
| anteaya | hosanai: yes, it has to me called in your repo section in jenkins/jobs/project.yaml | 15:44 |
| zigo | fungi: (note: I don't care much about the "uploads" folder, we can fix that later) | 15:44 |
| AJaeger | anteaya: see http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-06-15.log.html#t2016-06-15T14:01:41 for previous discussion - and I think the day before as well | 15:44 |
| pabelanger | odyssey4me: jeblair: I am around again to help provide support if needed. | 15:44 |
| anteaya | thank you | 15:44 |
| fungi | jeblair: your 331199 and 331210 changes which merged on friday, we restarted all the zuul-launchers over the weekend to pick those up (along with some other unrelated patches) | 15:45 |
| hosanai | anteaya: thanks! | 15:45 |
| *** tphummel has joined #openstack-infra | 15:45 | |
| *** pcaruana has quit IRC | 15:45 | |
| *** banix has quit IRC | 15:45 | |
| *** ddieterly is now known as ddieterly[away] | 15:46 | |
| odyssey4me | fungi jeblair pabelanger Yeah, unfortunately it is still happening. It seems to be less frequent, but that may just be due to less patches in the queue being tested... not sure. | 15:46 |
| fungi | zigo: almost, see inline comment for correct yamling | 15:46 |
| *** burgerk has quit IRC | 15:46 | |
| odyssey4me | fungi jeblair pabelanger They do get through eventually - sometimes it takes one recheck, sometimes many. | 15:46 |
| anteaya | AJaeger: ah yes, I remember the multiple gerritbot suggestion now | 15:47 |
| jeblair | fungi: ah yeah those would not address odyssey4me's problem which can be summarized as "ansible appears to timeout tasks far too early" | 15:47 |
| mordred | jeblair: I do not believe that change will work | 15:47 |
| odyssey4me | fungi jeblair pabelanger Unfortunately seeing this many rechecks happen is teaching people bad habits. :/ | 15:47 |
| anteaya | AJaeger: thanks for helping me get the bigger picture | 15:47 |
| mordred | jeblair: the callback plugin is pulling values from extra_vars | 15:47 |
| AJaeger | you're welcome, anteaya | 15:47 |
| *** Ravikiran_K has quit IRC | 15:47 | |
| fungi | jeblair: oh, got it. i was missing context on what sort of timeout symptoms those were | 15:47 |
| jeblair | mordred: oh, they don't end up in the same place? | 15:47 |
| mordred | jeblair: no. I also learned that this morning | 15:48 |
| jeblair | mordred: neat. i think i need a map. | 15:48 |
| AJaeger | odyssey4me: so, sometimes we run into the timeout and sometimes not? Was one of our zuul-launchers not updated? | 15:48 |
| *** tongli has joined #openstack-infra | 15:48 | |
| fungi | anteaya: the multiple gerritbots idea came before pleia2 mentioned that freenode staff are sometimes willing to increase chanmax on a per-account basis | 15:48 |
| mordred | jeblair: I believe that we'd have to look at something in result._host | 15:48 |
| mordred | jeblair: or something | 15:48 |
| mordred | jeblair: I honestly have no idea where the vars at the play level end up :) | 15:48 |
| anteaya | fungi: oh that sounds like a good first direction | 15:49 |
| mordred | I just know that they don't send up in manager.extra_vars | 15:49 |
| jeblair | fungi: the next step in our debugging is "get debug info to ensure that we are passing the timeout value we think we are". but before i land that, i'm also going to land "actually decrement the timeout correctly" | 15:49 |
| anteaya | anyone know the status on that suggestion? | 15:49 |
| *** gyee has joined #openstack-infra | 15:50 | |
| fungi | anteaya: status is nobody's had time to hunt down freenode staff and ask them to increase that value | 15:50 |
| fungi | i'm not sure who the request needs to come from | 15:50 |
| anteaya | fungi: I had thought so | 15:50 |
| anteaya | I'm guessing jeblair | 15:50 |
| mordred | fungi: we should maybe see about getting jeblair irc superpowers federated in some manner | 15:51 |
| jeblair | fungi, odyssey4me: however, i strongly suspect we are passing in timeout values that are *no less* than the entire job timeout, and so none of that will actually fix odyssey4me's problem. it's just necessary that we confirm that before we head off into crazyland. so while that's cooking, i'll look for othe correlations and perhaps hold a node running the job and see what there is to see. | 15:51 |
| mordred | jeblair: ++ | 15:51 |
| anteaya | jeblair: when you have a moment could you request the freenode staff up the gerritbot account from 120 channels to something higher? | 15:51 |
| openstackgerrit | Peter Stachowski proposed openstack-infra/project-config: [trove] Add trove-dashboard to (more) PROJECTS https://review.openstack.org/331738 | 15:51 |
| *** annegentle has joined #openstack-infra | 15:51 | |
| jeblair | mordred: i'd love that, though i don't know that they do that. i can ask. | 15:51 |
| openstackgerrit | Merged openstack-infra/project-config: Do not run any grenade jobs on mitaka and older branches https://review.openstack.org/331046 | 15:51 |
| jeblair | anteaya, fungi: i don't know that this request needs to come from me, but i can try. | 15:51 |
| anteaya | jeblair: we are currently at the 120 limit and are blocking patches to add more channels | 15:51 |
| anteaya | jeblair: thanks | 15:51 |
| *** Sukhdev_ has joined #openstack-infra | 15:52 | |
| jeblair | pleia2: have you asked about the chanmax by chance? | 15:52 |
| fungi | jeblair: it's more that i haven't had time to look up how to contact freenode staff since i do it so infrequently i can never remember the process | 15:52 |
| jeblair | fungi: topic in #freenode :) | 15:52 |
| fungi | something like /info and then /msg a random staffer listed as active | 15:53 |
| fungi | ahh, /join #freenode would be a good first step i guess | 15:53 |
| odyssey4me | jeblair the timeout too early sometimes does happen and can be seen in some of the logs I've linked... maybe it's historical now and perhaps some of the failures mentioned in the etherpad should be split into a new symptom which covers that one | 15:53 |
| *** ddieterly[away] is now known as ddieterly | 15:53 | |
| odyssey4me | pabelanger fungi ^ | 15:54 |
| openstackgerrit | Hisashi Osanai proposed openstack-infra/project-config: Fix Monasca Analytics python27 job https://review.openstack.org/331535 | 15:54 |
| odyssey4me | AJaeger I have not tried to see whether it's consistent in terms of the provider it's happening in, but yes - sometimes the job fails even if the log shows it passed... and the ansible log thinks it timed out. | 15:54 |
| *** rcernin has quit IRC | 15:55 | |
| jeblair | mordred: "the field 'async' has an invalid value (598.162073135), and could not be converted to an int. | 15:55 |
| odyssey4me | jeblair pabelanger fungi Should I -2/-w one of the unmerged jobs to ensure that it does not merge? | 15:55 |
| mordred | jeblair: piddle. we need to int that then | 15:55 |
| mordred | one sec - lemme test something locally | 15:55 |
| openstackgerrit | Merged openstack-infra/project-config: Disable Tempest plugins gate for Aodh in stable/liberty https://review.openstack.org/331501 | 15:56 |
| openstackgerrit | Hisashi Osanai proposed openstack-infra/project-config: Fix Monasca Analytics python27 job https://review.openstack.org/331535 | 15:56 |
| openstackgerrit | Merged openstack-infra/project-config: Promote existing OSA non-voting Xenial tests to voting https://review.openstack.org/331157 | 15:56 |
| *** ociuhandu has joined #openstack-infra | 15:56 | |
| odyssey4me | jeblair pabelanger fungi hmm, we don't have any of those left - I guess a dummy patch will do | 15:56 |
| *** mriedem has joined #openstack-infra | 15:57 | |
| openstackgerrit | Merged openstack-infra/project-config: Promote existing OSA non-voting CentOS-7 tests to voting https://review.openstack.org/331191 | 15:57 |
| openstackgerrit | Merged openstack-infra/project-config: Promote existing OSA experimental Xenial jobs to non-voting https://review.openstack.org/331203 | 15:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul: Do timeout calculation in jinja https://review.openstack.org/331686 | 15:57 |
| mordred | jeblair: sorry bout that - I have verified the jinja syntax for making the result of that math an int | 15:57 |
| *** amitgandhinz has quit IRC | 15:57 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Add the ability to permalink to a comment https://review.openstack.org/289078 | 15:58 |
| zigo | fungi: Got it. | 15:58 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Fix upload path to tarballs.o.o for packaging https://review.openstack.org/331008 | 15:58 |
| *** amitgandhinz has joined #openstack-infra | 15:58 | |
| openstackgerrit | Hisashi Osanai proposed openstack-infra/project-config: Fix Monasca Analytics python27 job https://review.openstack.org/331535 | 15:58 |
| *** jaosorior has quit IRC | 15:58 | |
| * zigo got to go home | 15:59 | |
| *** zul has quit IRC | 16:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Promote existing OSA experimental CentOS-7 jobs to non-voting https://review.openstack.org/331211 | 16:00 |
| openstackgerrit | Merged openstack-infra/project-config: Update ironic-multitenant network vairable names https://review.openstack.org/329919 | 16:00 |
| openstackgerrit | Anita Kuno proposed openstack-infra/puppet-gerrit: Add configurations for gerrit its plugins https://review.openstack.org/330922 | 16:00 |
| jeblair | mordred: oops, your new_timeout thing snuck in there... | 16:00 |
| locust | asselin: I see, I got the difference | 16:00 |
| locust | asselin: thank you | 16:00 |
| *** sorantis has quit IRC | 16:01 | |
| jeblair | mordred: so we could go with your approach -- but i just locally tested a different approach which was to change all the floats to int, and then set the elapsed_time fact as an int too (but still keep _elapsed_time internally as a float for better accuracy) | 16:01 |
| jeblair | mordred: should be the same either way... do you have a preference? | 16:01 |
| *** rbradfor_ is now known as rbradfor | 16:01 | |
| *** maishsk has quit IRC | 16:01 | |
| *** yamamoto has quit IRC | 16:01 | |
| *** jaosorior has joined #openstack-infra | 16:01 | |
| mordred | jeblair: I think it'll look clearner your way | 16:02 |
| jeblair | mordred: okay, i'll push it up then | 16:02 |
| mordred | jeblair: ( timeout | float - elapsed_time) | int ... is a bit ugly | 16:02 |
| *** maishsk has joined #openstack-infra | 16:02 | |
| asselin | locust, sorry had to step away a bit. looks like you're all set though. | 16:02 |
| Shrews | clearner is totally a cromulent word | 16:02 |
| mordred | the cromulentest | 16:03 |
| * fungi feels embiggened by it | 16:03 | |
| anteaya | is there a test for cromulentness? | 16:03 |
| *** ddieterly is now known as ddieterly[away] | 16:05 | |
| odyssey4me | AJaeger pabelanger I have an optimisation for the jobs in the queue and I'm not entirely sure if my regex is right. https://review.openstack.org/331574 | 16:06 |
| AJaeger | odyssey4me: those skip-ifs are always a good way to shoot in your own foot ;( Let me check... | 16:06 |
| *** ddieterly[away] is now known as ddieterly | 16:06 | |
| *** yamamoto has joined #openstack-infra | 16:07 | |
| AJaeger | odyssey4me: you have to check the layout job for details, like http://logs.openstack.org/74/331574/1/check/gate-project-config-layout/e27006b/ | 16:07 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Do timeout calculation in jinja https://review.openstack.org/331686 | 16:07 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add debug task before zuul_runner https://review.openstack.org/331355 | 16:07 |
| jeblair | mordred, pabelanger, Shrews: ^ | 16:07 |
| *** ociuhandu has quit IRC | 16:08 | |
| jeblair | mordred, pabelanger, Shrews: http://paste.openstack.org/show/520623/ | 16:08 |
| *** eezhova has quit IRC | 16:08 | |
| pabelanger | jeblair: looks good! | 16:08 |
| jeblair | it's a little bit of a white lie, since actually the timeout could be just a little bit less than what the debug message says | 16:09 |
| pabelanger | ya | 16:09 |
| mordred | jeblair: I think we have two | int 's in there now that we don't need any more | 16:09 |
| jeblair | (i don't suppose we could set a variable and then use it both in the debug statement and in the async?) | 16:09 |
| mordred | jeblair: do you think we shoudl fix them now or just do a follow up? | 16:09 |
| jeblair | mordred: "timeout | int" is still needed because it's a string | 16:09 |
| mordred | AH | 16:10 |
| mordred | sweet. thanks | 16:10 |
| jeblair | np. ansible told me that when i tried to remove them. :) | 16:10 |
| *** eezhova has joined #openstack-infra | 16:10 | |
| *** Sukhdev_ has quit IRC | 16:10 | |
| anteaya | how helpful | 16:10 |
| AJaeger | odyssey4me: Looking at the layout file, I don't think your regex is correct. Best to check it yourself locally... | 16:11 |
| AJaeger | odyssey4me: http://logs.openstack.org/74/331574/1/check/gate-project-config-layout/e27006b/console.html#_2016-06-20_14_11_48_188213 shows that the releasenotes job has the skip if as well... | 16:11 |
| *** jyuso1 has quit IRC | 16:12 | |
| *** jyuso1 has joined #openstack-infra | 16:13 | |
| pabelanger | jeblair: Actually, why don't we use the name field in the zuul_runner task to display our timeout? Not sure why I didn't do that first time around. | 16:13 |
| pabelanger | that reduces the need for a debug task | 16:13 |
| *** daemontool has joined #openstack-infra | 16:14 | |
| *** vhosakot has quit IRC | 16:15 | |
| jeblair | pabelanger: oh, like "name: async task with timeout {{timeout math}}" ? | 16:15 |
| *** sree has joined #openstack-infra | 16:15 | |
| pabelanger | ya | 16:15 |
| jeblair | pabelanger: cool, i'll try that | 16:15 |
| *** sarob has quit IRC | 16:15 | |
| openstackgerrit | Hisashi Osanai proposed openstack-infra/project-config: Fix Monasca Analytics python27 job https://review.openstack.org/331535 | 16:15 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Only request desired timeline events https://review.openstack.org/331646 | 16:15 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Add quick non-permanent filter buttons for timeline events https://review.openstack.org/331748 | 16:15 |
| *** hosanai has quit IRC | 16:15 | |
| *** sarob has joined #openstack-infra | 16:16 | |
| *** amitgandhinz has quit IRC | 16:16 | |
| *** Sukhdev_ has joined #openstack-infra | 16:17 | |
| *** derekh has quit IRC | 16:17 | |
| *** amitgandhinz has joined #openstack-infra | 16:17 | |
| mordred | pabelanger: ++ | 16:17 |
| *** vhosakot has joined #openstack-infra | 16:17 | |
| *** roaet has left #openstack-infra | 16:17 | |
| *** Nikolay_St has joined #openstack-infra | 16:18 | |
| odyssey4me | AJaeger so it's interfering with https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L2862-L2872 ? | 16:18 |
| odyssey4me | or those are interfering with the change? | 16:18 |
| jeblair | 2016-06-20 09:19:00,502 DEBUG zuul.NodeWorker.test: Ansible output: TASK [zuul_runner with 588 second timeout] ************************************* | 16:19 |
| jeblair | mordred, pabelanger: ^ | 16:19 |
| pabelanger | nice! | 16:19 |
| *** sree has quit IRC | 16:19 | |
| odyssey4me | AJaeger it'd be really nice to have some sort of regex checker which I can test against a known subset of jobs | 16:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Include timeout in name of zuul_runner tasks https://review.openstack.org/331355 | 16:20 |
| jeblair | pabelanger: ^ i updated your change to do that | 16:20 |
| mtreinish | pabelanger: any idea why this still can't pass the beaker tests: https://review.openstack.org/#/c/328467/ | 16:20 |
| odyssey4me | AJaeger the job result in that log makes my brain explode. It's not pretty. | 16:20 |
| mtreinish | it doesn't really look related to my patch | 16:20 |
| odyssey4me | AJaeger Maybe it's just safer if I add another rule which is specific to the other repositories? | 16:21 |
| pabelanger | jeblair: +2 better me thinks | 16:21 |
| *** inc0 has joined #openstack-infra | 16:22 | |
| jeblair | okay, both of those approved. i'll start a graceful shutdown of several zuul launchers in prep for that. | 16:22 |
| *** Apoorva has joined #openstack-infra | 16:22 | |
| pabelanger | mtreinish: I am not sure, one looks like an issue with apache2 and not sure about the other | 16:24 |
| pabelanger | jeblair: ack | 16:25 |
| anteaya | mtreinish: I have a failing beaker test as well I am looking at: https://review.openstack.org/#/c/330922/ | 16:26 |
| *** ccamacho is now known as ccamacho|away | 16:26 | |
| jeblair | pabelanger, mordred: i put zl01, zl02 and zl03 in graceful shutdown. | 16:26 |
| *** amitgandhinz has quit IRC | 16:26 | |
| jeblair | we'll see how the others react to that | 16:26 |
| *** amitgandhinz has joined #openstack-infra | 16:27 | |
| Shrews | jeblair: btw, the Used Nodes graph is pretty awesome these days | 16:28 |
| anteaya | mtreinish: there is the apache2 error: http://logs.openstack.org/22/330922/4/check/gate-openstackci-beaker-ubuntu-trusty/e8c1ccc/console.html#_2016-06-20_16_12_04_822734 | 16:28 |
| *** dprince has joined #openstack-infra | 16:28 | |
| *** nwkarsten has quit IRC | 16:28 | |
| jeblair | Shrews: yeah, that hack worked surprisingly well didn't it? | 16:28 |
| *** claudiub|2 has joined #openstack-infra | 16:28 | |
| Shrews | jeblair: yeah | 16:28 |
| anteaya | EmilienM: ^^ any idea why apache2 is failing to be installed on the beaker tests? | 16:28 |
| pabelanger | zl05 looks eager today. About 25 more workers then other launchers | 16:28 |
| *** nwkarsten has joined #openstack-infra | 16:28 | |
| *** nwkarsten has quit IRC | 16:29 | |
| *** claudiub has quit IRC | 16:29 | |
| EmilienM | anteaya: it could be related to why all our beaker jobs are broken in Puppet OpenStack Ci | 16:29 |
| EmilienM | chem reported it here: https://github.com/mizzy/specinfra/pull/558 | 16:29 |
| anteaya | EmilienM: ah, yeah that could be it | 16:29 |
| *** nwkarsten has joined #openstack-infra | 16:29 | |
| anteaya | EmilienM: thanks | 16:29 |
| EmilienM | we currently pin specinfra https://review.openstack.org/#/c/331717/ | 16:30 |
| anteaya | mtreinish: ^^ | 16:30 |
| anteaya | EmilienM: okay thanks, cup of tea, let that propogate out then recheck, thank you | 16:31 |
| pleia2 | jeblair, fungi - alas, looks like it's been many years (and maybe an ircd change?) since I last looked into it, seems we're stuck with our 120 channel limit for openstackgerrit | 16:31 |
| mtreinish | anteaya: so we need to mirror that change for the infra beaker jobs? | 16:31 |
| mtreinish | oh, nm it applies to all the beaker jobs | 16:32 |
| anteaya | EmilienM: would that same issue be affecting puppet apply jobs as well? | 16:32 |
| anteaya | my puppet-apply job is missing apache2 | 16:33 |
| EmilienM | anteaya: not related | 16:33 |
| anteaya | huh, okay thank you | 16:33 |
| EmilienM | anteaya: not sure you'll hit the same beaker failures as we have | 16:33 |
| * anteaya goes back to the log | 16:33 | |
| *** yolanda has joined #openstack-infra | 16:33 | |
| EmilienM | anteaya: I'll look your failure after lunch, maybe I can help | 16:33 |
| anteaya | EmilienM: okay | 16:33 |
| *** sarob has quit IRC | 16:34 | |
| anteaya | EmilienM: oh awesome thank you, enjoy lunch, I'll try to find useful bits of the log | 16:34 |
| *** nwkarsten has quit IRC | 16:34 | |
| *** sarob has joined #openstack-infra | 16:34 | |
| *** julim has quit IRC | 16:35 | |
| *** mohankumar has quit IRC | 16:35 | |
| anteaya | This error "Could not retrieve information from environment production source(s) file:/opt/ansible/contrib/inventory/openstack.py" looks like an ansible inventory issue, yes? | 16:36 |
| anteaya | from http://logs.openstack.org/22/330922/4/check/gate-infra-puppet-apply-ubuntu-trusty/af691c8/console.html#_2016-06-20_16_07_36_666650 | 16:37 |
| *** yaume_ has quit IRC | 16:37 | |
| *** nwkarsten has joined #openstack-infra | 16:37 | |
| *** julim has joined #openstack-infra | 16:38 | |
| *** ayoung has joined #openstack-infra | 16:38 | |
| *** jpich has quit IRC | 16:39 | |
| AJaeger | odyssey4me: I just think your regex is broken... | 16:39 |
| *** e0ne has joined #openstack-infra | 16:39 | |
| AJaeger | odyssey4me: write a short python program to test it | 16:39 |
| *** jlanoux has quit IRC | 16:40 | |
| anteaya | this is the first error after it tried to find the afs mirror: http://logs.openstack.org/22/330922/4/check/gate-infra-puppet-apply-ubuntu-trusty/af691c8/console.html#_2016-06-20_16_05_26_266761 | 16:42 |
| odyssey4me | AJaeger it seems that my corporate bastion is not allowing me any ssh access any more so I have other problems :/ let me see if I can work it out tomorrow - thanks for looking | 16:42 |
| *** nwkarsten has quit IRC | 16:42 | |
| anteaya | is there anyway for me to see the current status of the afs mirror? | 16:42 |
| openstackgerrit | Zara proposed openstack-infra/storyboard: Remove duplicated line https://review.openstack.org/331762 | 16:42 |
| anteaya | odyssey4me: :( | 16:43 |
| AJaeger | odyssey4me: http://paste.openstack.org/show/520640/ | 16:43 |
| *** sree has joined #openstack-infra | 16:43 | |
| AJaeger | run it and see results | 16:43 |
| *** ihrachys has quit IRC | 16:44 | |
| mordred | odyssey4me: that souds TERRIBLE | 16:45 |
| *** tlian2 has quit IRC | 16:45 | |
| pabelanger | anteaya: your first error you linked, /opt/ansible/contrib/inventory/openstack.py is related to our inventory script on puppet master however specific to the puppet-apply test, it is not considered a failure. If you look 3 lines below that error, you can see exit 0, which represents the applytest for puppetmaster.openstack.org actually passed | 16:47 |
| *** javeriak has joined #openstack-infra | 16:47 | |
| *** sree_ has joined #openstack-infra | 16:47 | |
| *** e0ne has quit IRC | 16:47 | |
| anteaya | pabelanger: thank you, I appreciate the detailed reply, thanks for helping me learn | 16:47 |
| *** harlowja_at_home has quit IRC | 16:47 | |
| *** sree_ is now known as Guest35134 | 16:47 | |
| *** sree has quit IRC | 16:48 | |
| pabelanger | anteaya: same applies to your 2nd question, while it is generating ERRORs, applytest does not actually fail. This is basically because of the way we run our puppet applytesting. | 16:48 |
| anteaya | okay thank you | 16:49 |
| anteaya | pabelanger: do you know what [Zuul] Task exit code: 123 means? | 16:49 |
| pabelanger | anteaya: to find the error in question, you can search or grep for 'exit 1', which in the log you posted is the actually reason for the failure | 16:49 |
| anteaya | if the puppet apply test was actually happy despite emitting all those warns, given as you say the exit 0 at the end | 16:49 |
| anteaya | pabelanger: ah ha, thank you http://logs.openstack.org/22/330922/4/check/gate-infra-puppet-apply-ubuntu-trusty/af691c8/console.html#_2016-06-20_16_05_50_237131 that is helpful | 16:50 |
| pabelanger | anteaya: I am not sure, I would have to see why zuul-launcher is displaying that. If I was to guess, it would see that our puppet apply test is actually return 123 as the error code. But I do not know wy | 16:50 |
| pabelanger | anteaya: Yup, that will point you to the problem if your patch. | 16:51 |
| pabelanger | of* | 16:51 |
| *** Guest35134 has quit IRC | 16:51 | |
| *** ddieterly is now known as ddieterly[away] | 16:52 | |
| *** ddieterly[away] is now known as ddieterly | 16:52 | |
| fungi | pleia2: oh, so that's a global max now and not configurable per user (or they're not willing to manage per-user deviations from the default so effectively the same end result)? | 16:52 |
| *** nwkarsten has joined #openstack-infra | 16:52 | |
| anteaya | pabelanger: thank you, I appreciate learning how to parse puppet apply logs | 16:52 |
| anteaya | I'm grateful | 16:52 |
| pabelanger | anteaya: np | 16:53 |
| pleia2 | fungi: they didn't offer up rationale, just "run another bot" | 16:53 |
| pleia2 | so I'm not sure | 16:53 |
| *** rcernin has joined #openstack-infra | 16:53 | |
| *** abregman has joined #openstack-infra | 16:53 | |
| AJaeger | should we just add a check to our gates that we not have more than 120 channels setup - so that we cannot merge more? | 16:54 |
| *** ddieterly is now known as ddieterly[away] | 16:54 | |
| *** andreww is now known as xarses | 16:56 | |
| *** bhavik has joined #openstack-infra | 16:56 | |
| anteaya | AJaeger: with an invitation that anyone interested can help us set up another gerritbot perhaps? | 16:56 |
| anteaya | pleia2: thanks for asking | 16:57 |
| fungi | pleia2: AJaeger: yeah, i guess we need to either scale back gerritbot use or shard across multiple accounts, and a check to avoid overrunning the chanmax limit would be good so we don't get caught off guard by it again | 16:57 |
| fungi | anteaya: ^ | 16:57 |
| anteaya | fungi: sounds good | 16:57 |
| *** inc0 has quit IRC | 16:58 | |
| * notmorgan wipes sleep from eyes and tries to focus on things. | 16:58 | |
| *** sree has joined #openstack-infra | 16:59 | |
| anteaya | would someone who knows puppet well be willing to review the following two patches to correct the test failures I found in the logs and pasted in an inline comment beside the lines the log said were making it uphappy? https://review.openstack.org/#/c/330922/4 and https://review.openstack.org/#/c/330925/4 | 16:59 |
| * pleia2 passes coffee to notmorgan | 16:59 | |
| *** tlian has joined #openstack-infra | 16:59 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Check number of channels for gerritbot https://review.openstack.org/331766 | 16:59 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Test gerritbot limit https://review.openstack.org/331767 | 16:59 |
| AJaeger | anteaya, pleia2, fungi ^ | 16:59 |
| anteaya | it is to get the storyboard plugin installed and configured on review-dev, something I would really be most appreciative if it could happen by wednesday | 16:59 |
| * AJaeger will test now... | 17:00 | |
| *** kzaitsev_mb has quit IRC | 17:00 | |
| anteaya | thanks | 17:00 |
| * notmorgan drinks coffee and thanks pleia2 for saving the lives of at least 5 people who aren't buffered by the general calm of the IRC interface :) | 17:00 | |
| AJaeger | change is wrong, let me do it properly... | 17:00 |
| *** haaj has joined #openstack-infra | 17:01 | |
| anteaya | AJaeger: of course | 17:03 |
| * anteaya lets AJaeger do it properly | 17:03 | |
| *** sree has quit IRC | 17:03 | |
| pleia2 | notmorgan: hah :) | 17:03 |
| *** ddieterly[away] is now known as ddieterly | 17:03 | |
| *** kzaitsev_mb has joined #openstack-infra | 17:04 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Test gerritbot limit https://review.openstack.org/331767 | 17:04 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Check number of channels for gerritbot https://review.openstack.org/331766 | 17:04 |
| AJaeger | fixed, fails now only the test change ^ | 17:04 |
| *** claudiub has joined #openstack-infra | 17:04 | |
| anteaya | AJaeger: nice | 17:04 |
| *** Nikolay_St has quit IRC | 17:05 | |
| AJaeger | anteaya: another revision with your revised text is coming... | 17:05 |
| *** raunak has joined #openstack-infra | 17:05 | |
| anteaya | AJaeger: okay thank you | 17:05 |
| *** jaosorior has quit IRC | 17:05 | |
| anteaya | what you had is good too, just a little more towards action in case we get a taker | 17:05 |
| *** claudiub|2 has quit IRC | 17:06 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Test gerritbot limit https://review.openstack.org/331767 | 17:06 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Check number of channels for gerritbot https://review.openstack.org/331766 | 17:06 |
| AJaeger | anteaya: and that's worth changing it ;) | 17:06 |
| *** eezhova has quit IRC | 17:06 | |
| anteaya | thank you | 17:07 |
| anteaya | :) | 17:07 |
| *** tkelsey has quit IRC | 17:07 | |
| anteaya | I'll let your test patch fail nicely | 17:08 |
| anteaya | AJaeger: I notice a newline in your patch, is that needed? | 17:08 |
| AJaeger | looked nicer to me in the output... | 17:08 |
| anteaya | okay great | 17:08 |
| *** sree has joined #openstack-infra | 17:09 | |
| anteaya | AJaeger: oh I mean line 32 | 17:09 |
| anteaya | line 32 is intentional? | 17:09 |
| AJaeger | no ;( | 17:09 |
| *** bhavik1 has joined #openstack-infra | 17:09 | |
| *** bhavik has quit IRC | 17:09 | |
| anteaya | ah | 17:09 |
| AJaeger | new version coming... | 17:10 |
| anteaya | thanks | 17:10 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Test gerritbot limit https://review.openstack.org/331767 | 17:10 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Check number of channels for gerritbot https://review.openstack.org/331766 | 17:10 |
| pabelanger | AJaeger: anteaya: left a suggestion on the patch | 17:10 |
| anteaya | pabelanger: thanks | 17:10 |
| AJaeger | thanks... | 17:10 |
| *** PollyZ has quit IRC | 17:11 | |
| *** maishsk has quit IRC | 17:12 | |
| *** bhavik has joined #openstack-infra | 17:12 | |
| openstackgerrit | Merged openstack-infra/zuul: Do timeout calculation in jinja https://review.openstack.org/331686 | 17:12 |
| AJaeger | thanks, pabelanger | 17:12 |
| anteaya | pabelanger: is the reason we have no output for the tempest job runtimes in this view intentional? http://grafana.openstack.org/dashboard/db/nodepool | 17:12 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Test gerritbot limit https://review.openstack.org/331767 | 17:12 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Check number of channels for gerritbot https://review.openstack.org/331766 | 17:12 |
| EmilienM | anteaya: http://logs.openstack.org/22/330922/4/check/gate-openstackci-beaker-ubuntu-trusty/e8c1ccc/console.html#_2016-06-20_16_12_05_001684 | 17:12 |
| EmilienM | anteaya: all your beaker jobs are broken too | 17:12 |
| EmilienM | you'll need to pin specinfra, like we do in our modules: https://review.openstack.org/#/c/331717/ | 17:13 |
| anteaya | EmilienM: awesome, I feel so included | 17:13 |
| EmilienM | nibalizer: ^ | 17:13 |
| *** sree has quit IRC | 17:13 | |
| *** dizquierdo has joined #openstack-infra | 17:13 | |
| *** bhavik1 has quit IRC | 17:13 | |
| EmilienM | Sofer reported the issue here: https://github.com/mizzy/specinfra/pull/558 | 17:13 |
| *** PollyZ has joined #openstack-infra | 17:14 | |
| anteaya | EmilienM: any idea what repo I need to offer that pin against? | 17:14 |
| openstackgerrit | Merged openstack-infra/zuul: Include timeout in name of zuul_runner tasks https://review.openstack.org/331355 | 17:14 |
| EmilienM | anteaya: see https://review.openstack.org/#/c/331717/ | 17:14 |
| anteaya | yeah I see it | 17:14 |
| *** amitgandhinz has quit IRC | 17:14 | |
| nibalizer | EmilienM: awesome | 17:14 |
| anteaya | you are telling me I need to pin something and I am only too happy to | 17:14 |
| anteaya | I don't know where to put the pin though | 17:14 |
| EmilienM | in Gemfile | 17:15 |
| EmilienM | of all our modules | 17:15 |
| nibalizer | yep | 17:15 |
| anteaya | oh, oh my | 17:15 |
| *** amitgandhinz has joined #openstack-infra | 17:15 | |
| nibalizer | yeaaaaa | 17:15 |
| EmilienM | we don't have to do it in our modules | 17:15 |
| EmilienM | because we have a Gem for it: | 17:15 |
| EmilienM | https://github.com/openstack/puppet-nova/blob/master/Gemfile#L4-L6 | 17:15 |
| pabelanger | anteaya: I am not sure, if you zoom out to a 7day view you can see the updates have stopped for some reason. I currently do not know why that is | 17:15 |
| nibalizer | i wonder if today is the day we create our own gem | 17:15 |
| EmilienM | and we manage deps here: https://github.com/openstack/puppet-openstack_spec_helper/blob/master/puppet-openstack_spec_helper.gemspec#L20-L43 | 17:15 |
| nibalizer | yep | 17:15 |
| nibalizer | i think today is that day | 17:15 |
| EmilienM | lol | 17:16 |
| EmilienM | it is :) | 17:16 |
| nibalizer | crinkle: thoughts? | 17:16 |
| EmilienM | nibalizer: puppet-openstack-infra_spec_helper or something similar | 17:16 |
| EmilienM | maybe we can even share bits !!! | 17:16 |
| *** hosanai has joined #openstack-infra | 17:16 | |
| nibalizer | its also concievabe that we just use yours | 17:17 |
| nibalizer | though that creates a kind of hard linking we may not want | 17:17 |
| EmilienM | I would start by creating your own gem | 17:17 |
| jeblair | EmilienM: have you seen a lot of jobs that were marked as failures but appear to have worked when you check the console log? | 17:17 |
| EmilienM | and see if we actually can share one | 17:17 |
| anteaya | pabelanger: okay thank you | 17:17 |
| EmilienM | jeblair: not today afik | 17:17 |
| EmilienM | nibalizer: can you +a https://review.openstack.org/#/c/331674/ please please please | 17:18 |
| *** rcernin has quit IRC | 17:18 | |
| anteaya | EmilienM: nibalizer crinkle I have no opinion on what is to be done, but I am following along so the patch to puppet-gerrit will pass tests, once the decision has been made I am happy to do any template following grunt work | 17:19 |
| anteaya | do keep me posted on a way forward | 17:20 |
| *** kmartin has joined #openstack-infra | 17:20 | |
| openstackgerrit | Merged openstack-infra/project-config: Run neutron-lbaas-dsvm-functional-nv job only on master https://review.openstack.org/330996 | 17:20 |
| crinkle | yay gem failures :( | 17:20 |
| openstackgerrit | Jimmy McCrory proposed openstack-infra/project-config: Add stable/mitaka branch for #openstack-ansible https://review.openstack.org/331775 | 17:20 |
| crinkle | nibalizer: yeah let's do it! | 17:21 |
| *** akshai has joined #openstack-infra | 17:21 | |
| jeblair | odyssey4me, EmilienM, pabelanger, mordred, Shrews: i put the full list of jobs that have failed with async timeouts in less than 60 minutes on the etherpad https://etherpad.openstack.org/p/zuul-timeouts | 17:21 |
| *** tkelsey has joined #openstack-infra | 17:21 | |
| nibalizer | crinkle: EmilienM ok sweet | 17:21 |
| mordred | jeblair: well that doesn't seem great | 17:21 |
| anteaya | crinkle: thanks for your review! | 17:21 |
| nibalizer | ill spin up a project config change to create the repo | 17:21 |
| jeblair | odyssey4me, EmilienM, pabelanger, mordred, Shrews: it has happened in the proportions i would expect on all 7 launchers, and i don't see a correlation among cloud providers | 17:21 |
| *** hosanai has quit IRC | 17:21 | |
| crinkle | anteaya: no problem | 17:22 |
| jeblair | odyssey4me, EmilienM, pabelanger, mordred, Shrews: i will say that it does seem to very heavily favor the puppet and ansible openstack projects | 17:22 |
| *** amitgandhinz has quit IRC | 17:22 | |
| crinkle | anteaya: i'm not sure what's up with 330925 though | 17:22 |
| EmilienM | jeblair: ack I'll let you know if I see it | 17:22 |
| anteaya | nibalizer: so is your new repo importing the repo EmilienM linked to above? | 17:22 |
| *** burgerk_ has quit IRC | 17:22 | |
| *** amitgandhinz has joined #openstack-infra | 17:22 | |
| crinkle | anteaya: i'm wondering if it didn't use the depends-on since the dependent change failed | 17:22 |
| mordred | jeblair: that makes me (unscientifically) think that something in those jobs is doing something that messes with the async system that ansible uses | 17:23 |
| anteaya | crinkle: thanks for looking at it, I'm grateful. If you don't know what is wrong with 330925 I'm not sure who can get that working | 17:23 |
| jeblair | mordred: i agree | 17:23 |
| nibalizer | anteaya: um im not sure we need to import the whole git history, we need to have our own gem list | 17:23 |
| anteaya | crinkle: oh, let's see if that is the issue | 17:23 |
| nibalizer | but our structure will be pretty similar to theirs | 17:23 |
| jeblair | mordred: maybe it's being oomed. or firewalls. or sshd config. | 17:23 |
| anteaya | nibalizer: ummm, okay will you be importing anything? | 17:23 |
| jeblair | mordred: so i do think the next step is to hold a node, and possibly even try to catch it in the act | 17:24 |
| crinkle | nibalizer: anteaya i don't think we want to use the same repo as them | 17:24 |
| anteaya | okay so an empty repo, no imported content? | 17:24 |
| crinkle | i think so | 17:24 |
| anteaya | if that is what you want can you say so clearly in the commit message please? | 17:25 |
| EmilienM | jeblair: Im looking | 17:25 |
| EmilienM | jeblair: we're consumming a lot of resources, our jobs run more than usual devstack jobs | 17:25 |
| EmilienM | we deploy all the things remember | 17:25 |
| EmilienM | jeblair: I don't see much puppet jobs | 17:26 |
| nibalizer | crinkle: agreed | 17:27 |
| pabelanger | jeblair: mordred: http://logs.openstack.org/71/331071/1/check/gate-openstack-ansible-lxc_hosts-linters/9d338a9/_zuul_ansible/logs/ansible.txt.gz has me concerned again any why ansible async_status is 404 | 17:28 |
| pabelanger | jeblair: mordred: that is an example log from section 2 of the etherpad | 17:29 |
| *** tqtran has joined #openstack-infra | 17:29 | |
| mordred | pabelanger: I was just working through reading ansible source to see what I can learn about how async polling works | 17:29 |
| anteaya | zaro: thanks for your work on the its-storyboard patches, currently everything is blocked on a ruby gem causing beaker tests to fail, in the meantime you could address colleen's comments here: https://review.openstack.org/#/c/330922/4 | 17:29 |
| *** sree has joined #openstack-infra | 17:29 | |
| *** ddieterly is now known as ddieterly[away] | 17:30 | |
| *** akshai_ has joined #openstack-infra | 17:30 | |
| mordred | pabelanger, jeblair: are the jobs changing the /home/jenkins permissions? | 17:31 |
| *** bhavik1 has joined #openstack-infra | 17:31 | |
| pabelanger | mordred: jeblair: https://github.com/ansible/ansible/issues/10855 seems to be related, reading that now | 17:31 |
| jeblair | pabelanger: interesting -- i could see that same error manifesting as "async didn't complete" depending on exactly when whatever happened happened | 17:31 |
| pabelanger | mordred: good question | 17:31 |
| *** tongli has quit IRC | 17:32 | |
| *** e0ne has joined #openstack-infra | 17:32 | |
| mordred | yah. because async status is done by checking that file | 17:32 |
| *** trown|mtg is now known as trown | 17:32 | |
| pabelanger | the PR seems to talk about keep_remote_files as a possible workaround | 17:32 |
| mordred | jeblair, pabelanger: keep_remote_files=true may be a workaround if that is, in fact, our problem | 17:32 |
| mordred | pabelanger: jinx | 17:32 |
| *** mhickey has quit IRC | 17:32 | |
| pabelanger | going to see what that is about | 17:32 |
| pabelanger | mordred: ya | 17:32 |
| notmorgan | pabelanger: ++ that seems to be the work around according to the issue | 17:33 |
| *** akshai has quit IRC | 17:33 | |
| *** abregman is now known as abregman|afk | 17:33 | |
| pabelanger | ya, I don't know why atm | 17:33 |
| *** sree has quit IRC | 17:34 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update Robot plugin to use convert xml https://review.openstack.org/331778 | 17:35 |
| notmorgan | pabelanger: there are a few people saying that workaround doesn't solve it | 17:35 |
| *** bhavik has quit IRC | 17:35 | |
| *** bhavik1 is now known as bhavik | 17:35 | |
| *** dprince has quit IRC | 17:35 | |
| openstackgerrit | Merged openstack-infra/project-config: zuul/layout: fix rule for beaker-xenial jobs https://review.openstack.org/331674 | 17:35 |
| *** dprince has joined #openstack-infra | 17:35 | |
| *** markusry has quit IRC | 17:35 | |
| notmorgan | pabelanger: it seems that async_wrapper [based on this issue] is either deleting the files too fast *or* acessing the files before they are written | 17:35 |
| *** markusry has joined #openstack-infra | 17:35 | |
| notmorgan | pabelanger: this sounds like async_wrapper has some serious issues in general. | 17:36 |
| *** kzaitsev_mb has quit IRC | 17:36 | |
| * mordred is going to ask bcoca over in ansible land | 17:36 | |
| pabelanger | The PR seem to revolve around keep_remote_files and remote_tmp. Either enabling keep_remote_files and / or setting remote_tmp to /tmp | 17:37 |
| *** sarob has quit IRC | 17:37 | |
| pabelanger | so, my only guess, is the cleanup handler in ansible is to aggressive on the node and deleting our async_wrapper files | 17:38 |
| pabelanger | but, that is a complete guess | 17:38 |
| *** markusry has quit IRC | 17:38 | |
| mordred | pabelanger: that's my guess too - I have asked in #ansible-devel fwiw | 17:38 |
| *** _nadya_ has quit IRC | 17:38 | |
| jeblair | pabelanger: my read is the remote_tmp doesn't solve the issue on its own. however, if you want to set keep_remote_files true, then some people also set remote_tmp to /tmp so that their OS tmp cleaning cron would eventually delete old ansible tmp files. | 17:38 |
| *** _nadya_ has joined #openstack-infra | 17:39 | |
| *** yamamoto has quit IRC | 17:39 | |
| pabelanger | jeblair: agreed. In our case, I don't think remote_tmp is needed | 17:39 |
| jeblair | that's not a concern on ephemeral nodes, however, it may be on our long-lived ones | 17:39 |
| *** sree has joined #openstack-infra | 17:39 | |
| pabelanger | ya | 17:39 |
| mordred | however, if it's a race, I'm not sure that explains a task aborting in the middle | 17:39 |
| mordred | nor why it would be disproportionately high with ansible/puppet jobs | 17:40 |
| pabelanger | I am open to trying the setting, since billyfoss (OP) in the issue does report success with keep_remote_files | 17:40 |
| jeblair | mordred: it may not be in the middle -- it looks like it may be happening near the end... | 17:40 |
| *** e0ne has quit IRC | 17:40 | |
| *** Sukhdev_ has quit IRC | 17:40 | |
| nibalizer | crinkle: EmilienM i think for now we run the unit tests | 17:41 |
| nibalizer | and leave beaker for later | 17:41 |
| nibalizer | probably something something openstackci beaker | 17:41 |
| *** slaweq has joined #openstack-infra | 17:41 | |
| mordred | jeblair: same concern with why does it happen more on puppet/ansible jobs - but I think trying keep_remote_files is fairly non-insane thing to try | 17:41 |
| jeblair | mordred: like maybe the race is "task finishes; async_wrapper cleans up; async polls". | 17:41 |
| *** PollyZ has quit IRC | 17:41 | |
| mordred | jeblair: yup. I could totally understand that being a race | 17:41 |
| *** sree_ has joined #openstack-infra | 17:42 | |
| crinkle | nibalizer: okay | 17:42 |
| jeblair | mordred: oh yeah, i am also concerned by that :) | 17:42 |
| *** sree_ is now known as Guest59953 | 17:42 | |
| *** slaweq has quit IRC | 17:43 | |
| anteaya | nibalizer: are you suggesting the beaker tests become non-voting for a time? | 17:43 |
| fungi | though that issue pabelanger linked also includes some speculation that it may be trying to run before the file gets into place | 17:43 |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: Add configurations for gerrit its plugins https://review.openstack.org/330922 | 17:43 |
| fungi | but yeah, wouldn't explain why it's more often on puppet/ansible jobs | 17:44 |
| *** slaweq has joined #openstack-infra | 17:44 | |
| *** ccamacho|away is now known as ccamacho | 17:44 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/project-config: Add infra puppet gem dependency holder repo https://review.openstack.org/331780 | 17:44 |
| *** sree has quit IRC | 17:44 | |
| nibalizer | anteaya: no | 17:44 |
| anteaya | crinkle: should erb files contain tabs? | 17:44 |
| anteaya | nibalizer: okay thank you | 17:44 |
| nibalizer | the puppet modules' system has a beaker test and we wont worry about running that at first | 17:44 |
| nibalizer | anteaya: erb files should be tabed/spaced based on the standard of the file that is being templated | 17:45 |
| *** esberglu has quit IRC | 17:45 | |
| jeblair | fungi, mordred, pabelanger: i'm going to work on trying to catch a node | 17:45 |
| anteaya | nibalizer: thank you | 17:45 |
| pabelanger | jeblair: ack | 17:46 |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: Add infra puppet gem dependency holder repo https://review.openstack.org/331780 | 17:46 |
| *** Guest59953 has quit IRC | 17:47 | |
| anteaya | nibalizer: edited your commit message to state the no import is intentional and added a topic | 17:47 |
| nibalizer | ok cool | 17:47 |
| mordred | jeblair: k. bcoca is looking at the async_helper code | 17:47 |
| mordred | pabelanger: why is http://logs.openstack.org/71/331071/1/check/gate-openstack-ansible-lxc_hosts-linters/9d338a9/console.html.gz a timeout? | 17:48 |
| HeOS | Hello, infra-team, I'd like to ask you to review the following request: https://review.openstack.org/#/c/284843/. That request was created more than 9 weeks ago, but it has no scores yet. :( | 17:48 |
| mordred | pabelanger: nevermind | 17:49 |
| pabelanger | mordred: I don't believe it is. | 17:49 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Configure its-storyboard plugin https://review.openstack.org/330925 | 17:50 |
| AJaeger | nibalizer, anteaya : We need a governance change for the new repo - together with needed-By | 17:51 |
| *** ramishra has quit IRC | 17:51 | |
| pabelanger | mordred: looking at the source, I actually think ansible is failing to copy our async_status file to the remote node. Specifically put_file in lib/ansible/plugins/connection/ssh.py | 17:52 |
| nibalizer | what does needed-by do? | 17:52 |
| pabelanger | mordred: which would then mean, some possible issue related to our SSH connection | 17:52 |
| AJaeger | nibalizer: just document the dependency | 17:53 |
| *** rcernin has joined #openstack-infra | 17:53 | |
| AJaeger | nibalizer: http://docs.openstack.org/infra/manual/creators.html#add-new-repository-to-the-governance-repository for the change | 17:53 |
| anteaya | AJaeger: yup | 17:53 |
| AJaeger | nibalizer: depends-on is for the one direction and needed-by the other direction - just as documentation. | 17:54 |
| *** piet_ has joined #openstack-infra | 17:54 | |
| anteaya | nibalizer: it helps us review, so we can find all the bits | 17:54 |
| pabelanger | mordred: http://docs.ansible.com/ansible/intro_configuration.html#scp-if-ssh seems to control how we PUT files onto a remote worker | 17:54 |
| mordred | pabelanger: yah. or some other thing that changes that disallows creation of files | 17:54 |
| *** tosky has quit IRC | 17:55 | |
| mordred | pabelanger: might be worthwhile setting that just to see? | 17:55 |
| pabelanger | mordred: right, which loops back to your chmod / chown /home/jenkins theory | 17:55 |
| nibalizer | okay i created the governance change too | 17:55 |
| pabelanger | or moving remote_tmp to some other folder | 17:55 |
| nibalizer | https://review.openstack.org/331787 | 17:55 |
| mordred | pabelanger, jeblair: in general, perhaps creating our own tmp location and setting remote_tmp isn't a bad idea anyway | 17:56 |
| jeblair | mordred: hrm | 17:56 |
| *** ramishra has joined #openstack-infra | 17:56 | |
| mordred | I'd love of jeblair succeeds in holding anode | 17:57 |
| anteaya | nibalizer: thanks | 17:57 |
| *** esberglu has joined #openstack-infra | 17:57 | |
| nibalizer | mordred: ill help by holding a cathode | 17:57 |
| nibalizer | anteaya: so none of the other infra modules have a gerrit acl specified | 17:57 |
| nibalizer | is that because they have a projectname acl? | 17:57 |
| jeblair | mordred: part of me says, sure, keep it out of the jenkins dir to stay away from things a test might touch. but then, where on the system would be better? perhaps ~jenkins really is the best place? | 17:57 |
| AJaeger | nibalizer: We first merge the project-config change, the nthe governance. | 17:57 |
| AJaeger | so, project-config has "needed-by" and governance has "depends-on" | 17:58 |
| jeblair | mordred: i'm logged into 63.251.114.160 which is running gate-openstack-ansible-os_heat-ansible-func-ubuntu-trusty | 17:58 |
| *** sree has joined #openstack-infra | 17:58 | |
| anteaya | nibalizer: oh look at that | 17:58 |
| AJaeger | nibalizer: So, the PTL +1/+2 on the project-config is an explicit +1 on the governance - this is how it's documented for repos to add to existing official team | 17:58 |
| anteaya | nibalizer: must have, there must be an overarching gerrit acl rule somewhere that has permissions for everything in openstack-infra going to the infra team | 17:59 |
| anteaya | nibalizer: ignore me then | 17:59 |
| *** inc0 has joined #openstack-infra | 18:00 | |
| mordred | ooooooo | 18:00 |
| mordred | jeblair: in ~jenkins, do this: ls -ltra .ansible/tmp/ansible-tmp-1466445635.76-239152944675193 | 18:01 |
| mordred | jeblair: nevermind | 18:01 |
| Shrews | so... those timeouts... | 18:01 |
| mordred | jeblair: it was just going quicker than me | 18:01 |
| Shrews | are short | 18:01 |
| openstackgerrit | Spencer Krum proposed openstack-infra/project-config: Add infra puppet gem dependency holder repo https://review.openstack.org/331780 | 18:01 |
| Shrews | http://logs.openstack.org/71/331071/1/check/gate-openstack-ansible-lxc_hosts-ansible-func-ubuntu-trusty/5648d7e/_zuul_ansible/logs/ansible.txt | 18:01 |
| Shrews | fails 1 minute almost exactly into the playbook | 18:02 |
| *** ifarkas has quit IRC | 18:02 | |
| *** ifarkas has joined #openstack-infra | 18:02 | |
| *** sree has quit IRC | 18:02 | |
| anteaya | AJaeger: where in the logs do I find your helpful message on the irc access job failure? https://review.openstack.org/#/c/331767/5 | 18:02 |
| jeblair | mordred: from what i'm seeing, .ansible/tmp/ is usually empty, and every few seconds, ansible runs a task to check on it. that appears to happen in the normal way tasks are run -- an ansible module is uploaded to that location and executed, and then removed | 18:02 |
| nibalizer | anteaya: i think its just that it implicitly gives <reponame> acls from <reponame>.config | 18:02 |
| jeblair | mordred: so when i watch that directory, i see a file like that appear and disappear quickly | 18:03 |
| mordred | jeblair: yah | 18:03 |
| mordred | jeblair: I agree with observing this behavior | 18:03 |
| nibalizer | AJaeger: so you want me to reverse the dependency? | 18:03 |
| anteaya | nibalizer: yes it usually does in which case you would need a reponame.config patch with the patchset | 18:03 |
| Shrews | i see one failing at 3 minutes | 18:03 |
| anteaya | nibalizer: that is what I normally look for, hence my comment | 18:03 |
| *** HeOS has quit IRC | 18:04 | |
| anteaya | nibalizer: yeah there they all are: http://git.openstack.org/cgit/openstack-infra/project-config/tree/gerrit/acls/openstack-infra/puppet-accessbot.config | 18:04 |
| nibalizer | anteaya: ya i updated with a config | 18:04 |
| anteaya | nibalizer: awesome, thanks | 18:05 |
| Shrews | jeblair: mordred: due to the short timeouts, i'm beginning to suspect the plugin | 18:05 |
| mordred | jeblair, Shrews: is there any chance that somewhere we're doing a seconds/minutes exchange improperly? | 18:05 |
| AJaeger | nibalizer: Yes - as documented in the infra manual section that I referenced. | 18:05 |
| zaro | waynr: just released https://pypi.python.org/pypi/python-jenkins/0.4.13 | 18:05 |
| AJaeger | nibalizer: Or do you like to wait 7 days to get your repo approved? | 18:05 |
| Shrews | mordred: possibly. looking now | 18:05 |
| AJaeger | pabelanger, anteaya: my gerritbot change passed the testsuite - do you want to review now? | 18:06 |
| anteaya | nibalizer: if you spin again, please state in the commit message you are creating an empty repo intentionally | 18:06 |
| *** daemontool has quit IRC | 18:06 | |
| mordred | I mean, we set timeouts all over the place, so I' expect way more failures if it was just a straigh multiply-by-60 issue | 18:06 |
| openstackgerrit | Spencer Krum proposed openstack-infra/project-config: Add infra puppet gem dependency holder repo https://review.openstack.org/331780 | 18:06 |
| anteaya | AJaeger: yeah I just asked a question about that | 18:06 |
| zaro | electrofelix: do you have an account on RTD? | 18:06 |
| anteaya | AJaeger: how do I see the message your test leaves in the logs | 18:06 |
| anteaya | AJaeger: I don't know the zuul log tree well enough yet to find it | 18:07 |
| AJaeger | anteaya: click on the *second change on the failing irc test | 18:07 |
| jeblair | mordred: i think the values are correct now based on our work we merged on friday (we'll have more confidence once we restart with the extra logging) | 18:07 |
| AJaeger | http://logs.openstack.org/67/331767/5/check/gate-project-config-irc-access/ec0ae46/console.html | 18:07 |
| jeblair | mordred: but if we were really doing 1/60th the time, every job would be legitimately timing out | 18:07 |
| *** maishsk has joined #openstack-infra | 18:07 | |
| mordred | yah | 18:07 |
| bknudson_ | JJB question -- Is there a way to "pass in" a value that would be used in a job definition? (for example, the auth-token to use for the slack publisher) | 18:08 |
| anteaya | AJaeger: yeah, I looked there | 18:08 |
| Shrews | jeblair: mordred: or, those jobs install ansible and configure it in some way that overrides our behavior? | 18:08 |
| *** rbrndt has quit IRC | 18:08 | |
| AJaeger | anteaya: failing line is http://logs.openstack.org/67/331767/5/check/gate-project-config-irc-access/ec0ae46/console.html#_2016-06-20_17_53_18_953772 | 18:08 |
| jeblair | Shrews: that's certainly on my mind with the openstack-ansible ones; EmilienM -- strang question: do the puppet jobs do anything with ansible? :) | 18:08 |
| zaro | bknudson_: i think you might be able to do that with the config file | 18:09 |
| *** jtomasek has quit IRC | 18:09 | |
| * AJaeger waves good bye for today... | 18:09 | |
| anteaya | AJaeger: ah thanks, shame it isn't red | 18:09 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/zuul: highlight the error condition when reporting a config issue https://review.openstack.org/272262 | 18:09 |
| anteaya | AJaeger: thank you | 18:09 |
| EmilienM | jeblair: not afik | 18:09 |
| openstackgerrit | Merged openstack-infra/storyboard: Remove duplicated line https://review.openstack.org/331762 | 18:10 |
| jeblair | mordred: that job completed without incident; want to see anything on the node or should i delete it? | 18:10 |
| mordred | jeblair: I need nothing | 18:10 |
| nibalizer | AJaeger: anteaya 331780 should be ready | 18:10 |
| anteaya | I'm looking | 18:11 |
| *** PollyZ has joined #openstack-infra | 18:11 | |
| anteaya | clarkb: I recall there was a thing when a repo used the noop-jobs template and had jobs, did the problem arise with the noop-jobs template and experimental jobs? | 18:13 |
| clarkb | anteaya: the problem is it puts long running jobs in the same queue as the noop jobs | 18:15 |
| *** jed56 has quit IRC | 18:15 | |
| clarkb | making all the noops wait in line when they should just return success | 18:15 |
| anteaya | ah so that is the problem, okay so in the case of experimental jos specify check: noop and gate: noop, yes? | 18:15 |
| anteaya | clarkb: I'm looking at: https://review.openstack.org/#/c/331780/4/zuul/layout.yaml | 18:16 |
| *** sree has joined #openstack-infra | 18:16 | |
| anteaya | clarkb: and are you feeling any better? | 18:16 |
| anteaya | and yay house | 18:16 |
| jeblair | mordred: i have 2 more nodes held | 18:17 |
| *** hosanai has joined #openstack-infra | 18:17 | |
| *** esberglu has quit IRC | 18:18 | |
| clarkb | anteaya: exhausted still have much yo do | 18:18 |
| anteaya | clarkb: I understand, hope you are drinking lots and lots of water | 18:18 |
| *** ramishra has quit IRC | 18:18 | |
| *** jesusaur has quit IRC | 18:18 | |
| anteaya | so check: noop, gate: noop, experimental: gate-foo-job | 18:18 |
| anteaya | clarkb: would that work? | 18:19 |
| *** ajmiller has quit IRC | 18:19 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/project-config: Add infra puppet gem dependency holder repo https://review.openstack.org/331780 | 18:19 |
| openstackgerrit | Khai Do proposed openstack-infra/project-config: Add its-storyboard plugin build to periodic pipeline https://review.openstack.org/331794 | 18:20 |
| *** ramishra has joined #openstack-infra | 18:20 | |
| *** sree has quit IRC | 18:21 | |
| clarkb | anteaya: yes I think that is fine. the jssue is mixing noop in the same pipeline | 18:21 |
| anteaya | clarkb: thanks | 18:21 |
| anteaya | nibalizer: I steered you wrong sorry, I'll clarify in a comment on your patch | 18:22 |
| *** maishsk has quit IRC | 18:22 | |
| *** hosanai has quit IRC | 18:22 | |
| bknudson_ | zaro: I think I could do !include-raw: to read a file that has the key/password -- http://docs.openstack.org/infra/jenkins-job-builder/definition.html#module-jenkins_jobs.local_yaml | 18:22 |
| bknudson_ | I didn't see anything about defining variables in the jjb config file | 18:23 |
| *** ams__ has quit IRC | 18:23 | |
| pabelanger | jeblair: Shrews: mordred: looking at put_file in lib/ansible/plugins/connection/ssh.py a trivial patch could add mkdir to the SCP command ensuring the directory exists on the remote side before doing the PUT. Which I believe is the current reason async_status fails to copy. I don't know why remote_tmp is missing or incorrect permissions | 18:24 |
| *** ajmiller has joined #openstack-infra | 18:24 | |
| zaro | bknudson_: http://docs.openstack.org/infra/jenkins-job-builder/execution.html#hipchat-section | 18:24 |
| zaro | bknudson_: looks like it’s setup for hitchat and stash. you would need to make a change to support that for slack as well | 18:25 |
| odyssey4me | Shrews jeblair mordred pabelanger The jobs that are failing for openstack-ansible are implementing ansible in tox's venv, FYI as a data point. But yeah, it is plausible that the two may somehow be interfering with each other - perhaps somehow with the tmp file locations or some such. | 18:25 |
| odyssey4me | Shrews jeblair mordred pabelanger That said, that theory doesn't really with the the puppet jobs exhibiting the same behaviour - unless puppet is also interfering with the same tmp file locations. | 18:26 |
| odyssey4me | Perhaps through the use of some common library which is naughty. | 18:26 |
| odyssey4me | (ie using a static tmp file location and perhaps file name too, or perhaps some sort of environment flag) | 18:27 |
| anteaya | pleia2: pabelanger could I trouble you for a review of an its-storyboard patch to project-config please? https://review.openstack.org/#/c/331794/ | 18:27 |
| bknudson_ | zaro: ok, that makes sense. So hipchat and stash passwords are in the config but no general variable support. | 18:27 |
| *** sree has joined #openstack-infra | 18:27 | |
| *** banix has joined #openstack-infra | 18:27 | |
| bknudson_ | and slack could work like hipchat and stash but it doesn't probably because nobody asked for it yet. | 18:27 |
| zaro | bknudson_: yes | 18:28 |
| anteaya | AJaeger: actually if you read the exchange between clarkb and I above it does make a difference | 18:28 |
| Shrews | odyssey4me: if it's within a tox env, then it is unlikely they'd intefere. seems like that may be a bad theory | 18:28 |
| anteaya | AJaeger: clarkb noticed a problem with running tests clogging the noop jobs queue | 18:29 |
| pabelanger | jeblair: mordred: Shrews: maybe we should also be sandboxing our local_tmp directory for ansible-playbook? http://docs.ansible.com/ansible/intro_configuration.html#local-tmp Maybe the file doesn't exist issue is actually happening on the side of our launcher? http://docs.ansible.com/ansible/intro_configuration.html#local-tmp | 18:29 |
| pabelanger | anteaya: looking | 18:29 |
| anteaya | pabelanger: thanks | 18:29 |
| openstackgerrit | Merged openstack-infra/project-config: Check number of channels for gerritbot https://review.openstack.org/331766 | 18:29 |
| pleia2 | anteaya: lgtm, I'll let pabelanger approve if he agrees | 18:29 |
| anteaya | pleia2: thanks for looking, and also thanks for waiting :) | 18:30 |
| pabelanger | pleia2: anteaya: agreed! +3 | 18:30 |
| anteaya | thank you | 18:30 |
| *** ddieterly[away] has quit IRC | 18:30 | |
| zaro | pleia2: regarding your comment in https://review.openstack.org/#/c/330925/ | 18:30 |
| *** dprince has quit IRC | 18:31 | |
| *** chem is now known as chem|off | 18:31 | |
| anteaya | AJaeger: I don't think I mis-understood clarkb | 18:31 |
| zaro | pleia2: that info doesn’t help me figure out which user info to retreieve from heira | 18:31 |
| *** sree_ has joined #openstack-infra | 18:31 | |
| *** sree has quit IRC | 18:32 | |
| *** sree_ is now known as Guest20432 | 18:32 | |
| anteaya | nibalizer: do you mind making the change I suggest so I can +2 your patch? | 18:32 |
| *** dprince has joined #openstack-infra | 18:32 | |
| pleia2 | zaro: I don't think it's in hiera | 18:33 |
| pleia2 | zaro: wouldn't we need to add it with this patch? | 18:33 |
| zaro | pleia2: ohh, none of em? | 18:33 |
| anteaya | nibalizer: I really would like to get beaker tests working | 18:33 |
| *** Guest20432 has quit IRC | 18:33 | |
| *** rbrndt has joined #openstack-infra | 18:33 | |
| pleia2 | zaro: well, the sync feature is new, so I don't know why we would have put it in there until now | 18:33 |
| zaro | pleia2: the patch intends to get the sb user from heira | 18:33 |
| zaro | pleia2: sync feature? | 18:34 |
| pleia2 | zaro: yeah, so a root person will need to add that to hiera before we land this | 18:34 |
| nibalizer | anteaya: AJaeger clarkb sounds like the patch is fine? | 18:34 |
| pleia2 | zaro: the purpose of this patch :) to sync updates between gerrit and storyboard bugs :) | 18:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Add local_tmp to ansible.cfg https://review.openstack.org/331797 | 18:34 |
| *** yamahata has quit IRC | 18:35 | |
| jeblair | pabelanger, mordred: i've restarted zl01 | 18:35 |
| anteaya | nibalizer: actually no, clarkb found that if you have the noop template and jobs in the experiemental queue when you run the experimental jobs they clog the noop jobs | 18:35 |
| jeblair | 2016-06-20 18:35:10,052 DEBUG zuul.NodeWorker.centos-7-ovh-bhs1-1817274: Ansible output: TASK [zuul_runner with 3587 second timeout] ************************************ | 18:35 |
| pleia2 | I was providing the users and UIDs so that whoever adds this to hiera has some guidance and to confirm that we don't need to create new storyboard users | 18:35 |
| mordred | jeblair: woot | 18:35 |
| jeblair | i'll start 02 and 03 now since that looks good | 18:35 |
| anteaya | nibalizer: which is why I suggest you specify the check and gate noop | 18:35 |
| zaro | pleia2: ahh yeah. so I’m asking.. can you put that user into heira and tell me what var i would need to reference to get the user info? | 18:36 |
| *** ddieterly has joined #openstack-infra | 18:36 | |
| anteaya | nibalizer: if you read the scrollback of the chat I had with clarkb, he said that would work | 18:36 |
| nibalizer | anteaya: did you see andrea's comments on the patch? | 18:36 |
| pabelanger | jeblair: ack | 18:36 |
| nibalizer | andreas's i mean | 18:36 |
| anteaya | nibalizer: yes, I disagree with AJaeger | 18:36 |
| nibalizer | ok | 18:36 |
| nibalizer | ill change it | 18:36 |
| anteaya | nibalizer: but he has left channel and isn't here to discuss | 18:36 |
| anteaya | nibalizer: thank you | 18:36 |
| anteaya | also clarkb is sick and moving house | 18:37 |
| bknudson_ | zaro: I don't see in the code that the hipchat plugin is going to the config -- https://git.openstack.org/cgit/openstack-infra/jenkins-job-builder/tree/jenkins_jobs/modules/publishers.py#n5901 | 18:37 |
| anteaya | so I don't want to wait on him either | 18:37 |
| bknudson_ | maybe I'm reading it wrong. | 18:37 |
| openstackgerrit | Spencer Krum proposed openstack-infra/project-config: Add infra puppet gem dependency holder repo https://review.openstack.org/331780 | 18:37 |
| *** rossella_s has quit IRC | 18:37 | |
| pleia2 | zaro: I can for production, but as I said in my comment, there isn't a gerrit user on storyboard-dev yet | 18:37 |
| pleia2 | zaro: so whoever is responsible for creating such a user has to do that first | 18:37 |
| *** rossella_s has joined #openstack-infra | 18:38 | |
| pleia2 | I'd rather add them all at once if possible | 18:38 |
| zaro | anteaya, Zara ^ would that be you guys? | 18:38 |
| fungi | i expect whatever gerrit user ended up on storyboard.openstack.org got there by way of our bug import script | 18:38 |
| anteaya | fungi: when you are available this patch needs the infra-ptl +3, it creates a new infra repo, a ruby gem which all puppet repos will import to pin gem versions: https://review.openstack.org/#/c/331780/6 | 18:39 |
| fungi | i think SotK/Zara were going to work on patches to grant themselves admin access in storyboard-dev, but i don't know if that came to pass yet | 18:39 |
| odyssey4me | pabelanger I've popped a question in that review, | 18:39 |
| anteaya | I had thought they did that | 18:39 |
| anteaya | I think they have admin access on -dev now | 18:39 |
| fungi | anteaya: nibalizer: makes sense. this is consistent with what teh puppet-openstack team are doing with their repos i gather? | 18:39 |
| *** yamamoto has joined #openstack-infra | 18:40 | |
| nibalizer | fungi: yep, we're stealing all their stuff :) | 18:40 |
| mordred | odyssey4me: it shouldn't - those are directories on the launcher, not on the node | 18:40 |
| odyssey4me | mordred ok cool - just making sure :) | 18:40 |
| *** _oanson has quit IRC | 18:40 | |
| pleia2 | zaro: I'm also not sure how to find the token (or password?) for accounts that are not mine, so for for adding to hiera I'll need some pointers there | 18:40 |
| odyssey4me | afk for a bit while I relocate | 18:40 |
| fungi | nibalizer: yoink! | 18:41 |
| anteaya | fungi: thank you | 18:41 |
| pabelanger | odyssey4me: replied | 18:42 |
| anteaya | nibalizer: once that merges, then what needs to happen? | 18:42 |
| nibalizer | anteaya: we need to push some content in | 18:42 |
| *** degorenko is now known as _degorenko|afk | 18:43 | |
| nibalizer | then probably flail a little bit getting the unit test passing, then we can patch all 60+ puppet modules (for the last time) to use this new repo as a dependency | 18:43 |
| anteaya | seems Zara and SotK are afk, so let's see if we can get this sorted out first thing tomorrow, what exactly do you need Zara/Sotk to do? | 18:43 |
| anteaya | nibalizer: okay, I would like to get the puppet-gerrit patch merged by tomorrow | 18:43 |
| *** dimtruck is now known as zz_dimtruck | 18:43 | |
| nibalizer | thanks for your help anteaya AJaeger fungi | 18:43 |
| anteaya | nibalizer: what do I need to do for that to happen? | 18:43 |
| openstackgerrit | Merged openstack-infra/project-config: [trove] Add trove-dashboard to (more) PROJECTS https://review.openstack.org/331738 | 18:44 |
| anteaya | nibalizer: thanks for creating it | 18:44 |
| nibalizer | anteaya: yea im gonna work on it today | 18:44 |
| anteaya | awesome, I'm standing by to help | 18:44 |
| *** yamamoto has quit IRC | 18:44 | |
| *** kzaitsev_mb has joined #openstack-infra | 18:44 | |
| nibalizer | cool | 18:44 |
| zaro | pleia2: REST api http://docs.openstack.org/infra/storyboard/webapi/v1.html#get--v1-users--user_id--tokens | 18:44 |
| *** r-mibu_ has quit IRC | 18:44 | |
| *** HeOS has joined #openstack-infra | 18:45 | |
| anteaya | fungi: zaro pleia2 so what do Zara/Sotk need to do tomorrow, I can pass the message along? | 18:45 |
| *** r-mibu has joined #openstack-infra | 18:45 | |
| fungi | anteaya: i don't know whether storyboard needs an admin to create some sort of bot account for this, or whether we just need to somehow log into it with the lp account we want used | 18:46 |
| zaro | anteaya: they need to create a a user that gerrit can use and pass that info along to pleia2 | 18:46 |
| anteaya | thank you | 18:46 |
| pleia2 | zaro: thanks | 18:46 |
| *** jesusaur has joined #openstack-infra | 18:46 | |
| anteaya | fungi: pleia2 zaro do we have a user name we want it to be called? | 18:46 |
| fungi | i _think_ in storyboard's world, accounts are "created" by trying to log in with an openid | 18:47 |
| *** dtardivel has quit IRC | 18:47 | |
| anteaya | yes, I think so | 18:47 |
| anteaya | I don't know if admin can do this or not | 18:47 |
| anteaya | but let's pretend for a moment it is possible | 18:47 |
| fungi | and it seems like there's no special account adding feature available to sb admins to bypass that (yet at least) | 18:47 |
| anteaya | what should the user be called? | 18:47 |
| zaro | it does not have to be an SB admin account | 18:48 |
| pleia2 | it shouldn't be | 18:48 |
| anteaya | zaro: pleia2 right | 18:48 |
| anteaya | what do we want it to be called? | 18:48 |
| pleia2 | probably match what we have in production (it's in my comment) | 18:48 |
| fungi | so i have a feeling there's nothing SotK/Zara need to (or can) do in this case. we just need to create an account with an lp user under the control of our root team | 18:48 |
| zaro | if we already have an open id account we can use then we don’t need to create a new one | 18:48 |
| anteaya | pleia2: which patch? | 18:48 |
| pleia2 | fungi: ok | 18:49 |
| anteaya | fungi: oh okay well, then we don't have to wait for them | 18:49 |
| *** banix has quit IRC | 18:49 | |
| zaro | pleia2, fungi : do we have an open id account that would be appropriate for this purpose? | 18:50 |
| pleia2 | I think we can use the same lp user as for production, assuming that has a lp user | 18:50 |
| pleia2 | and not just some automatically generated thing | 18:50 |
| pleia2 | anteaya: zaro's patch for adding this support, I'll find the link again | 18:50 |
| anteaya | thanks | 18:50 |
| pleia2 | anteaya: https://review.openstack.org/#/c/330925/ | 18:50 |
| *** zz_dimtruck is now known as dimtruck | 18:50 | |
| pleia2 | there are 3 users I noted there | 18:50 |
| zaro | pleia2: sounds good, we just need to make sure that account has a long-live token | 18:50 |
| *** sree has joined #openstack-infra | 18:51 | |
| zaro | pleia2: yeah, just pick one and let me know | 18:51 |
| anteaya | should it be OpenStack Gerrit for both storyboard-dev and storyboard? | 18:51 |
| pleia2 | what I'm saying is that I don't know how "Openstack Gerrit" etc were created, so I don't know what lp account it is using | 18:52 |
| *** rhochmuth has quit IRC | 18:52 | |
| pleia2 | anteaya: probably | 18:52 |
| anteaya | oh | 18:52 |
| anteaya | sorry I missed that part of what you were trying to convey | 18:52 |
| zaro | pleia2: ahh ok. anyway to find out? | 18:52 |
| fungi | i have a very strong feeling they were created by our bug import script | 18:52 |
| pleia2 | if needed I can probably dig into the database to see if it has one associated with it | 18:52 |
| pleia2 | fungi: yeah | 18:52 |
| *** zul has joined #openstack-infra | 18:53 | |
| pleia2 | so we may need a new account, or find a way to have a lp user take over the automatically-created-from-import user (if that's possible) | 18:53 |
| *** zul has quit IRC | 18:54 | |
| *** zul has joined #openstack-infra | 18:54 | |
| AJaeger | anteaya: let's ask clarkb... | 18:54 |
| AJaeger | anteaya: clarkb said "the issue is mixing noop in the same pipeline". | 18:55 |
| clarkb | ya so dont have check noop + pep8 or whatver | 18:55 |
| zaro | pleia2: ok. i just need to know the heira variable for whichever account you decide is most appropriate. | 18:55 |
| AJaeger | anteaya: using the template is *Exactly* the same as what the current change does. | 18:55 |
| *** sree has quit IRC | 18:55 | |
| anteaya | well it was created in 2011 in July | 18:55 |
| pleia2 | zaro: I know, it just takes some untangling to figure this out, I'll poke around the database and see what I can figure out | 18:55 |
| AJaeger | anteaya: let me propose the followup... | 18:55 |
| zaro | pleia2: coolio, thanks. | 18:56 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add a script to list change owner statistics https://review.openstack.org/263971 | 18:56 |
| anteaya | AJaeger: that's fine, if clarkb is +2 on it when he is back around again, I'm good with that | 18:56 |
| *** ihrachys has joined #openstack-infra | 18:56 | |
| anteaya | you said you were done for the day | 18:56 |
| anteaya | clarkb: is sick and in the middle of moving house | 18:57 |
| AJaeger | anteaya: I wanted to follow while it's still in our minds... | 18:57 |
| anteaya | I need the storyboard patch merged | 18:57 |
| anteaya | that is fine | 18:57 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use noop-jobs template https://review.openstack.org/331805 | 18:57 |
| AJaeger | anteaya, clarkb ^ | 18:57 |
| anteaya | when clark is +2 on it, so am I | 18:57 |
| anteaya | thank you | 18:57 |
| *** pilgrimstack has joined #openstack-infra | 18:57 | |
| AJaeger | anteaya: it's not urgent, it's just cosmetics ;) | 18:57 |
| AJaeger | clarkb: get well! | 18:57 |
| anteaya | yup, I just didn't want to create the mess clarkb noticed in the past | 18:58 |
| zaro | anteaya: so you want storyboard tomorrow? | 18:58 |
| anteaya | zaro: well by wednesday 11 utc yes | 18:58 |
| anteaya | zaro: and I am working to get beaker tests working so your approved patch can merge | 18:58 |
| AJaeger | anteaya: templates are just syntactic sugar and the noop-jobs template is exactly the same as what I replace it with. | 18:58 |
| anteaya | then hopefully your other system-config patch will pass tests then too | 18:59 |
| AJaeger | anteaya: yes, I agree, we don't want that mess again... | 18:59 |
| *** haaj has quit IRC | 18:59 | |
| AJaeger | ok, now really gone ;) bye | 18:59 |
| anteaya | AJaeger: thanks for understanding my position | 18:59 |
| anteaya | AJaeger: have a good night | 18:59 |
| zaro | anteaya: i can probably install on review-dev.o.o without those patches. | 18:59 |
| anteaya | zaro: oh would you? | 19:00 |
| anteaya | fungi: would you allow zaro to do that? | 19:00 |
| *** tkelsey has quit IRC | 19:00 | |
| anteaya | the patches are all in progress and I will continue to follow up on them | 19:00 |
| anteaya | and pleia2 has been most helpful to keep things moving along | 19:01 |
| anteaya | thanks pleia2 | 19:01 |
| zaro | anteaya: i’ve verified that it good to go. so i have no reservations installing it on review-dev.o.o prior to landing patches to do the puppet way | 19:01 |
| fungi | anteaya: sure, we still want the automation so i wouldn't want it divergent for long but this seems like a reasonable compromise to have it working in time for the sprint | 19:01 |
| anteaya | zaro: fungi thank you thank you thank you thank you | 19:01 |
| anteaya | nibalizer: I'm still standing by to help get beaker tests working | 19:01 |
| notmorgan | jeblair: email re: gear py3/encode/decode sent to -dev ML | 19:01 |
| notmorgan | jeblair: as promised. | 19:01 |
| *** dprince has quit IRC | 19:02 | |
| *** ihrachys has quit IRC | 19:02 | |
| jeblair | notmorgan: the -dev list? | 19:02 |
| jeblair | notmorgan: not -infra? | 19:02 |
| notmorgan | jeblair: openstack-dev. | 19:02 |
| jeblair | :( | 19:02 |
| zaro | anteaya: only real thing i need is an LP user with a long-live SB token. | 19:02 |
| notmorgan | oops, should have been -infra | 19:02 |
| notmorgan | let me just re-send. | 19:02 |
| jeblair | notmorgan: cool | 19:02 |
| anteaya | zaro: understood | 19:02 |
| openstackgerrit | Merged openstack-infra/project-config: Add its-storyboard plugin build to periodic pipeline https://review.openstack.org/331794 | 19:02 |
| notmorgan | doesn't hurt to have it on -dev either. | 19:02 |
| anteaya | zaro: I think pleia2 was digging to see if we can reclaim OpenStack Gerrit | 19:02 |
| pleia2 | the OpenStack Gerrit user (240) on storyboard.o.o does have a lp account associated with it in the database | 19:03 |
| nibalizer | anteaya: im gonna get lunch | 19:03 |
| notmorgan | jeblair: done. | 19:03 |
| jeblair | notmorgan: thanks! | 19:03 |
| anteaya | nibalizer: enjoy lunch, I think I too will eat | 19:03 |
| *** amitgandhinz has quit IRC | 19:03 | |
| nibalizer | we need to populate that repo when it shows up with whats in the puppet modules' repo | 19:03 |
| anteaya | pleia2: is that good? | 19:03 |
| anteaya | nibalizer: /me nods | 19:03 |
| *** pilgrimstack has quit IRC | 19:03 | |
| zaro | pleia2: is it possible for you to create a long-live token for that user on SB? either login or rest api ? | 19:03 |
| *** amitgandhinz has joined #openstack-infra | 19:03 | |
| notmorgan | SpamapS: ^ re gear, since you're interested in such things typically. | 19:04 |
| pleia2 | zaro: the admin web ui doesn't allow me to change it, but maybe through the api? | 19:04 |
| *** pilgrimstack has joined #openstack-infra | 19:04 | |
| zaro | pleia2: yeah, i think you need to login as the Openstack Gerrit user | 19:04 |
| openstackgerrit | Doug Hellmann proposed openstack-dev/pbr: generate the reno cache during sdist if reno is installed https://review.openstack.org/330814 | 19:04 |
| zaro | pleia2: you should be able to create the token for that user as long as you are an sb-dev admin | 19:05 |
| pleia2 | https://login.launchpad.net/+id/kJLEFJA is the OpenStack Gerrit user, but I'm not sure how to find the launchpad profile from there (or to tell whether https://launchpad.net/~openstack-gerrit is the same thing) | 19:06 |
| *** ramishra has quit IRC | 19:06 | |
| pleia2 | zaro: ok | 19:07 |
| *** esberglu has joined #openstack-infra | 19:07 | |
| fungi | pleia2: i have a script which uses the lp api to do a reverse lookup for that, just a sec | 19:07 |
| pleia2 | fungi: great | 19:07 |
| *** alex_xu has quit IRC | 19:07 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: Python3 Fix: tools use six.moves.urllib https://review.openstack.org/327459 | 19:07 |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/zuul: WIP: Add web-based console log streaming https://review.openstack.org/320563 | 19:08 |
| *** sree has joined #openstack-infra | 19:08 | |
| *** alex_xu has joined #openstack-infra | 19:09 | |
| *** piet_ has quit IRC | 19:09 | |
| *** denisra has joined #openstack-infra | 19:10 | |
| *** acoles is now known as acoles_ | 19:11 | |
| *** claudiub has quit IRC | 19:11 | |
| *** yamahata has joined #openstack-infra | 19:11 | |
| *** ramishra has joined #openstack-infra | 19:11 | |
| *** armax has joined #openstack-infra | 19:11 | |
| *** sree has quit IRC | 19:12 | |
| *** chem|off has quit IRC | 19:13 | |
| fungi | pleia2: i've been using http://paste.openstack.org/show/520664 (run it as gerrit2 on review.o.o) but i'm simplifying it for the current use case | 19:13 |
| *** esberglu has quit IRC | 19:13 | |
| *** yamamoto has joined #openstack-infra | 19:13 | |
| pleia2 | fungi: thanks | 19:14 |
| *** piet has joined #openstack-infra | 19:15 | |
| fungi | pleia2: try http://paste.openstack.org/show/520665 and pass the openid url as the only command-line argument | 19:16 |
| fungi | pleia2: that's basically distilled from http://git.openstack.org/cgit/openstack-infra/jeepyb/tree/jeepyb/cmd/update_bug.py#n121 | 19:17 |
| *** ihrachys has joined #openstack-infra | 19:17 | |
| *** chem|off has joined #openstack-infra | 19:17 | |
| fungi | where we try to use an account's openid listed in gerrit's account_external_ids table to figure out their lp account so we can assign bugs to them | 19:17 |
| *** pilgrimstack has quit IRC | 19:18 | |
| *** Nikolay_St has joined #openstack-infra | 19:18 | |
| fungi | (an api feature addition we were able to convince lp devs to add for us... hooray for small victories!) | 19:18 |
| pabelanger | jeblair: another failure on gate-openstack-ansible-galera_server-linters but ran on zl07. It does expose a minor issue in how we do zuul_log, http://logs.openstack.org/86/331786/3/check/gate-openstack-ansible-galera_server-linters/52b390c/console.html#_2016-06-20_19_08_40_612942 | 19:18 |
| pleia2 | https://launchpad.net/~openstack-gerrit | 19:19 |
| pleia2 | \o/ | 19:19 |
| pleia2 | thanks fungi | 19:19 |
| fungi | np | 19:19 |
| *** hosanai has joined #openstack-infra | 19:19 | |
| pabelanger | jeblair: I wonder if there is a way to stop logging to /tmp/console.log once we report a failure | 19:19 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update Robot Framework plugin https://review.openstack.org/331814 | 19:19 |
| *** Sukhdev_ has joined #openstack-infra | 19:20 | |
| mriedem | umm, got a py34 job failure here but nothing really jumping out at me why it failed http://logs.openstack.org/54/174854/35/check/gate-nova-python34-db/393609d/console.html | 19:20 |
| mriedem | tests all passed | 19:20 |
| mriedem | jenkins 3: jenkin's revenge? | 19:21 |
| mriedem | *jenkins* | 19:21 |
| pabelanger | mriedem: http://logs.openstack.org/54/174854/35/check/gate-nova-python34-db/393609d/_zuul_ansible/logs/ansible.txt there was a failure with ansible | 19:21 |
| pabelanger | that is affecting your test | 19:21 |
| *** sree has joined #openstack-infra | 19:21 | |
| pabelanger | An exception occurred during task execution. To see the full traceback, use -vvv. The error was: IOError: [Errno 2] No such file or directory: '/usr/local/lib/python2.7/dist-packages/zuul/ansible/library/zuul_log.py' | 19:21 |
| *** Sukhdev_ has quit IRC | 19:21 | |
| mriedem | super | 19:22 |
| *** yamamoto has quit IRC | 19:22 | |
| *** Sukhdev_ has joined #openstack-infra | 19:22 | |
| mordred | mriedem: yah - this is a known issue, we've got plans for a fix. for now, just recheck | 19:22 |
| mriedem | is there a bug for it? | 19:22 |
| anteaya | mriedem: you find all the good ones | 19:22 |
| anteaya | pleia2: so you have what you need? | 19:22 |
| mordred | pabelanger: did we make a bug? | 19:22 |
| pabelanger | mordred: I don't know, that is the first time I've seen that error | 19:23 |
| mriedem | shite, well, i doubt we index http://logs.openstack.org/54/174854/35/check/gate-nova-python34-db/393609d/_zuul_ansible/logs/ansible.txt anyway right? | 19:23 |
| mordred | pabelanger: this is the pip install deletes then creates files errors | 19:23 |
| mordred | pabelanger: so when we land a new version, there is an instant where some of the zuul files don't exist, and jobs can fail like this | 19:23 |
| *** Sukhdev_ has quit IRC | 19:23 | |
| pabelanger | mordred: Oh, okay | 19:23 |
| pleia2 | anteaya: for now, I think | 19:23 |
| *** ramishra has quit IRC | 19:24 | |
| pabelanger | mordred: Right, was going to say the file location of zuul_log.py didn't look right | 19:24 |
| *** Sukhdev_ has joined #openstack-infra | 19:24 | |
| *** hosanai has quit IRC | 19:24 | |
| *** inc0 has quit IRC | 19:24 | |
| pabelanger | mordred: so, to answer your question, I have not created a bug for it | 19:24 |
| mordred | pabelanger: k. me either | 19:24 |
| *** maishsk has joined #openstack-infra | 19:24 | |
| *** zeih has joined #openstack-infra | 19:25 | |
| *** Sukhdev_ has quit IRC | 19:25 | |
| * zaro afk bbl | 19:25 | |
| *** sree has quit IRC | 19:25 | |
| *** sree has joined #openstack-infra | 19:25 | |
| *** ifarkas has quit IRC | 19:26 | |
| *** piet has quit IRC | 19:27 | |
| *** ddieterly is now known as ddieterly[away] | 19:28 | |
| *** baoli_ has quit IRC | 19:29 | |
| *** sarob has joined #openstack-infra | 19:29 | |
| *** sree has quit IRC | 19:29 | |
| *** sambetts is now known as sambetts|afk | 19:30 | |
| anteaya | pleia2: great thank you | 19:32 |
| *** Jeffrey4l_ has joined #openstack-infra | 19:32 | |
| *** amoralej is now known as amoralej|brb | 19:33 | |
| *** PollyZ has quit IRC | 19:33 | |
| *** Jeffrey4l has quit IRC | 19:33 | |
| *** sarob has quit IRC | 19:33 | |
| *** sarob has joined #openstack-infra | 19:37 | |
| *** nwkarsten has quit IRC | 19:39 | |
| *** sarob has quit IRC | 19:41 | |
| *** PollyZ has joined #openstack-infra | 19:41 | |
| *** inc0 has joined #openstack-infra | 19:42 | |
| *** akshai_ has quit IRC | 19:43 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul: Enable keep_remote_files in ansible.cfg https://review.openstack.org/331819 | 19:44 |
| EmilienM | the only ansible thing I see is http://logs.openstack.org/29/331229/4/check/gate-puppet-nova-puppet-beaker-rspec-ubuntu-xenial/006671e/logs/syslog.txt.gz#_Jun_20_19_00_43 | 19:44 |
| EmilienM | ansible-async_wrapper[1972]: 1973 still running (3295) | 19:44 |
| EmilienM | jeblair: ^ | 19:44 |
| pabelanger | mordred: jeblair: ^ if we want to consider keep_remote_files | 19:45 |
| *** gyee has quit IRC | 19:45 | |
| *** nwkarsten has joined #openstack-infra | 19:45 | |
| jeblair | EmilienM: thanks. that's expected. | 19:45 |
| *** nwkarsten has quit IRC | 19:45 | |
| *** nwkarsten has joined #openstack-infra | 19:46 | |
| *** burgerk has joined #openstack-infra | 19:46 | |
| *** PollyZ has quit IRC | 19:46 | |
| pabelanger | jeblair: failure with timeout passed into name field: http://logs.openstack.org/86/331786/3/check/gate-openstack-ansible-galera_server-ansible-func-centos-7/7fe28b0/_zuul_ansible/logs/ansible.txt | 19:48 |
| jeblair | pabelanger: zl01-03 are restarted. i gracefuld 04 and 05 | 19:48 |
| pabelanger | that ran on zl02 | 19:49 |
| pabelanger | jeblair: acl | 19:49 |
| pabelanger | ack* | 19:49 |
| jeblair | pabelanger: cool, seems like we're on the right track | 19:49 |
| *** sree_ has joined #openstack-infra | 19:49 | |
| *** Sukhdev_ has joined #openstack-infra | 19:49 | |
| jeblair | pabelanger, mordred: i have held around 8 nodes so far, and have not gotten lucky | 19:49 |
| *** sree_ is now known as Guest39399 | 19:49 | |
| *** zeih has quit IRC | 19:49 | |
| *** dizquierdo has quit IRC | 19:50 | |
| *** Sukhdev_ has quit IRC | 19:50 | |
| *** baoli has joined #openstack-infra | 19:51 | |
| *** akshai has joined #openstack-infra | 19:51 | |
| *** sree has joined #openstack-infra | 19:51 | |
| *** yamamoto has joined #openstack-infra | 19:51 | |
| *** baoli has quit IRC | 19:52 | |
| *** locust has quit IRC | 19:52 | |
| mordred | \o/ | 19:52 |
| mordred | 19:52 | |
| mordred | If you provided an IRC nic - do you prefer the IRC nic to be the most prominent identifier printed on your Summit name badge? | 19:52 |
| mordred | jbryce: ^^ thank you | 19:52 |
| *** PollyZ has joined #openstack-infra | 19:52 | |
| *** Guest39399 has quit IRC | 19:53 | |
| *** esberglu has joined #openstack-infra | 19:53 | |
| *** zeih has joined #openstack-infra | 19:54 | |
| fungi | yeah, that's a pretty awesome addition. i didn't even know that was coming! | 19:54 |
| mordred | fungi: please pass on my joy, should the opportunity arise | 19:55 |
| pabelanger | jeblair: mordred: looks like we'd only hit this code path in ansible when poll > 0. As a fallback, we could also try 328649 which moves zuul_runner to fire and forget, which a follow up async_status task | 19:55 |
| fungi | mordred: i shall also add my own joy to it | 19:55 |
| mordred | pabelanger: interesting | 19:55 |
| *** yolanda has left #openstack-infra | 19:56 | |
| *** baoli has joined #openstack-infra | 19:56 | |
| pabelanger | I had some success with it a 2 weekends ago when I was testing locally | 19:56 |
| *** sree has quit IRC | 19:56 | |
| *** yolanda has joined #openstack-infra | 19:56 | |
| *** annegentle has quit IRC | 19:58 | |
| mordred | pabelanger: maybe perhaps this is happening when the task finishes really quickly - so it runs, exits and deletes all before anything is able to check the poll? | 19:58 |
| *** akshai has quit IRC | 19:59 | |
| *** Apoorva has quit IRC | 19:59 | |
| mordred | the link you pasted in was done in less than one minute - althouhg nevermind, I guess we're polling on 5 second boundaries | 19:59 |
| * mordred goes back to dark hole | 19:59 | |
| *** mhickey has joined #openstack-infra | 20:00 | |
| *** ddieterly[away] is now known as ddieterly | 20:00 | |
| *** akshai has joined #openstack-infra | 20:00 | |
| pabelanger | mordred: I think we need to understand what is getting returned: https://github.com/ansible/ansible/blob/devel/lib/ansible/executor/task_executor.py#L585 | 20:01 |
| pabelanger | because loop break, ansible then looks at async_result.finished | 20:01 |
| pabelanger | which, in our case, is not actually finished | 20:01 |
| pabelanger | and fails | 20:01 |
| *** Apoorva has joined #openstack-infra | 20:01 | |
| *** yamamoto has quit IRC | 20:01 | |
| pabelanger | moving to poll: 0 and using async_status ourself, moves this code path from ansible. allow us in the task to better control the failure conditions | 20:02 |
| notmorgan | pabelanger: oh, so we're just doing something that is relatively unexpected. | 20:02 |
| notmorgan | pabelanger: it seems. | 20:02 |
| pabelanger | ya, not sure how normal_handler.run() works | 20:02 |
| pabelanger | that's my next thing to look at | 20:03 |
| *** xyang1 has joined #openstack-infra | 20:03 | |
| * notmorgan nods. | 20:03 | |
| pabelanger | either we let ansible continue to do our polling for async, or 328649 is an attempt to handle it via a task | 20:03 |
| *** dtantsur is now known as dtantsur|afk | 20:06 | |
| notmorgan | pabelanger: hmmmmmm. don't have a good answer for you there :P | 20:06 |
| *** sarob has joined #openstack-infra | 20:07 | |
| pabelanger | the only thing that make sense, is async_result is failed or skipped. Which breaks our loop and causes task did not complete within requested time | 20:08 |
| *** sarob has quit IRC | 20:08 | |
| *** Nikolay_St has quit IRC | 20:08 | |
| *** sarob has joined #openstack-infra | 20:09 | |
| dougwig | hi infra, "If you provided an IRC nic - do you prefer the IRC nic to be the most prominent identifier printed on your Summit name badge?". three cheers for whoever put that in for barcelona! :) | 20:11 |
| waynr | zaro++ thanks! | 20:12 |
| waynr | zaro: is there any way i can help look into getting release-specific documentation for python-jenkins on rtd? | 20:12 |
| fungi | dougwig: indeed! i'm waiting for my code to show up so i can check that checkbox | 20:12 |
| fungi | thrilled! | 20:12 |
| *** sree has joined #openstack-infra | 20:13 | |
| *** baoli has quit IRC | 20:13 | |
| *** baoli has joined #openstack-infra | 20:14 | |
| *** kgiusti has left #openstack-infra | 20:14 | |
| fungi | dougwig: i'm passing along everyone's thanks for that to our awesome conference organizers | 20:14 |
| *** baoli has quit IRC | 20:15 | |
| *** sree has quit IRC | 20:17 | |
| dmsimard | fungi: thanks for the OpenStack summit invite :p | 20:18 |
| *** zul has quit IRC | 20:18 | |
| fungi | dmsimard: you're welcome! (though really, i just run some scripts to generate these... thank _you_ for making openstack better!) | 20:19 |
| *** amitgandhinz has quit IRC | 20:19 | |
| EmilienM | dmsimard: you can't go before passing the spanish test though | 20:19 |
| *** tonytan4ever has joined #openstack-infra | 20:19 | |
| *** amitgandhinz has joined #openstack-infra | 20:19 | |
| dmsimard | EmilienM: oh, you speak spanish ? TIL | 20:19 |
| EmilienM | dmsimard: I don't :( | 20:20 |
| *** hosanai has joined #openstack-infra | 20:20 | |
| *** sree has joined #openstack-infra | 20:20 | |
| *** javeriak has quit IRC | 20:21 | |
| fungi | dmsimard: EmilienM: though it might not help you since they speak primarily catalan there anyway ;) | 20:21 |
| *** zul has joined #openstack-infra | 20:22 | |
| *** zeih has quit IRC | 20:22 | |
| EmilienM | fungi: we'll speak OpenStack then | 20:24 |
| inc0 | fungi, nibalizer hey, quck intro, me and mrhillsman already talked to nibalizer and we're going to add few servers to infra | 20:24 |
| *** hosanai has quit IRC | 20:24 | |
| fungi | i lie. the most recent census cited by wikipedia at least says spanish is considered the first language for 46.53% of the catalonian population, followed by 37.26% for catalan | 20:24 |
| *** sree has quit IRC | 20:24 | |
| *** amoralej|brb is now known as amoralej | 20:25 | |
| fungi | inc0: running what? to what purpose? | 20:25 |
| inc0 | fungi gates, I'm part of OSIC | 20:25 |
| nibalizer | we had a brief chat | 20:25 |
| fungi | inc0: oh! adding servers to our quota in osic? | 20:25 |
| *** sree has joined #openstack-infra | 20:25 | |
| inc0 | I know you already have 100 vms on OSIC, but we'll be making new cloud aside from that | 20:25 |
| nibalizer | fungi: they want to provide some quota oto us | 20:25 |
| inc0 | much bigger | 20:25 |
| nibalizer | sounds like a different cloud | 20:26 |
| *** alexm2 has joined #openstack-infra | 20:26 | |
| fungi | exciting! | 20:26 |
| inc0 | different cloud, different quota, much much bigger | 20:26 |
| inc0 | there are caveats to that tho | 20:26 |
| fungi | i'm all for much, much bigger ;) | 20:26 |
| inc0 | we'll be redeploying this stuff every now and then | 20:26 |
| mtreinish | mordred: time for my somewhat regular pestering about the mysql stuff :) | 20:26 |
| mtreinish | mordred: anything I can do to help move that along? | 20:26 |
| inc0 | so we need a trigger mechanism to turn off scheduling of new jobs on it | 20:26 |
| cinerama | greghaynes, around? i'm just poking at the IPA dib image support | 20:27 |
| inc0 | and some automated bootstrap so the moment we're done, you can just start using it | 20:27 |
| fungi | inc0: we do have configuration already continuously consumed to indicate how many instances to boot in a provider/region. we can just set that to 0 | 20:27 |
| nibalizer | inc0: so if you pop in here and ask, we can turn it down on a pretty quick turnaround | 20:27 |
| inc0 | so let's talk what's involved | 20:27 |
| greghaynes | cinerama: in a call at the moment, should be around in a little while | 20:27 |
| cinerama | greghaynes, cool | 20:27 |
| inc0 | fungi I was thinking about sth like that | 20:27 |
| * notmorgan has a barcelona ticket - woo. | 20:27 | |
| inc0 | so, what's involved in including our hardware to gate pool? | 20:28 |
| inc0 | we'll have ipv6 vms | 20:28 |
| dmsimard | You get a Barcelona ticket.. and YOU get a Barcelona ticket.. and everybody gets a Barcelona ticket ! | 20:28 |
| *** PollyZ has quit IRC | 20:28 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder: Refactor: block-device handling (local loop) https://review.openstack.org/319591 | 20:28 |
| inc0 | I guess there will be specific tenant you want to use | 20:29 |
| notmorgan | dmsimard: lol | 20:29 |
| fungi | inc0: we set up a new stanza for the credentials at http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/nodepool/clouds.yaml.erb | 20:29 |
| *** dimtruck is now known as zz_dimtruck | 20:29 | |
| fungi | inc0: and then we add some configuration to http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml saying what to build/boot there and how many | 20:30 |
| *** BobBall is now known as BobBall_AWOL | 20:30 | |
| *** sree has quit IRC | 20:30 | |
| *** PollyZ has joined #openstack-infra | 20:31 | |
| clarkb | fungi: keep in mind that nodepool will currently be unhappy if the cloud is gone, even when the quota is set to 0 | 20:31 |
| fungi | inc0: will glance images get carried over across the outages you're describing? if not, we may need to add a step to clear out our image tracking for that environment so we don't try to boot anything until we upload fresh images | 20:31 |
| *** HeOS has quit IRC | 20:31 | |
| *** rcernin has quit IRC | 20:31 | |
| fungi | clarkb: yep, though i think we all agree that's a bug we want to fix | 20:31 |
| inc0 | fungi I was rather thinking of you giving us these images and we'll just add them after deployment | 20:31 |
| fungi | clarkb: and afaik it's just nodepool service restarts which are impacted by that? | 20:31 |
| *** sree has joined #openstack-infra | 20:32 | |
| clarkb | fungi: it may also be config reloads | 20:32 |
| clarkb | fungi: I am not completely sure | 20:32 |
| *** flepied has joined #openstack-infra | 20:32 | |
| fungi | inc0: well, we create about half a dozen new images and upload them every day | 20:32 |
| *** yamamoto has joined #openstack-infra | 20:32 | |
| inc0 | ok, so either image tracking or downloading of latest | 20:32 |
| inc0 | ? | 20:32 |
| fungi | inc0: so it's easiest for us to "give" them to you via glance uploads anyway | 20:32 |
| fungi | inc0: we push them into glance | 20:32 |
| fungi | so not really "downloading" | 20:33 |
| inc0 | kk, so we'll need to figure out a way to ensure they're there when we're done with redeploy | 20:33 |
| pabelanger | jeblair: mordred: notmorgan: Shrews: We should rsync the contents of ~/.ansible_async as from the worker. That will contain the bits used by async | 20:33 |
| pabelanger | trying to see if we can update our JJB publisher to do that | 20:33 |
| *** gyee has joined #openstack-infra | 20:33 | |
| inc0 | I guess flavors are predefined, so we can bootstrap these | 20:33 |
| fungi | inc0: yeah, if we clear out our tracked images for that environment in nodepool's database it will automatically try to upload new ones before it boots anything there | 20:34 |
| inc0 | I hope you're not hardcoding IDs;) | 20:34 |
| fungi | inc0: hardcoding what ids? image uuids? | 20:34 |
| inc0 | or flavors or user uuids | 20:34 |
| Shrews | pabelanger: we really shouldn't have to | 20:34 |
| inc0 | you know, stuff that changes deploy to deploy | 20:35 |
| notmorgan | pabelanger: I'm going to have to defer to Shrews and mordred on that one. I really don't know. | 20:35 |
| inc0 | so word of explanation - we'll going to test performance and such | 20:35 |
| inc0 | and we need cloud that is used | 20:35 |
| clarkb | inc0: fungi no we don't hardcode those it does a search over flaovrs for the best matching and it uses the images it has uploaded. This does mean that if cloud goes down then up it will think the images it uploaded should still be there afterwards | 20:35 |
| inc0 | so you'll be our workload;) | 20:35 |
| fungi | inc0: image uuids are what i'm saying we'd need to delete on the nodepool side so that it would upload fresh ones | 20:35 |
| clarkb | fungi: ya that | 20:35 |
| *** csomerville has joined #openstack-infra | 20:36 | |
| Shrews | pabelanger: jeblair: mordred: why haven't we turned on verbose logging with ansible-playbook yet? | 20:36 |
| *** baoli has joined #openstack-infra | 20:36 | |
| *** sree has quit IRC | 20:36 | |
| pabelanger | Shrews: my only reason would be to see if Exception is running: https://github.com/ansible/ansible-modules-core/blob/devel/utilities/logic/async_wrapper.py | 20:36 |
| fungi | inc0: for flavors, we use minimum ram heuristics (optionally filtered by patterns in the flavor name) to select an appropriate flavor, so no hard ids there | 20:37 |
| fungi | inc0: and for accounts, we just track username, password and project name | 20:37 |
| pabelanger | Shrews: I think we want too | 20:37 |
| pabelanger | just going to be a lot of data | 20:37 |
| *** yamamoto has quit IRC | 20:38 | |
| *** daemontool has joined #openstack-infra | 20:38 | |
| inc0 | so we're good | 20:38 |
| *** maishsk has quit IRC | 20:38 | |
| *** cody-somerville has quit IRC | 20:38 | |
| inc0 | fungi do you need any help with this tracking image code? | 20:38 |
| fungi | clarkb: i wonder if it would make sense for nodepool to grow a feature to compare image-list to its tracked table of images and delete references those not present | 20:38 |
| inc0 | that would solve image issue | 20:39 |
| *** zul has quit IRC | 20:39 | |
| fungi | inc0: well, it's a simple sql delete query one of us can run when we knock the max server count to 0, but it seems like something not entirely terrible for nodepool to be able to do | 20:39 |
| fungi | on its own | 20:40 |
| clarkb | fungi: ya that owuld handle the situation well | 20:40 |
| alexm2 | Hi All. A new project gerrit/build question: we contributed an unofficial neutron plugin as a project called "quark". The contributed code does not pass the Jenkins checks, such as doc build, and some required files, i.e. .gitreview, are missing. What is the appropriate way to include the build targets and any missing files? | 20:40 |
| *** _nadya_ has quit IRC | 20:41 | |
| fungi | inc0: as clarkb mentioned though, the bigger issue is hunting down what part of the main scheduler is dependent on being able to reach all its providers at start (and possibly also at reconfiguration) and patching that | 20:41 |
| inc0 | well, think about it, we're still waiting for access to hardware, but I hope we to provide you a tenant acces later this week | 20:41 |
| *** gouthamr has quit IRC | 20:41 | |
| Shrews | pabelanger: i think we should have your keep_remote_files change, regardless of whether or not it fixes this issue | 20:41 |
| inc0 | yeah, we need a way to turn off scheduling | 20:42 |
| Shrews | jeblair: ^^^ | 20:42 |
| *** sree has joined #openstack-infra | 20:42 | |
| inc0 | it doesn't have to me instant, and we can do this here | 20:42 |
| fungi | inc0: nibalizer: what frequency of teardown/redeploy did you discuss, and how long would the ups and downs be? | 20:42 |
| inc0 | but to streamline process it would be ideal if we'd do this on our own | 20:42 |
| nibalizer | fungi: our conversation didn't get that far (was largely an introduction) | 20:42 |
| inc0 | fungi I'd assume every couple of weeks | 20:42 |
| inc0 | ? | 20:42 |
| inc0 | and downtime will be day or so | 20:42 |
| fungi | inc0: oh, that sounds beneficial. how many 8gb ram instances do you think you'd be able to muster when it's up? | 20:43 |
| inc0 | so our process will be to create new conf of openstack, test it and when we're ok with it roll it out | 20:43 |
| *** trown is now known as trown|outtypewww | 20:43 | |
| Shrews | mordred: pabelanger: jeblair: oh, you know, this could totally be some issue with the new ziploader not properly packaging up the modules | 20:43 |
| *** sree_ has joined #openstack-infra | 20:43 | |
| inc0 | depends on how we arch it, but we'll have 22 nodes total, 256gig ram each and all for you | 20:44 |
| Shrews | mordred: pabelanger: jeblair: but i have no idea how to debug such a thing | 20:44 |
| *** sree_ is now known as Guest22942 | 20:44 | |
| Shrews | unless we can hold a node with the problem, then maybe | 20:44 |
| inc0 | so if we assume 16 computes, so I'd say somwhere between 500 and 800? | 20:44 |
| inc0 | so I guess worthwile?;) | 20:45 |
| fungi | inc0: yep, that basically doubles our current capacity | 20:45 |
| pabelanger | Shrews: never heard of ziploader before, any docs on it? | 20:45 |
| inc0 | yeah, I figured that would make quite a difference | 20:45 |
| fungi | inc0: so definitely a help | 20:46 |
| inc0 | even with this redeploy caveat | 20:46 |
| *** sree has quit IRC | 20:46 | |
| fungi | right, roughly double our current capacity even if we assume 6/7 uptime | 20:46 |
| inc0 | and since we decide when to redeploy, it's no problem to schedule it outside of normal peaks like milestones | 20:46 |
| pleia2 | fungi: thanks for your work on summit registration code finagling once again | 20:46 |
| *** jcoufal has quit IRC | 20:47 | |
| fungi | pleia2: this time the patch was _very_ small. just needed to make it handle accounts with a missing username | 20:47 |
| pleia2 | that's good to hear | 20:47 |
| *** ihrachys has quit IRC | 20:47 | |
| Shrews | pleia2: not really. you could check the PR where it was added. but a small bit about the packaging can be read here: http://docs.ansible.com/ansible/developing_modules.html#debugging-ansiblemodule-based-modules | 20:47 |
| pleia2 | ^ pabelanger | 20:47 |
| Shrews | pabelanger: ^^^^ , not pleia2 :) | 20:47 |
| fungi | pleia2: so the current state seems to be at feature parity with the old script for purposes of election rolls and summit invites | 20:47 |
| inc0 | fungi so think about this scheduler trigger mechanism and images, feel free to ping me or mrhillsman for questions/help | 20:47 |
| inc0 | we're prepping stuff locally in the meantime | 20:48 |
| *** sree has joined #openstack-infra | 20:48 | |
| Shrews | pabelanger: https://github.com/ansible/ansible/pull/15246 | 20:48 |
| *** Guest22942 has quit IRC | 20:48 | |
| inc0 | ah one more thing | 20:48 |
| fungi | inc0: great to hear. we'll likely need to recruit someone to help get a couple patches into nodepool to make this smoother, but in teh meantime we do have some semi-manual workarounds which are reasonably quick | 20:48 |
| inc0 | any chances we could get access to your monitoring data too? We could use failed/success stats for uptime and such | 20:49 |
| fungi | (can be enacted in a matter of minutes normally) | 20:49 |
| * pleia2 lunch | 20:49 | |
| fungi | inc0: yep! it's already public at grafana.openstack.org | 20:49 |
| inc0 | fungi if you need coder...I'm coder;) | 20:49 |
| *** daemontool has quit IRC | 20:49 | |
| inc0 | just point me to the correct place;) | 20:49 |
| inc0 | yeah, I know grafana, we might need raw data tho | 20:49 |
| fungi | inc0: for current osic boot stats, see http://grafana.openstack.org/dashboard/db/nodepool-osic | 20:50 |
| inc0 | but that's minor, we'll figure it out | 20:50 |
| fungi | inc0: that's all being graphed from a real-time source at graphite.openstack.org, and you can query it for json dumps of raw data | 20:50 |
| inc0 | we'll be doing all sorts of measurements on it, also I hope to produce few nice blogposts and such about perf of that stuff | 20:50 |
| pabelanger | Shrews: thanks, looking now | 20:50 |
| inc0 | cool | 20:51 |
| fungi | inc0: the grafana config we're using is also public, in git, so you can see the exact graphite resources we're polling that way too | 20:51 |
| inc0 | glad to hear, let's get it up and running first tho | 20:51 |
| fungi | yep | 20:52 |
| *** yolanda has quit IRC | 20:52 | |
| *** sree has quit IRC | 20:52 | |
| *** banix has joined #openstack-infra | 20:52 | |
| *** sree has joined #openstack-infra | 20:53 | |
| inc0 | ok, that's it from me now, I'll let you know when we'll have setup up and running | 20:53 |
| inc0 | we'll take it from there later, we have time till our first deploy;) | 20:53 |
| *** akshai has quit IRC | 20:53 | |
| inc0 | re-deploy | 20:53 |
| fungi | inc0: sounds great--thanks again! | 20:53 |
| fungi | and as i said, easy to work around initially as well (just revert the nodepool config addition and a quick sql delete query before we put the config back after the redeploy) | 20:54 |
| *** _ari_ has quit IRC | 20:55 | |
| *** pvaneck has joined #openstack-infra | 20:55 | |
| *** Sukhdev has joined #openstack-infra | 20:56 | |
| inc0 | cool, off to ipv6 config I go then, cya around | 20:56 |
| *** Anticime1 is now known as Anticimex | 20:57 | |
| *** zul has joined #openstack-infra | 20:57 | |
| *** shamail has joined #openstack-infra | 20:57 | |
| *** sree has quit IRC | 20:57 | |
| *** tlian has quit IRC | 20:57 | |
| greghaynes | cinerama: ok, freedom - whats up? | 20:57 |
| *** sree has joined #openstack-infra | 20:58 | |
| *** claudiub has joined #openstack-infra | 20:59 | |
| cinerama | greghaynes, woo. anyway so i think we talked the other day about the ability to override an element in base dib with another in the elements path | 20:59 |
| *** julim has quit IRC | 20:59 | |
| greghaynes | cinerama: Yep - it basically should all 'just work' if you have multiple elements with the same name in the elements-path | 20:59 |
| *** nwkarsten has quit IRC | 21:00 | |
| greghaynes | whetever is later in the path should overwrite whatever is earlier | 21:00 |
| *** baoli has quit IRC | 21:00 | |
| pabelanger | Shrews: Ya, looks like a copy of syslog on the worker would be helpful too. Going to see if I can track down a devstack job that is failing | 21:01 |
| cinerama | greghaynes: svc-map seems unhappy when i try it, so i'm wondering if we shoudl make some specific changes to support overrides | 21:01 |
| greghaynes | cinerama: ah, if it doesnt work then we definitely should | 21:01 |
| Shrews | pabelanger: syslog? | 21:01 |
| greghaynes | cinerama: what happens to svc map? | 21:01 |
| *** raildo is now known as raildo-afk | 21:02 | |
| pabelanger | Shrews: for async_wrapper logs seems to output notice to syslog | 21:02 |
| Shrews | pabelanger: ah | 21:02 |
| cinerama | greghaynes, throws the "blah already found in services list" | 21:02 |
| *** rockyg has joined #openstack-infra | 21:02 | |
| *** sree has quit IRC | 21:03 | |
| greghaynes | cinerama: oh! hah | 21:03 |
| cinerama | greghaynes, obviously we want to avoid accidental dupes but where the element is the same... | 21:03 |
| greghaynes | cinerama: ya, so its an element issue not a dib tool issue. Fun | 21:03 |
| *** sarob has quit IRC | 21:04 | |
| *** sree has joined #openstack-infra | 21:04 | |
| *** nwkarsten has joined #openstack-infra | 21:04 | |
| greghaynes | cinerama: but yes, its defintely worth making that work - we have to do this every now and then and I would expect it to continue. Its common when we move an element from something like tripleo-image-elements in to dib | 21:04 |
| greghaynes | since they have to maintain backwards compat as well | 21:04 |
| cinerama | greghaynes: well both copies of the element are presumably going to have the same service name | 21:05 |
| *** amoralej is now known as amoralej|off | 21:05 | |
| cinerama | (s) | 21:05 |
| *** nwkarsten has quit IRC | 21:05 | |
| *** esberglu has quit IRC | 21:06 | |
| *** lucasagomes has quit IRC | 21:06 | |
| greghaynes | cinerama: I dont totally understand what is failing - I have a high level idea (I am guessing svc-map tries to prevent name collisions) but I dont remember how that works entirely... | 21:07 |
| *** sree has quit IRC | 21:08 | |
| cinerama | greghaynes, it's looking for the service name in a dict. we don't keep track of what element path it comes from | 21:08 |
| *** armax has left #openstack-infra | 21:08 | |
| greghaynes | cinerama: but maybe svc-map should compare the service definition and if it finds an already defined one it doesnt error if the definition is identical? | 21:08 |
| *** sree has joined #openstack-infra | 21:09 | |
| cinerama | greghaynes, that sounds reasonable | 21:10 |
| cinerama | greghaynes, we have midcycle on this week so i might not get stuck into it straightaway | 21:11 |
| greghaynes | cinerama: awesome, sounds good | 21:11 |
| *** Sukhdev has quit IRC | 21:11 | |
| *** lucasagomes has joined #openstack-infra | 21:12 | |
| *** thorst_ has quit IRC | 21:12 | |
| *** zul has quit IRC | 21:14 | |
| *** sree has quit IRC | 21:14 | |
| *** sree has joined #openstack-infra | 21:14 | |
| *** roxanaghe has quit IRC | 21:15 | |
| *** roxanaghe has joined #openstack-infra | 21:16 | |
| *** zul has joined #openstack-infra | 21:16 | |
| lifeless | apuimedo: ? | 21:16 |
| *** sree has quit IRC | 21:18 | |
| nibalizer | fungi: anteaya my change failed in gate :( fixing | 21:18 |
| *** sree has joined #openstack-infra | 21:20 | |
| *** nmagnezi has quit IRC | 21:20 | |
| *** hosanai has joined #openstack-infra | 21:21 | |
| *** salv-orlando has joined #openstack-infra | 21:22 | |
| *** ldnunes has quit IRC | 21:22 | |
| *** asselin_ has joined #openstack-infra | 21:22 | |
| *** akshai has joined #openstack-infra | 21:22 | |
| *** akshai_ has joined #openstack-infra | 21:23 | |
| *** asselin has quit IRC | 21:23 | |
| *** sree has quit IRC | 21:24 | |
| *** sree has joined #openstack-infra | 21:24 | |
| *** zul has quit IRC | 21:25 | |
| *** hosanai has quit IRC | 21:26 | |
| *** akshai has quit IRC | 21:27 | |
| *** nwkarsten has joined #openstack-infra | 21:28 | |
| *** inc0 has quit IRC | 21:28 | |
| *** sree has quit IRC | 21:29 | |
| *** jamesmcarthur has quit IRC | 21:29 | |
| *** sree has joined #openstack-infra | 21:30 | |
| *** thorst_ has joined #openstack-infra | 21:31 | |
| *** zul has joined #openstack-infra | 21:31 | |
| *** rhochmuth has joined #openstack-infra | 21:32 | |
| *** akshai_ has quit IRC | 21:32 | |
| *** HeOS has joined #openstack-infra | 21:32 | |
| *** nwkarsten has quit IRC | 21:32 | |
| zaro | waynr: you can try to setup a project yourself to see if the doc will generate for every release. and once you figure that out let me know what needs to be set to get python-jenkins to do the same. | 21:34 |
| *** sree has quit IRC | 21:35 | |
| *** ffrank has joined #openstack-infra | 21:35 | |
| *** mhickey has quit IRC | 21:35 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/project-config: Add infra puppet gem dependency holder repo https://review.openstack.org/331780 | 21:36 |
| *** thorst_ has quit IRC | 21:36 | |
| *** Sukhdev has joined #openstack-infra | 21:37 | |
| *** sree has joined #openstack-infra | 21:37 | |
| waynr | zaro: okay, i'll eventually try that once I open source my JJB wrapper project | 21:37 |
| *** sree_ has joined #openstack-infra | 21:39 | |
| *** ddieterly is now known as ddieterly[away] | 21:39 | |
| *** sree_ is now known as Guest98026 | 21:39 | |
| *** Sukhdev has quit IRC | 21:41 | |
| *** sree has quit IRC | 21:41 | |
| *** banix has quit IRC | 21:42 | |
| *** Sukhdev has joined #openstack-infra | 21:42 | |
| *** Guest98026 has quit IRC | 21:43 | |
| *** ddieterly[away] is now known as ddieterly | 21:44 | |
| *** asselin has joined #openstack-infra | 21:44 | |
| *** HeOS has quit IRC | 21:45 | |
| *** amitgandhinz has quit IRC | 21:45 | |
| *** shamail has left #openstack-infra | 21:45 | |
| *** sree has joined #openstack-infra | 21:46 | |
| *** Sukhdev__ has joined #openstack-infra | 21:46 | |
| *** Sukhdev has quit IRC | 21:47 | |
| *** eharney has quit IRC | 21:50 | |
| *** sree has quit IRC | 21:50 | |
| *** sree has joined #openstack-infra | 21:51 | |
| *** abregman|afk has quit IRC | 21:52 | |
| *** matt-borland has quit IRC | 21:54 | |
| *** Sukhdev__ has quit IRC | 21:54 | |
| *** tonytan4ever has quit IRC | 21:54 | |
| *** sree has quit IRC | 21:55 | |
| *** sree has joined #openstack-infra | 21:55 | |
| *** kmartin has quit IRC | 21:58 | |
| *** tkelsey has joined #openstack-infra | 21:59 | |
| *** sree has quit IRC | 22:00 | |
| *** sree has joined #openstack-infra | 22:01 | |
| *** PollyZ has quit IRC | 22:01 | |
| *** PollyZ has joined #openstack-infra | 22:02 | |
| *** zul has quit IRC | 22:03 | |
| *** csomerville has quit IRC | 22:03 | |
| *** raunak has quit IRC | 22:03 | |
| *** tkelsey has quit IRC | 22:03 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:04 | |
| *** tonytan4ever has joined #openstack-infra | 22:05 | |
| *** Sukhdev_ has joined #openstack-infra | 22:05 | |
| *** raunak has joined #openstack-infra | 22:05 | |
| *** sree has quit IRC | 22:05 | |
| *** raunak has quit IRC | 22:06 | |
| *** edmondsw has quit IRC | 22:07 | |
| *** HeOS has joined #openstack-infra | 22:08 | |
| *** mdrabe has quit IRC | 22:08 | |
| *** sree has joined #openstack-infra | 22:09 | |
| *** ociuhandu has joined #openstack-infra | 22:09 | |
| *** raunak has joined #openstack-infra | 22:09 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Fix tempest configuration https://review.openstack.org/326055 | 22:10 |
| *** dizquierdo has joined #openstack-infra | 22:10 | |
| craige | o/ | 22:10 |
| *** ddieterly is now known as ddieterly[away] | 22:12 | |
| openstackgerrit | Christopher Aedo proposed openstack-infra/irc-meetings: Switch to odd, update chairs https://review.openstack.org/331863 | 22:13 |
| *** aysyd has quit IRC | 22:13 | |
| *** aysyd has joined #openstack-infra | 22:13 | |
| *** sree has quit IRC | 22:14 | |
| *** rwsu_ has joined #openstack-infra | 22:14 | |
| *** sree has joined #openstack-infra | 22:15 | |
| *** rwsu has quit IRC | 22:17 | |
| apuimedo | lifeless: gone. It was already merged | 22:18 |
| apuimedo | thanks for pinging back though ;-) | 22:18 |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update ScriptTrigger plugin https://review.openstack.org/331866 | 22:19 |
| *** sree has quit IRC | 22:20 | |
| openstackgerrit | Merged openstack-dev/pbr: Restore warnerrors behavior and support Sphinx 1.4 https://review.openstack.org/229951 | 22:20 |
| *** sree has joined #openstack-infra | 22:20 | |
| *** hosanai has joined #openstack-infra | 22:21 | |
| *** sree has quit IRC | 22:25 | |
| *** sree has joined #openstack-infra | 22:26 | |
| *** hosanai has quit IRC | 22:26 | |
| *** hasharAway has quit IRC | 22:28 | |
| *** sree has quit IRC | 22:31 | |
| *** gordc has quit IRC | 22:34 | |
| *** ddieterly[away] is now known as ddieterly | 22:35 | |
| *** aysyd has quit IRC | 22:36 | |
| *** sree has joined #openstack-infra | 22:37 | |
| *** sshnaidm has quit IRC | 22:37 | |
| *** rossella_s has quit IRC | 22:37 | |
| *** rossella_s has joined #openstack-infra | 22:38 | |
| *** sree_ has joined #openstack-infra | 22:38 | |
| *** sree_ is now known as Guest90590 | 22:39 | |
| *** sree has quit IRC | 22:41 | |
| *** sree has joined #openstack-infra | 22:43 | |
| *** Guest90590 has quit IRC | 22:43 | |
| jhesketh | Morning | 22:44 |
| *** ociuhandu has quit IRC | 22:46 | |
| *** sree has quit IRC | 22:47 | |
| *** sree has joined #openstack-infra | 22:47 | |
| *** tonytan4ever has quit IRC | 22:48 | |
| *** sree has quit IRC | 22:52 | |
| *** amotoki has quit IRC | 22:53 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Stop destroying VM's that don't exist https://review.openstack.org/312863 | 22:55 |
| *** sree has joined #openstack-infra | 22:55 | |
| *** ddieterly is now known as ddieterly[away] | 22:57 | |
| *** gyee has quit IRC | 22:58 | |
| openstackgerrit | Kien Ha proposed openstack-infra/jenkins-job-builder: Update ScriptTrigger plugin https://review.openstack.org/331866 | 22:59 |
| *** sree has quit IRC | 22:59 | |
| *** sdague has quit IRC | 23:00 | |
| *** tkelsey has joined #openstack-infra | 23:00 | |
| *** sree has joined #openstack-infra | 23:00 | |
| *** gouthamr has joined #openstack-infra | 23:01 | |
| *** ddieterly[away] is now known as ddieterly | 23:02 | |
| *** rockyg has quit IRC | 23:02 | |
| *** bradm_ is now known as bradm | 23:03 | |
| *** Sukhdev_ has quit IRC | 23:03 | |
| *** bhavik has quit IRC | 23:03 | |
| jeblair | inc0, fungi: there was an infra-cloud todo item assigned to me to add a quiesce feature to nodepool to assist in our redeploys | 23:04 |
| *** tkelsey has quit IRC | 23:04 | |
| jeblair | inc0, fungi: i have not written it yet, because i'm apparently practicing just-in-time development | 23:04 |
| *** sree has quit IRC | 23:04 | |
| jeblair | inc0, fungi: but if it would help with what you're talking about ('quiesce nodepool, redeploy, unquiesce nodepool') i will be happy to write it. it should not take long. | 23:05 |
| *** sree has joined #openstack-infra | 23:05 | |
| *** asselin__ has joined #openstack-infra | 23:06 | |
| *** kien-ha has quit IRC | 23:06 | |
| *** rbrndt has quit IRC | 23:07 | |
| SpamapS | jeblair: got a second to answer an AFS question? | 23:07 |
| jeblair | Shrews, pabelanger: i think keep_remote_files is okay -- however, we will need to think about our long-running workers -- they will accumulate those files indefinitely. also -- thinking about zuulv3, if we have to keep using this, it will make it harder for others to use as it adds extra-zuul requirements (like a tmpreaper or something) | 23:07 |
| jeblair | SpamapS: always | 23:07 |
| *** tpsilva has quit IRC | 23:07 | |
| *** roxanaghe has quit IRC | 23:08 | |
| *** hongbin has quit IRC | 23:08 | |
| SpamapS | jeblair: :) so, we're using /afs/openstack.org for some things .. and I just started wondering.. how does AFS prevent somebody from MITM'ing openstack.org's SRV records and feeding us evil packages? | 23:08 |
| *** roxanaghe has joined #openstack-infra | 23:08 | |
| jeblair | SpamapS: dnssec? :) | 23:08 |
| jeblair | SpamapS: seeing as how i don't think openstack.org is dnssec enabled.... | 23:09 |
| *** sree has quit IRC | 23:09 | |
| SpamapS | jeblair: ok, so we should maybe not use it that way then? | 23:09 |
| *** asselin_ has quit IRC | 23:09 | |
| *** rhochmuth has quit IRC | 23:10 | |
| jeblair | SpamapS: probably the next best answer is if you're worried about that, you could use the cellservdb file instead of dns | 23:10 |
| *** pradk has quit IRC | 23:10 | |
| jeblair | SpamapS: (it contains a cell-name -> ip address mapping) | 23:10 |
| SpamapS | jeblair: and then how would we prevent a more directed MITM of our AFS traffic? | 23:10 |
| SpamapS | (far less likely than a DNS attack, but I need to understand how it works is all) | 23:11 |
| *** jamesmcarthur has joined #openstack-infra | 23:11 | |
| jeblair | SpamapS: (AFS pre-dates DNS, so it doesn't depend on it ;) | 23:12 |
| pabelanger | jeblair: agreed, a tmpreaper sounds right | 23:12 |
| SpamapS | that's amazing :) | 23:12 |
| jeblair | SpamapS: i don't have a good answer for that without authenticated access.... :/ | 23:12 |
| fungi | jeblair: quiesce as in continue to queue events (state transitions) but don't make new api calls? | 23:13 |
| jeblair | fungi: as in mark the provider offline and gracefully stop using it | 23:13 |
| fungi | oh, got it | 23:13 |
| jeblair | fungi: (delete instances but do not create new ones) | 23:13 |
| fungi | so better than just setting max-servers for it to 0 or -1 | 23:13 |
| SpamapS | jeblair: that's rather disturbing. :-P | 23:14 |
| jeblair | fungi: ya, something that could be put in a playbook | 23:14 |
| jeblair | SpamapS: a possible answer might be "dude, can you imagine the effort needed to implement a MITM for AFS?" but i could see how that might be unsatisfactory. | 23:15 |
| *** jamesmcarthur has quit IRC | 23:16 | |
| *** amotoki has joined #openstack-infra | 23:16 | |
| jeblair | SpamapS: but yeah, outside authenticated access, there's no trust anchor i can think of that a client would have. | 23:17 |
| SpamapS | jeblair: Right, I am entirely fine with that answer actually. I just have to have a rough estimate of risk. | 23:17 |
| *** PollyZ has quit IRC | 23:17 | |
| *** ddieterly has quit IRC | 23:17 | |
| jeblair | SpamapS: (with authenticated access, we're talking kerberos, so i imagine the answer would be different) | 23:18 |
| SpamapS | jeblair: There'd be no issue if infra just signed the repos. I could import the public key and be done. | 23:18 |
| jeblair | SpamapS: also, happy to help, but if you're writing up risk profiles and this is important, you should totally hire an afs consultant if you want correct answers that don't involve the word "dude" :) | 23:18 |
| SpamapS | jeblair: somehow I doubt kerberos really solves the problem, unless each data stream includes signatures. | 23:19 |
| jeblair | SpamapS: yeah, i think repo signing of some kind is still desirable. that may be actionable. :) | 23:19 |
| SpamapS | jeblair: I think I'd rather solve the problem in the contents of the mirrors. Would patches to infra repos be accepted that caused those repos to be signed by a key that infra-root had to protect? | 23:19 |
| SpamapS | fungi: ^ ? | 23:20 |
| mordred | SpamapS: so ... fungi has been working on the key management stuff needed for us to be able to sign things | 23:20 |
| *** claudiub has quit IRC | 23:20 | |
| jeblair | SpamapS: i'm going to invoke the names of fungi and mordred to continue this thread :) | 23:20 |
| SpamapS | because.. those AFS mirrors that don't 404 are really nice. :) | 23:20 |
| *** claudiub has joined #openstack-infra | 23:20 | |
| SpamapS | and that only require a few rewrite rules and a /afs dir :) | 23:20 |
| jeblair | SpamapS: you are so demanding. you want data to be available *and* legitimate? no one has ever asked for both at the same time before. | 23:21 |
| *** amotoki has quit IRC | 23:21 | |
| *** Sukhdev_ has joined #openstack-infra | 23:21 | |
| *** bhavik has joined #openstack-infra | 23:21 | |
| jeblair | i forgot to say: ;) | 23:21 |
| SpamapS | jeblair: don't make me stomp my feet. | 23:21 |
| mordred | SpamapS: so that'll fix that - we've also talked, additionally, about shifting from reprepro to pull the packages and build our own repo to doing an rsync so that we'd have the original signatures on the repo - but we need to write a repo validity checker to be able to do that | 23:21 |
| *** dizquierdo has quit IRC | 23:21 | |
| fungi | SpamapS: happy to have help on it honestly. working on item 3 at http://specs.openstack.org/openstack-infra/infra-specs/specs/artifact-signing.html#work-items right now | 23:21 |
| mordred | SpamapS: in any case, there are two different pieces of work that either one of would make your life better | 23:22 |
| mordred | SpamapS: one of them is a script that I believe you could probably write more easily than I could | 23:22 |
| jeblair | either would solve this problem -- but having our own key sets us up for self-publishing packages in future, yeah? | 23:22 |
| mordred | uip | 23:22 |
| mordred | gah | 23:22 |
| mordred | jeblair: yup | 23:22 |
| mordred | jeblair: I think at some point in the future doing _both_ would be lovely | 23:22 |
| SpamapS | Well, at this point, deploying is happening.. now.. and I'm pretty sure the risk is too high to keep using the AFS mirrors until that stuff is done. | 23:22 |
| *** banix has joined #openstack-infra | 23:22 | |
| mordred | SpamapS: how hard do you think it would be to write a script that would validate a Packages.gz against what's on disk? | 23:23 |
| pabelanger | FWIW: centos-7 should be fine. Those packages are using the original gpgkey | 23:23 |
| lifeless | headsup - pep8 has been renamed upstream | 23:23 |
| lifeless | its now pycodestyle | 23:23 |
| mordred | SpamapS: basically, we want to make sure we don't do a vos release if the things we have are in an internally inconsistent state | 23:23 |
| mordred | lifeless: thank you - I think I noticed that | 23:23 |
| lifeless | this will need a change to requirements to update teh blacklist | 23:23 |
| mordred | lifeless: and then promptly forgot about it | 23:23 |
| jeblair | pabelanger: yeah, that's the rsync approach, right? but missing the 'validate' step? | 23:24 |
| pabelanger | jeblair: yes, I should loop back to the validation step. | 23:24 |
| pabelanger | we | 23:24 |
| fungi | lifeless: yeah, that was a fun thread with guido | 23:24 |
| pabelanger | we've been luck so far | 23:24 |
| lifeless | mordred: yeah. so it needs expand-contract: add pycodestyle to requirements and the blacklist etc. Then later once nothing references pep8 contract | 23:24 |
| pabelanger | lucky* | 23:24 |
| mordred | lifeless: ++ | 23:24 |
| *** salv-orlando has quit IRC | 23:25 | |
| lifeless | mordred: I'm looking for someone to do that for me, since so many folk are paid to do OpenStack stuff ... I'd just get distracted and forget the contract | 23:25 |
| lifeless | fungi: aye :) | 23:25 |
| jeblair | pabelanger, mordred: i have started zl04 and 05; gracefuld 06 and 07 | 23:26 |
| mordred | lifeless: nod. I'm about to walk away for the evening, but happy to help | 23:26 |
| jeblair | pabelanger, mordred: next time we should run pabelanger's playbook. it would probably be faster than me getting distracted. | 23:26 |
| mordred | jeblair: ++ | 23:26 |
| pabelanger | \o/ | 23:26 |
| fungi | jeblair: also he tested it over the weekend for that restart, so should be good to go | 23:28 |
| *** sshnaidm has joined #openstack-infra | 23:28 | |
| lifeless | mordred: so yeah, either do, or find someone, :) | 23:28 |
| lifeless | mordred: this is basically just a heads-up | 23:28 |
| *** Sukhdev_ has quit IRC | 23:28 | |
| *** jamesmcarthur has joined #openstack-infra | 23:28 | |
| mordred | lifeless: thanks! | 23:28 |
| SpamapS | lifeless: ^^ is there something in launchpad that you know of that validates the contents of the archive? | 23:29 |
| lifeless | SpamapS: fsvo | 23:30 |
| lifeless | SpamapS: obviously validation is a huge space. LP itself takes care of gc'ing old sources, etc | 23:30 |
| *** yuanying has joined #openstack-infra | 23:31 | |
| lifeless | SpamapS: is there something in particular you're worried about | 23:31 |
| lifeless | ? | 23:31 |
| SpamapS | lifeless: mordred wants to actually make sure we haven't listed any packages that are not still on disk in pool/ | 23:32 |
| lifeless | SpamapS: we're running our own archive now? | 23:33 |
| *** jamesmcarthur has quit IRC | 23:33 | |
| SpamapS | lifeless: no, we're running reprepro's right now, but we'd rather mirror _the_ archive and just make sure it isn't going to throw 404's before publishing it. | 23:33 |
| lifeless | SpamapS: there are still versions of apt in use that don't support the race-free format | 23:34 |
| mordred | lifeless: we don't delete things from ours - so the race isn't so much a problem with us | 23:35 |
| lifeless | SpamapS: ubuntu's reference rsync mirror scripts will get you a consistent mirror each time, so if you're on top of an atomically updatable fs, you can minimise the client impact | 23:35 |
| mordred | lifeless: awesome | 23:35 |
| lifeless | but clients will still race, until they switch over to the atomic data structures | 23:35 |
| SpamapS | yeah this isn't really about the hash sum mismatch IIRC | 23:36 |
| SpamapS | that's just a "pre-xenial" problem | 23:36 |
| *** fungi_lounge has quit IRC | 23:36 | |
| lifeless | ok, so what is it about? Assume I don't know your assumptions :) | 23:36 |
| mordred | lifeless, SpamapS: so if we use the rsync scripts minus the deletes, we should wind up with good atomic repo states, yeah? | 23:36 |
| mordred | regardless of the client age | 23:37 |
| lifeless | mordred: no | 23:37 |
| jamielennox | so https://review.openstack.org/#/c/330845/ didn't fix the ACLs for oslo.context. anyone know why? | 23:37 |
| mordred | no? what's the other problem I'm protecting against? | 23:37 |
| lifeless | mordred: you cannot be atomic with older clients, its not a script / ordering limitation | 23:37 |
| mordred | why not? | 23:37 |
| lifeless | mordred: external gpg signatures on constant paths | 23:38 |
| SpamapS | mordred: because older clients will fetch Packages.gz just before you update. | 23:38 |
| mordred | the problem is usually that the client has pulled the packages list, then 5 minutes later try to install a package that doesn't exist anymore | 23:38 |
| *** sree has joined #openstack-infra | 23:38 | |
| SpamapS | or Releases | 23:38 |
| SpamapS | or one of those | 23:38 |
| lifeless | mordred: the file, and the signature are in separate files, and they are at fixed paths. | 23:38 |
| *** sdake has joined #openstack-infra | 23:38 | |
| lifeless | mordred: so Read file-1, atomic-update-happens, Read file-1 signature -- boom. | 23:39 |
| *** sree_ has joined #openstack-infra | 23:39 | |
| SpamapS | lifeless: that's a problem for 'apt-get update' | 23:39 |
| openstackgerrit | Spencer Krum proposed openstack-infra/project-config: Add infra puppet gem dependency holder repo https://review.openstack.org/331780 | 23:39 |
| SpamapS | and very real | 23:39 |
| openstackgerrit | Ian Wienand proposed openstack-infra/os-loganalyze: Match ansible output for console https://review.openstack.org/331514 | 23:39 |
| *** sree_ is now known as Guest66053 | 23:39 | |
| SpamapS | even if you update all the contents at the same time, and synchronize expires headers | 23:39 |
| SpamapS | some clients will still fall in the window | 23:39 |
| SpamapS | but the problem mordred is talking about is a much much bigger window. | 23:39 |
| mordred | right | 23:39 |
| lifeless | right, its intrinsic, which is why we designed a fix years ago and apt-hackers have implemented it and its spreading out into use atm | 23:40 |
| SpamapS | and would happen even w/ xenial | 23:40 |
| mordred | the problem I'm talking about is the "apt-get update ; sleep 10 ; apt-get install foo" | 23:40 |
| lifeless | what problem is mordred talking about ? | 23:40 |
| lifeless | ah | 23:40 |
| mordred | that one is what the gate hits | 23:40 |
| lifeless | so the archive can hold old versions | 23:40 |
| SpamapS | so yeah, if you just rsync, but don't --delete.. | 23:40 |
| mordred | and that one is the one that just not gc-ing hepls with | 23:40 |
| mordred | right | 23:40 |
| SpamapS | then you don't get that problem | 23:40 |
| mordred | so if rsync gets me the other side of that without issues | 23:40 |
| lifeless | I forget what the gc time period in Ubuntu's main archive is | 23:40 |
| mordred | then I think it should be fairly easy to move to it | 23:40 |
| lifeless | I'd actually be inclined to fix this upstream, by getting a longer grace period on the archive | 23:41 |
| mordred | we do a 2 hour GC delay right now | 23:41 |
| lifeless | but yeah, not deleting would be a starting point | 23:41 |
| mordred | and have basically seen those errors go away | 23:41 |
| mordred | moving to rsync would make the current GC strategy a bit harder, so we'd likely just stop deleting completely | 23:42 |
| *** sree has quit IRC | 23:42 | |
| mordred | oh - amusingly enough - I have no idea if we'd see the signing race or not | 23:42 |
| *** sdake_ has joined #openstack-infra | 23:42 | |
| mordred | because our current repo isn't signed - so there is nothing to be out of sync :) | 23:42 |
| *** sdake has quit IRC | 23:43 | |
| lifeless | mordred: its actually on the hashes, so yes, you can still see it | 23:43 |
| *** Guest66053 has quit IRC | 23:43 | |
| mordred | ok. cool | 23:43 |
| lifeless | bigjools/wgrant/cjwatson would be good folk to ping about what gc period upstream has | 23:44 |
| ianw | jeblair: just adding to class Console() in zuul_runner? | 23:45 |
| *** amitgandhinz has joined #openstack-infra | 23:46 | |
| *** vhosakot has quit IRC | 23:46 | |
| *** sree has joined #openstack-infra | 23:46 | |
| stajkowski | our project was added, need to add myself to the quark-core and quark-release groups so I can add others | 23:46 |
| *** hosanai has joined #openstack-infra | 23:47 | |
| *** hosanai_ has joined #openstack-infra | 23:48 | |
| *** gouthamr_ has joined #openstack-infra | 23:48 | |
| *** sree has quit IRC | 23:51 | |
| *** amitgandhinz has quit IRC | 23:51 | |
| *** chlong has joined #openstack-infra | 23:51 | |
| *** hosanai has quit IRC | 23:51 | |
| *** gouthamr has quit IRC | 23:52 | |
| *** chlong is now known as chlong|rhce_trng | 23:52 | |
| *** sree has joined #openstack-infra | 23:54 | |
| *** banix has quit IRC | 23:55 | |
| *** zhurong has joined #openstack-infra | 23:56 | |
| SpamapS | lifeless: seems like GC could be 5 days and nobody would care. I wonder why it's aggressive in any way. | 23:56 |
| lifeless | SpamapS: indeed :) | 23:56 |
| SpamapS | I mean, oh noes your mirror gets a few hundred MB bigger on openoffice updates. | 23:56 |
| fungi | lifeless: mordred: one of the more painful race issues when using other mirrors (not ours) was that we would frequently see the package list out of sync with the actual packages on disk, sometimes for up to 24 hours (depending on how frequently the mirrors were updating themselves). so what i really want to be sure of is that whatever package list we're serving, all the packages listed in it are on | 23:57 |
| fungi | disk (at least the names, possibly also the sizes? validating all the checksums would be a lot more onerous cpu/io time wise) | 23:57 |
| *** thorst_ has joined #openstack-infra | 23:57 | |
| *** sree has quit IRC | 23:58 | |
| mordred | yah. validating the checksums would require keeping a database for sure | 23:58 |
| *** xarses has quit IRC | 23:58 | |
| mordred | so that we only calculated the checksums of things we didn't have already | 23:58 |
| *** Sukhdev has joined #openstack-infra | 23:59 | |
| *** Sukhdev has quit IRC | 23:59 | |
| *** aysyd has joined #openstack-infra | 23:59 | |
| *** thorst_ has quit IRC | 23:59 | |
| openstackgerrit | Hisashi Osanai proposed openstack-infra/project-config: Fix Monasca Analytics python27 job https://review.openstack.org/331535 | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!