| *** ihrachys has quit IRC | 00:01 | |
| *** gmann has quit IRC | 00:11 | |
| *** gmann has joined #openstack-infra | 00:11 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add docs for web server component https://review.openstack.org/482630 | 00:13 |
|---|---|---|
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add link to Zuul v3 docs to the README https://review.openstack.org/482349 | 00:14 |
| *** thorst has quit IRC | 00:17 | |
| *** jbadiapa_ has quit IRC | 00:17 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix ZuulWeb() invocation https://review.openstack.org/482321 | 00:19 |
| *** dingyichen has joined #openstack-infra | 00:19 | |
| *** Swami has quit IRC | 00:22 | |
| *** sree has joined #openstack-infra | 00:23 | |
| *** Julien-zte has joined #openstack-infra | 00:25 | |
| *** vhosakot has quit IRC | 00:26 | |
| *** makowals has quit IRC | 00:27 | |
| *** sree has quit IRC | 00:27 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Use package module to install bindep packages https://review.openstack.org/482584 | 00:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Simplify bindep logic removing fallback support https://review.openstack.org/482650 | 00:29 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Move ssh-agent cleanup into existing exception handler https://review.openstack.org/482735 | 00:30 |
| *** jbadiapa_ has joined #openstack-infra | 00:32 | |
| *** rloo has left #openstack-infra | 00:34 | |
| *** rloo has quit IRC | 00:35 | |
| *** xyang1 has quit IRC | 00:35 | |
| *** MatthewLi has joined #openstack-infra | 00:37 | |
| *** EricGonczer_ has quit IRC | 00:42 | |
| *** Julien-zte has quit IRC | 00:42 | |
| *** gildub has joined #openstack-infra | 00:43 | |
| *** zhurong has joined #openstack-infra | 00:43 | |
| *** slaweq has joined #openstack-infra | 00:43 | |
| *** Julien-zte has joined #openstack-infra | 00:43 | |
| *** Apoorva has quit IRC | 00:48 | |
| *** slaweq has quit IRC | 00:48 | |
| *** Julien-zte has quit IRC | 00:48 | |
| *** MatthewLi has quit IRC | 00:49 | |
| *** Julien-zte has joined #openstack-infra | 00:49 | |
| *** baoli has joined #openstack-infra | 00:49 | |
| *** Julien-zte has quit IRC | 00:50 | |
| *** MatthewLi has joined #openstack-infra | 00:51 | |
| *** cuongnv has joined #openstack-infra | 00:53 | |
| *** LindaWang has joined #openstack-infra | 00:54 | |
| *** bobh has joined #openstack-infra | 00:55 | |
| *** liujiong has joined #openstack-infra | 00:56 | |
| *** gcb has quit IRC | 00:57 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Remove isJobRegistered https://review.openstack.org/482755 | 01:00 |
| *** gouthamr has quit IRC | 01:06 | |
| *** bobh has quit IRC | 01:10 | |
| *** masuberu has joined #openstack-infra | 01:14 | |
| *** cuongnv has quit IRC | 01:16 | |
| *** bobh has joined #openstack-infra | 01:17 | |
| *** masber has quit IRC | 01:18 | |
| *** pahuang has joined #openstack-infra | 01:20 | |
| *** cshastri has joined #openstack-infra | 01:21 | |
| *** rossella_s has quit IRC | 01:26 | |
| *** dfflanders has quit IRC | 01:26 | |
| *** ihrachys has joined #openstack-infra | 01:31 | |
| *** bobh has quit IRC | 01:33 | |
| *** mriedem has quit IRC | 01:33 | |
| *** aviau has quit IRC | 01:34 | |
| *** aviau has joined #openstack-infra | 01:34 | |
| *** s-shiono has joined #openstack-infra | 01:42 | |
| *** makowals has joined #openstack-infra | 01:44 | |
| *** slaweq has joined #openstack-infra | 01:45 | |
| *** vhosakot has joined #openstack-infra | 01:46 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config master: Add gerribot to #openstack-tc for governance changes https://review.openstack.org/482761 | 01:46 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/project-config master: Add gerribot to #openstack-tc https://review.openstack.org/482761 | 01:48 |
| *** slaweq has quit IRC | 01:49 | |
| *** mwarad has quit IRC | 01:55 | |
| *** masber has joined #openstack-infra | 01:57 | |
| *** annegentle has joined #openstack-infra | 01:58 | |
| *** masuberu has quit IRC | 02:00 | |
| *** yamahata has quit IRC | 02:00 | |
| *** iyamahat has quit IRC | 02:00 | |
| *** bobh has joined #openstack-infra | 02:03 | |
| *** dargains has joined #openstack-infra | 02:05 | |
| *** cuongnv has joined #openstack-infra | 02:07 | |
| *** hongbin has joined #openstack-infra | 02:09 | |
| *** dargains has quit IRC | 02:11 | |
| *** dtroyer has quit IRC | 02:11 | |
| *** dtroyer has joined #openstack-infra | 02:12 | |
| *** gildub has quit IRC | 02:15 | |
| *** mwarad has joined #openstack-infra | 02:19 | |
| fungi | inc0: i've locked out the kolla images again. tarballs.o.o started getting flooded at about 01:45 and sites became unresponsive again | 02:22 |
| *** gouthamr has joined #openstack-infra | 02:26 | |
| *** annegentle has quit IRC | 02:29 | |
| vhosakot | fungi: do you know which kolla gate job caused the flood by any chance? | 02:29 |
| fungi | vhosakot: i couldn't begin to guess. one of the ones that downloads registry tarballs from tarballs.o.o | 02:30 |
| *** GregHouse has joined #openstack-infra | 02:31 | |
| fungi | they're between 2 to 4 gigabytes apiece and that site is capped a 800 megabits per second combined inbound plus outbound | 02:31 |
| fungi | so it only takes a few dozen downloads that size at a time to overload the server for a while | 02:32 |
| openstackgerrit | wes hayutin proposed openstack-infra/project-config master: add tripleo container periodic job https://review.openstack.org/482771 | 02:32 |
| *** makowals has quit IRC | 02:33 | |
| vhosakot | fungi: all gates that build from source pull tarballs from tarballs.openstack.org. I think running such gates is parellal is causing the flood. | 02:37 |
| fungi | i agree | 02:37 |
| *** thorst has joined #openstack-infra | 02:39 | |
| fungi | testing more than a couple of kolla changes at a time is probably our limit until we can finish getting the caching capacity upfit on the mirror servers | 02:39 |
| *** cuongnv has quit IRC | 02:40 | |
| vhosakot | ok | 02:40 |
| fungi | i had hoped i'd have a chance to finish that today, but i was busy with meetings all day so pabelanger has generously offered to do the two server rebuilds it's blocked on | 02:44 |
| *** thorst has quit IRC | 02:44 | |
| *** slaweq has joined #openstack-infra | 02:45 | |
| *** ramishra has joined #openstack-infra | 02:45 | |
| clarkb | fungi: thats ti increase disk available to the cache? | 02:45 |
| clarkb | fungi: was it only infracloud we need to do that in? | 02:46 |
| fungi | clarkb: yeah, i already did cinder volumes everywhere else and have the stack of changes to move the cache location | 02:46 |
| *** makowals has joined #openstack-infra | 02:46 | |
| clarkb | neat | 02:46 |
| fungi | except ovh where the rootfs is huge so didn't really need anythinfg | 02:46 |
| fungi | current bottom of the stack is https://review.openstack.org/482228 which is wip until those two instances are rebuilt with the updated flavor | 02:47 |
| *** yamahata has joined #openstack-infra | 02:50 | |
| *** slaweq has quit IRC | 02:50 | |
| clarkb | we'll also likely need to test that apache2s default file serving serves files such that they are cacheable, but thats much easier once this gets in place | 02:51 |
| *** zzzeek_ has quit IRC | 02:52 | |
| clarkb | when did tarballs get https? thats nice | 02:52 |
| clarkb | fungi: comment on https://review.openstack.org/#/c/482203/4 I +2'd the others | 02:52 |
| fungi | been https-able for a year or more, when i started doing a multi-domain cert on static.o.o | 02:56 |
| fungi | an thanks for the reviews! | 02:56 |
| *** makowals has quit IRC | 03:01 | |
| *** zhurong has quit IRC | 03:08 | |
| *** gildub has joined #openstack-infra | 03:08 | |
| *** bobh has quit IRC | 03:09 | |
| *** ramishra has quit IRC | 03:10 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Caching proxies for tarballs.openstack.org https://review.openstack.org/482203 | 03:13 |
| fungi | clarkb: ^ fixed | 03:14 |
| *** cuongnv has joined #openstack-infra | 03:16 | |
| *** dave-mccowan has joined #openstack-infra | 03:19 | |
| *** dingyichen has quit IRC | 03:21 | |
| *** baoli has quit IRC | 03:22 | |
| *** rlandy has quit IRC | 03:23 | |
| *** baoli has joined #openstack-infra | 03:26 | |
| *** baoli has quit IRC | 03:31 | |
| *** gcb has joined #openstack-infra | 03:36 | |
| fungi | inc0: i reset the kolla images to readable again a while ago and the spike didn't return so maybe we're okay again for the time being | 03:38 |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=311&rra_id=all | 03:38 |
| *** makowals has joined #openstack-infra | 03:39 | |
| *** dave-mccowan has quit IRC | 03:40 | |
| openstackgerrit | Goutham Pacha Ravi proposed openstack-infra/project-config master: Limit jobs in manila driverfixes branches https://review.openstack.org/482785 | 03:41 |
| *** rodrigopaiva has joined #openstack-infra | 03:41 | |
| *** zhurong has joined #openstack-infra | 03:44 | |
| *** gouthamr has quit IRC | 03:46 | |
| *** slaweq has joined #openstack-infra | 03:46 | |
| fungi | infra-root: if we get another traffic spike on static.o.o, i've been doing `sudo chmod o-r /srv/static/tarballs/kolla/images/*.tar.gz` and then restarting apache2 to kill the in-progress downloads | 03:49 |
| *** slaweq has quit IRC | 03:51 | |
| openstackgerrit | Amrith Kumar proposed openstack/diskimage-builder master: the current 08-debootstrap doesn't allow cross platform development https://review.openstack.org/482792 | 04:00 |
| *** dingyichen has joined #openstack-infra | 04:03 | |
| *** tonyb has quit IRC | 04:05 | |
| *** sshnaidm|afk has joined #openstack-infra | 04:08 | |
| *** hughhalf has quit IRC | 04:12 | |
| *** ykarel has joined #openstack-infra | 04:12 | |
| *** adisky__ has joined #openstack-infra | 04:21 | |
| *** rodrigopaiva has quit IRC | 04:27 | |
| *** cuongnv has quit IRC | 04:28 | |
| *** tonyb has joined #openstack-infra | 04:28 | |
| *** psachin has joined #openstack-infra | 04:30 | |
| *** makowals has quit IRC | 04:32 | |
| *** yamamoto has joined #openstack-infra | 04:33 | |
| *** hongbin has quit IRC | 04:35 | |
| *** gongysh has joined #openstack-infra | 04:35 | |
| *** thorst has joined #openstack-infra | 04:40 | |
| *** pcaruana has joined #openstack-infra | 04:41 | |
| *** thorst has quit IRC | 04:45 | |
| *** cuongnv has joined #openstack-infra | 04:45 | |
| *** ramishra has joined #openstack-infra | 04:47 | |
| *** coolsvap has joined #openstack-infra | 04:56 | |
| *** pcaruana has quit IRC | 04:58 | |
| *** makowals has joined #openstack-infra | 05:01 | |
| *** slaweq has joined #openstack-infra | 05:05 | |
| *** sree has joined #openstack-infra | 05:05 | |
| *** slaweq has quit IRC | 05:05 | |
| openstackgerrit | Caleb Boylan proposed openstack-infra/system-config master: (WIP) Get install_puppet.sh to install puppet 4 https://review.openstack.org/481955 | 05:05 |
| *** vhosakot has quit IRC | 05:06 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/irc-meetings master: update time, tony's back https://review.openstack.org/482803 | 05:09 |
| *** MatthewLi has quit IRC | 05:11 | |
| *** armax has quit IRC | 05:14 | |
| *** armax has joined #openstack-infra | 05:14 | |
| *** armax has quit IRC | 05:15 | |
| *** armax has joined #openstack-infra | 05:15 | |
| *** armax has quit IRC | 05:16 | |
| *** armax has joined #openstack-infra | 05:16 | |
| *** armax has quit IRC | 05:16 | |
| *** armax has joined #openstack-infra | 05:17 | |
| *** armax has quit IRC | 05:17 | |
| *** krtaylor has quit IRC | 05:17 | |
| *** armax has joined #openstack-infra | 05:17 | |
| *** armax has quit IRC | 05:18 | |
| *** armax has joined #openstack-infra | 05:18 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/irc-meetings master: update time, tony's back https://review.openstack.org/482803 | 05:18 |
| *** armax has quit IRC | 05:19 | |
| *** armax has joined #openstack-infra | 05:19 | |
| *** armax has quit IRC | 05:20 | |
| *** baoli has joined #openstack-infra | 05:22 | |
| *** baoli has quit IRC | 05:27 | |
| *** cuongnv has quit IRC | 05:29 | |
| *** rcernin has joined #openstack-infra | 05:33 | |
| *** sanket_ has joined #openstack-infra | 05:36 | |
| *** gcb has quit IRC | 05:37 | |
| *** nicolasbock has joined #openstack-infra | 05:38 | |
| *** pgadiya has joined #openstack-infra | 05:39 | |
| *** gildub has quit IRC | 05:41 | |
| *** gyee has quit IRC | 05:52 | |
| *** prateek has joined #openstack-infra | 05:58 | |
| *** MatthewLi has joined #openstack-infra | 06:02 | |
| *** slaweq has joined #openstack-infra | 06:02 | |
| *** ramishra has quit IRC | 06:04 | |
| *** cuongnv has joined #openstack-infra | 06:04 | |
| *** dargains has joined #openstack-infra | 06:06 | |
| *** ramishra has joined #openstack-infra | 06:06 | |
| *** slaweq_ has joined #openstack-infra | 06:06 | |
| *** slaweq has quit IRC | 06:06 | |
| *** slaweq has joined #openstack-infra | 06:07 | |
| *** dargains_ has joined #openstack-infra | 06:07 | |
| *** tjones has quit IRC | 06:09 | |
| *** dargains has quit IRC | 06:11 | |
| *** florianf has joined #openstack-infra | 06:11 | |
| *** dargains_ has quit IRC | 06:11 | |
| *** slaweq_ has quit IRC | 06:12 | |
| AJaeger | infra-root, I get 502s from review.o.o ;(. Anybody around to check and potentially restart, please? | 06:13 |
| AJaeger | After a reload it works but is slowish. | 06:13 |
| *** gtrxcb has joined #openstack-infra | 06:13 | |
| *** gtrxcb has quit IRC | 06:13 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: send graphite stats from ARA https://review.openstack.org/480121 | 06:14 |
| *** liujiong has quit IRC | 06:19 | |
| *** liujiong has joined #openstack-infra | 06:19 | |
| sshnaidm|afk | seems like gerrit is kind of stuck.. | 06:20 |
| openstackgerrit | Merged openstack-infra/project-config master: Add gerribot to #openstack-tc https://review.openstack.org/482761 | 06:21 |
| *** isaacb has joined #openstack-infra | 06:25 | |
| AJaeger | sshnaidm|afk: read my comment from 7 minutes before yours | 06:26 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Use ara generation from collect-logs role https://review.openstack.org/478549 | 06:26 |
| *** wolverineav has joined #openstack-infra | 06:31 | |
| *** isaacb_ has joined #openstack-infra | 06:32 | |
| *** isaacb has quit IRC | 06:33 | |
| *** gongysh has quit IRC | 06:34 | |
| calebb | http://ask.openstack.org/ also looks down from here | 06:39 |
| *** smcginnis has quit IRC | 06:39 | |
| *** thorst has joined #openstack-infra | 06:41 | |
| rakhmerov | AJaeger: hi, I'd like to develop a new gate. Not really experienced in that yet. Let's say I sent a patch to add a new gate into CI. Is there a way to activate it per request even if it wasn't merged yet into project-config? | 06:41 |
| rakhmerov | in other words, how to debug it | 06:41 |
| AJaeger | rakhmerov: not with current zuul - but zuul v3 will do that (coming in September/October) | 06:42 |
| rakhmerov | aah | 06:42 |
| rakhmerov | ok | 06:42 |
| AJaeger | So, what I advise: Add to experimental queue and trigger with "run experimental" - and iterate on that. | 06:42 |
| rakhmerov | yep, got it | 06:42 |
| AJaeger | And once it works reliable, move to check queue | 06:42 |
| rakhmerov | thanks | 06:42 |
| AJaeger | "check experimental" is the key I think | 06:42 |
| AJaeger | calebb: indeed - thanks for telling | 06:43 |
| *** cuongnv has quit IRC | 06:43 | |
| calebb | np | 06:43 |
| *** smcginnis has joined #openstack-infra | 06:43 | |
| *** kjackal_ has quit IRC | 06:43 | |
| rakhmerov | AJaeger: ok | 06:44 |
| AJaeger | infra-root, ask.o.o is down | 06:44 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Remove journal binary log https://review.openstack.org/482818 | 06:44 |
| *** andreas_s has joined #openstack-infra | 06:44 | |
| *** cuongnv has joined #openstack-infra | 06:44 | |
| *** d0ugal has quit IRC | 06:44 | |
| *** thorst has quit IRC | 06:46 | |
| *** openstackgerrit has quit IRC | 06:48 | |
| *** liujiong_lj has joined #openstack-infra | 06:50 | |
| AJaeger | #status notice FYI, ask.openstack.org is down, review.o.o is slow - please have patience until this is fixed | 06:50 |
| openstackstatus | AJaeger: sending notice | 06:50 |
| *** openstackgerrit has joined #openstack-infra | 06:50 | |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/shade master: Improve doc formatting a bit https://review.openstack.org/482819 | 06:50 |
| *** dhajare has joined #openstack-infra | 06:51 | |
| -openstackstatus- NOTICE: FYI, ask.openstack.org is down, review.o.o is slow - please have patience until this is fixed | 06:51 | |
| *** cuongnv has quit IRC | 06:53 | |
| openstackstatus | AJaeger: finished sending notice | 06:53 |
| *** liujiong has quit IRC | 06:54 | |
| *** cuongnv has joined #openstack-infra | 06:54 | |
| *** zhurong has quit IRC | 06:55 | |
| *** cuongnv has quit IRC | 07:03 | |
| *** ykarel_ has joined #openstack-infra | 07:04 | |
| berendt | looks like review.o.o is working normal now | 07:04 |
| *** cuongnv has joined #openstack-infra | 07:05 | |
| *** ykarel has quit IRC | 07:07 | |
| *** kjackal_ has joined #openstack-infra | 07:07 | |
| *** slaweq_ has joined #openstack-infra | 07:07 | |
| cmurphy | it's off and on | 07:09 |
| AJaeger | berendt: if it's in the usual garbage collection mess, you might be lucky if I scared enough people away ;) | 07:11 |
| *** tesseract has joined #openstack-infra | 07:11 | |
| *** slaweq_ has quit IRC | 07:12 | |
| *** shardy has joined #openstack-infra | 07:13 | |
| openstackgerrit | Merged openstack-infra/project-config master: Updating tripleo tempest periodic job https://review.openstack.org/477879 | 07:19 |
| openstackgerrit | Arx Cruz proposed openstack-infra/tripleo-ci master: Fix stackviz collect data https://review.openstack.org/474559 | 07:20 |
| openstackgerrit | Merged openstack-infra/project-config master: Add gating jobs to quickstart extras repo https://review.openstack.org/482588 | 07:20 |
| *** cuongnv has quit IRC | 07:22 | |
| openstackgerrit | Merged openstack-infra/project-config master: Skip non-doc jobs for doc-only python-ironic-inspector-client changes https://review.openstack.org/482451 | 07:23 |
| AJaeger | yolanda, here're a few small cleanups of mine that I'd like to be reviewed https://review.openstack.org/#/c/481806/ https://review.openstack.org/#/c/481892/ https://review.openstack.org/#/c/481951/ https://review.openstack.org/#/c/482117/ - and could you check review.o.o whether it's in Garbage collect mode again, please? | 07:24 |
| *** alexchadin has joined #openstack-infra | 07:30 | |
| *** pcaruana has joined #openstack-infra | 07:31 | |
| *** isaacb_ has quit IRC | 07:31 | |
| *** zhurong has joined #openstack-infra | 07:32 | |
| *** aarefiev_afk is now known as aarefiev | 07:32 | |
| *** egonzalez has joined #openstack-infra | 07:34 | |
| *** tuanluong has joined #openstack-infra | 07:36 | |
| *** mwarad has quit IRC | 07:37 | |
| *** ccamacho has joined #openstack-infra | 07:38 | |
| *** claudiub has joined #openstack-infra | 07:39 | |
| openstackgerrit | Sergey Kulanov proposed openstack-infra/jeepyb master: Ensure acl_config has absolute path https://review.openstack.org/482612 | 07:39 |
| *** pcaruana has quit IRC | 07:41 | |
| yolanda | hi AJaeger ... yep, review.o.o is very slow today | 07:44 |
| yolanda | i'll check it | 07:44 |
| *** amotoki_away is now known as amotoki | 07:45 | |
| *** pcaruana has joined #openstack-infra | 07:45 | |
| yolanda | #status notice Gerrit is going to be restarted, due to low performance | 07:48 |
| openstackstatus | yolanda: sending notice | 07:48 |
| -openstackstatus- NOTICE: Gerrit is going to be restarted, due to low performance | 07:49 | |
| AJaeger | yolanda: thanks | 07:49 |
| openstackstatus | yolanda: finished sending notice | 07:51 |
| yolanda | #status ok Gerrit has been successfully restarted | 07:53 |
| openstackstatus | yolanda: sending ok | 07:53 |
| *** sree has quit IRC | 07:54 | |
| *** jpich has joined #openstack-infra | 07:54 | |
| -openstackstatus- NOTICE: Gerrit has been successfully restarted | 07:56 | |
| *** sree_ has joined #openstack-infra | 07:58 | |
| *** sree_ is now known as Guest58976 | 07:58 | |
| openstackstatus | yolanda: finished sending ok | 07:59 |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:00 | |
| *** gouthamr has joined #openstack-infra | 08:02 | |
| openstackgerrit | Merged openstack-infra/project-config master: layout.yaml: Move mutex list last https://review.openstack.org/481806 | 08:02 |
| *** liujiong_lj is now known as liujiong | 08:02 | |
| *** Guest58976 has quit IRC | 08:03 | |
| *** sshnaidm|afk has quit IRC | 08:03 | |
| *** rossella_s has joined #openstack-infra | 08:04 | |
| *** sshnaidm|afk has joined #openstack-infra | 08:05 | |
| openstackgerrit | Merged openstack-infra/project-config master: Export IRONIC_USE_RESOURCE_CLASSES for Mogan https://review.openstack.org/482484 | 08:06 |
| openstackgerrit | Merged openstack-infra/project-config master: Set Tempest concurrency for non-multinode Patrole jobs https://review.openstack.org/482693 | 08:06 |
| *** rkukura has quit IRC | 08:08 | |
| *** slaweq_ has joined #openstack-infra | 08:08 | |
| *** cuongnv has joined #openstack-infra | 08:09 | |
| *** rkukura has joined #openstack-infra | 08:12 | |
| *** slaweq_ has quit IRC | 08:13 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add grafana devstack failure rate graph for openSUSE https://review.openstack.org/482706 | 08:14 |
| openstackgerrit | Merged openstack-infra/project-config master: Update zuul-status rendering for the Tempest/Grenade graphs https://review.openstack.org/482711 | 08:15 |
| openstackgerrit | Merged openstack-infra/project-config master: Add neutron-dynamic-routing periodic job with ryu master https://review.openstack.org/482340 | 08:16 |
| openstackgerrit | Merged openstack-infra/project-config master: Add fedora-26 support for nodepool dsvm jobs https://review.openstack.org/482566 | 08:17 |
| *** ralonsoh has joined #openstack-infra | 08:17 | |
| *** ykarel__ has joined #openstack-infra | 08:17 | |
| *** efoley has joined #openstack-infra | 08:17 | |
| *** makowals_ has joined #openstack-infra | 08:17 | |
| *** makowals has quit IRC | 08:20 | |
| *** ykarel_ has quit IRC | 08:20 | |
| openstackgerrit | Merged openstack-infra/project-config master: OSA: Add periodic test for translations scenario https://review.openstack.org/482120 | 08:21 |
| *** gouthamr has quit IRC | 08:21 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use python-jobs template for python-heatclient https://review.openstack.org/482434 | 08:22 |
| openstackgerrit | Merged openstack-infra/project-config master: jenkins: jobs: ansible-role-jobs: Check if yum-config-manager is present https://review.openstack.org/482465 | 08:22 |
| *** makowals has joined #openstack-infra | 08:22 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove oslo.* from the oslo group https://review.openstack.org/480667 | 08:23 |
| openstackgerrit | Merged openstack-infra/project-config master: Set up DLM in n-g-s tempest job https://review.openstack.org/482155 | 08:24 |
| *** makowals_ has quit IRC | 08:24 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add functional job for watcherclient https://review.openstack.org/480495 | 08:24 |
| openstackgerrit | Merged openstack-infra/project-config master: Add diskimage-builder-ci group https://review.openstack.org/479057 | 08:24 |
| *** makowals has quit IRC | 08:30 | |
| *** lucas-afk is now known as lucasagomes | 08:32 | |
| *** pcaruana has quit IRC | 08:33 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: Add settings for ovb in rdocloud https://review.openstack.org/480950 | 08:36 |
| *** makowals has joined #openstack-infra | 08:41 | |
| *** makowals has quit IRC | 08:41 | |
| *** thorst has joined #openstack-infra | 08:42 | |
| *** s-shiono has quit IRC | 08:43 | |
| *** makowals has joined #openstack-infra | 08:44 | |
| *** jascott1 has quit IRC | 08:44 | |
| *** dklyle has joined #openstack-infra | 08:45 | |
| *** jascott1 has joined #openstack-infra | 08:45 | |
| *** david-lyle has quit IRC | 08:45 | |
| *** d0ugal has joined #openstack-infra | 08:45 | |
| *** pcaruana has joined #openstack-infra | 08:46 | |
| *** Douhet has quit IRC | 08:46 | |
| *** thorst has quit IRC | 08:47 | |
| *** liujiong has quit IRC | 08:47 | |
| *** liujiong has joined #openstack-infra | 08:47 | |
| *** electrofelix has joined #openstack-infra | 08:48 | |
| *** raissa_ has joined #openstack-infra | 08:49 | |
| *** jascott1 has quit IRC | 08:49 | |
| *** jbadiapa_ has quit IRC | 08:50 | |
| *** katkapilatova has joined #openstack-infra | 08:51 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: Add settings for ovb in rdocloud https://review.openstack.org/480950 | 08:51 |
| *** jbadiapa_ has joined #openstack-infra | 08:52 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Case sensitive label matching https://review.openstack.org/482856 | 08:53 |
| *** d0ugal has quit IRC | 08:54 | |
| *** d0ugal has joined #openstack-infra | 08:55 | |
| *** d0ugal has quit IRC | 08:55 | |
| *** d0ugal has joined #openstack-infra | 08:55 | |
| *** makowals has quit IRC | 08:58 | |
| *** makowals has joined #openstack-infra | 09:04 | |
| *** Douhet has joined #openstack-infra | 09:05 | |
| *** iyamahat has joined #openstack-infra | 09:06 | |
| *** slaweq_ has joined #openstack-infra | 09:09 | |
| *** jpich has quit IRC | 09:09 | |
| *** jpich has joined #openstack-infra | 09:11 | |
| *** claudiub|2 has joined #openstack-infra | 09:11 | |
| *** claudiub has quit IRC | 09:14 | |
| *** slaweq_ has quit IRC | 09:14 | |
| *** liujiong has quit IRC | 09:15 | |
| *** iyamahat has quit IRC | 09:15 | |
| *** liujiong has joined #openstack-infra | 09:15 | |
| *** claudiub has joined #openstack-infra | 09:15 | |
| *** isaacb has joined #openstack-infra | 09:17 | |
| *** claudiub|2 has quit IRC | 09:18 | |
| *** tosky has joined #openstack-infra | 09:19 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Add support for custom ssh port https://review.openstack.org/468752 | 09:22 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: EOL ubuntu-precise for dsvm job https://review.openstack.org/482613 | 09:24 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove isJobRegistered https://review.openstack.org/482755 | 09:25 |
| *** e0ne has joined #openstack-infra | 09:30 | |
| *** pcaruana has quit IRC | 09:32 | |
| *** askb has quit IRC | 09:32 | |
| *** pcaruana has joined #openstack-infra | 09:32 | |
| *** yamahata has quit IRC | 09:34 | |
| *** kjackal__ has joined #openstack-infra | 09:34 | |
| openstackgerrit | Andy McCrae proposed openstack-infra/project-config master: Add os_tacker role to OpenStack-Ansible https://review.openstack.org/482873 | 09:36 |
| *** kjackal___ has joined #openstack-infra | 09:36 | |
| *** dargains has joined #openstack-infra | 09:37 | |
| *** kjackal_ has quit IRC | 09:37 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add job dependencies to status.json https://review.openstack.org/482061 | 09:37 |
| openstackgerrit | Amrith Kumar proposed openstack/diskimage-builder master: the current 08-debootstrap doesn't allow cross platform development https://review.openstack.org/482792 | 09:39 |
| *** sambetts|afk is now known as sambetts | 09:40 | |
| *** kjackal__ has quit IRC | 09:40 | |
| sshnaidm|afk | sorry for dumb question, but why do I see in repo commit like "Something changed" and then 1 day after commit from Jenkins "Merged Something changed"? | 09:42 |
| *** thorst has joined #openstack-infra | 09:43 | |
| openstackgerrit | Kien Nguyen proposed openstack-infra/project-config master: Setup multinode CI for Zun https://review.openstack.org/482877 | 09:46 |
| *** thorst has quit IRC | 09:47 | |
| openstackgerrit | Kien Nguyen proposed openstack-infra/project-config master: Setup multinode CI for Zun https://review.openstack.org/482877 | 09:47 |
| *** dargains has quit IRC | 09:48 | |
| openstackgerrit | Merged openstack-infra/shade master: Add Neutron QoS bandwidth limit rule commands https://review.openstack.org/482325 | 09:55 |
| openstackgerrit | Amrith Kumar proposed openstack/diskimage-builder master: the current 08-debootstrap doesn't allow cross platform development https://review.openstack.org/482792 | 09:59 |
| AJaeger | sshnaidm|afk: which repo? | 10:04 |
| AJaeger | best to explain on specific example | 10:04 |
| sshnaidm|afk | AJaeger, any, for example: https://github.com/openstack/tripleo-heat-templates/commits/master | 10:05 |
| *** isaacb has quit IRC | 10:06 | |
| sshnaidm|afk | AJaeger, "Switch from oslosphinx to openstackdocstheme" 15 days ago, and by Jenkins "Merge "Switch from oslosphinx to openstackdocstheme"" today | 10:06 |
| AJaeger | sshnaidm|afk: that'S the order those are merged in - the order is different from when they were initially submitted. | 10:06 |
| *** dizquierdo has joined #openstack-infra | 10:07 | |
| AJaeger | The Somethign changed is when you run "git commit" | 10:07 |
| AJaeger | The "Merged something" is when gerrit merged it after approval | 10:07 |
| sshnaidm|afk | oh, I see.. | 10:07 |
| sshnaidm|afk | and is this commit merged? "https://github.com/openstack/tripleo-heat-templates/commit/366825ec09ac44fe0440ed31685f5870032680a1" | 10:08 |
| sshnaidm|afk | because I don't see merge commit by Jenkins | 10:08 |
| *** isaacb has joined #openstack-infra | 10:09 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/project-config master: zuul: layout: Promote the openSUSE bifrost tinyipa job to gate https://review.openstack.org/482885 | 10:10 |
| *** slaweq_ has joined #openstack-infra | 10:10 | |
| *** isaacb has quit IRC | 10:10 | |
| *** wolverineav has quit IRC | 10:11 | |
| *** isaacb has joined #openstack-infra | 10:12 | |
| cmurphy | sshnaidm|afk: it's a fast-forward merge, it was a simple enough history that it didn't need to create a merge commit | 10:13 |
| *** slaweq_ has quit IRC | 10:14 | |
| *** slaweq_ has joined #openstack-infra | 10:14 | |
| *** tuanluong has quit IRC | 10:14 | |
| *** cuongnv has quit IRC | 10:14 | |
| sshnaidm|afk | cmurphy, thanks, so it's merged | 10:15 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Switch to tox for running tempest plugin sanity check https://review.openstack.org/474168 | 10:18 |
| *** isaacb has quit IRC | 10:18 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Add support for zuul-web and scheduler connections https://review.openstack.org/482889 | 10:19 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Remove function_cache variables from executor client https://review.openstack.org/482890 | 10:22 |
| openstackgerrit | Tomasz Trębski proposed openstack-infra/project-config master: Make Py35 gate votable for monasca-notification https://review.openstack.org/482892 | 10:22 |
| *** cshastri has quit IRC | 10:23 | |
| *** liujiong has quit IRC | 10:23 | |
| *** stakeda has quit IRC | 10:24 | |
| *** rossella_s has quit IRC | 10:26 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Add support for zuul-web and scheduler connections https://review.openstack.org/482889 | 10:26 |
| *** xarses_ has quit IRC | 10:27 | |
| *** dklyle has quit IRC | 10:28 | |
| *** rossella_s has joined #openstack-infra | 10:28 | |
| *** david-lyle has joined #openstack-infra | 10:28 | |
| *** alexchadin has quit IRC | 10:28 | |
| *** dargains has joined #openstack-infra | 10:29 | |
| *** xarses has joined #openstack-infra | 10:29 | |
| *** dargains has quit IRC | 10:34 | |
| *** dargains has joined #openstack-infra | 10:35 | |
| *** rossella_s has quit IRC | 10:36 | |
| *** zhurong has quit IRC | 10:41 | |
| pabelanger | fungi: clarkb: going to work in infracloud mirrors this morning | 10:42 |
| *** dargains has quit IRC | 10:43 | |
| *** dargains has joined #openstack-infra | 10:45 | |
| *** dargains_ has joined #openstack-infra | 10:47 | |
| *** xinliang has quit IRC | 10:48 | |
| *** rossella_s has joined #openstack-infra | 10:48 | |
| *** d0ugal has quit IRC | 10:49 | |
| *** dargains has quit IRC | 10:50 | |
| *** dargains_ has quit IRC | 10:57 | |
| AJaeger | morning pabelanger , you're up early | 10:57 |
| pabelanger | AJaeger: indeed, kids still jet lagged | 10:58 |
| AJaeger | ;( | 10:58 |
| openstackgerrit | Merged openstack-infra/shade master: Add flag to include all images in image list https://review.openstack.org/481702 | 10:59 |
| *** xinliang has joined #openstack-infra | 11:01 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Avoid using apt-add-repository https://review.openstack.org/482683 | 11:03 |
| *** jkilpatr has quit IRC | 11:06 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Add support to test fedora-26 https://review.openstack.org/482568 | 11:07 |
| *** wolverineav has joined #openstack-infra | 11:07 | |
| *** rhallisey has joined #openstack-infra | 11:07 | |
| *** markvoelker has quit IRC | 11:09 | |
| *** lucasagomes is now known as lucas-hungry | 11:10 | |
| *** markvoelker has joined #openstack-infra | 11:10 | |
| *** dargains has joined #openstack-infra | 11:11 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack-infra/project-config master: zuul: layout: Promote the openSUSE bifrost tinyipa job to gate https://review.openstack.org/482885 | 11:11 |
| AJaeger | mordred, pabelanger, here're a few small cleanups of mine that I'd like to be reviewed to cleanup the queue https://review.openstack.org/#/c/481892/ https://review.openstack.org/#/c/481951/ https://review.openstack.org/#/c/482117/ - could you push this on your review list, please? | 11:13 |
| pabelanger | +3 | 11:14 |
| mordred | ++ | 11:14 |
| *** markvoelker has quit IRC | 11:14 | |
| AJaeger | thanks to both of you! | 11:14 |
| *** slaweq_ has joined #openstack-infra | 11:15 | |
| pabelanger | clarkb: where did we end up with ipv6 on citycloud? Did we ever do your network test to see if it would work properly? | 11:16 |
| *** dargains has quit IRC | 11:19 | |
| *** dave-mccowan has joined #openstack-infra | 11:19 | |
| *** slaweq_ has quit IRC | 11:20 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove unused jobs for paunch https://review.openstack.org/481892 | 11:20 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove api-ref-src from api-ref job-group https://review.openstack.org/481951 | 11:21 |
| openstackgerrit | Merged openstack-infra/project-config master: Stop translation the cisco horizon plug-in https://review.openstack.org/482117 | 11:21 |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci master: DNM - hijack 3nodes job to test docker+HA https://review.openstack.org/482913 | 11:23 |
| *** isaacb has joined #openstack-infra | 11:25 | |
| openstackgerrit | Tomasz Trębski proposed openstack-infra/project-config master: Make Py35 gate votable for monasca-notification https://review.openstack.org/482892 | 11:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Remove default value for mirror_host https://review.openstack.org/482915 | 11:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Replace hand-written slug with ansible_managed https://review.openstack.org/482916 | 11:26 |
| *** LindaWang has quit IRC | 11:28 | |
| *** isaacb has quit IRC | 11:28 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Remove openstack-doc-build from zuul-jobs https://review.openstack.org/482921 | 11:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add openstack-docs-build to zuul-jobs https://review.openstack.org/482922 | 11:30 |
| mordred | pabelanger: ^^ those last two move openstack-docs-build from zuul-jobs to project-config | 11:31 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Enable verify_host for synchronize https://review.openstack.org/482655 | 11:32 |
| *** shardy is now known as shardy_lunch | 11:32 | |
| pabelanger | mordred: wait, how do we have 2 project definitions for openstack-infra/zuul-jobs? | 11:35 |
| pabelanger | I never knew that was valid | 11:35 |
| pabelanger | is this the new shadow jobs configuration? | 11:35 |
| pabelanger | Oh wait, I think I see | 11:35 |
| pabelanger | okay, was confused for a second | 11:36 |
| mordred | pabelanger: we can define things centrally in project-config - then a project can additinally add jobs to itself in its own zuul.yaml | 11:36 |
| mordred | pabelanger: it's a whole new thing :) | 11:36 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove openstack-doc-build from zuul-jobs https://review.openstack.org/482921 | 11:36 |
| *** xarses_ has joined #openstack-infra | 11:36 | |
| pabelanger | mordred: Ya, remember now. We had to do that to define the default pipelines also | 11:36 |
| mordred | pabelanger: this is how we can centrally mandate that all projects run pep8 and py27, but still let them add their own jobs locally :) | 11:36 |
| *** gordc has joined #openstack-infra | 11:36 | |
| pabelanger | little rusty on zuulv3 :D | 11:37 |
| mordred | yah | 11:37 |
| mordred | pabelanger: :) | 11:37 |
| *** ldnunes has joined #openstack-infra | 11:37 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Run zuul-web on zuulv3.o.o https://review.openstack.org/482926 | 11:37 |
| mordred | pabelanger: speaking of - https://review.openstack.org/#/c/482889/ and ^^ | 11:37 |
| mordred | pabelanger: should get us the websocket streaming running | 11:38 |
| *** thorst has joined #openstack-infra | 11:38 | |
| pabelanger | mordred: excellent | 11:39 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Remove legacy and unused code from tripleo-ci https://review.openstack.org/482929 | 11:39 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-openstackci master: DNM: Test spec helper changes https://review.openstack.org/481948 | 11:40 |
| *** xarses has quit IRC | 11:40 | |
| *** jkilpatr has joined #openstack-infra | 11:43 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add openstack-docs-build to zuul-jobs https://review.openstack.org/482922 | 11:44 |
| *** florianf has quit IRC | 11:45 | |
| *** isaacb has joined #openstack-infra | 11:51 | |
| *** markvoelker has joined #openstack-infra | 11:55 | |
| *** pradk has joined #openstack-infra | 11:55 | |
| *** florianf has joined #openstack-infra | 11:59 | |
| chandankumar | AJaeger: yolanda please have a look on this https://review.openstack.org/#/c/474168/ when you are free thanks :-) | 11:59 |
| *** alexchadin has joined #openstack-infra | 12:00 | |
| *** isaacb has quit IRC | 12:00 | |
| *** isaacb has joined #openstack-infra | 12:01 | |
| *** dprince has joined #openstack-infra | 12:05 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/project-config master: Make puppet-openstackci beaker jobs voting https://review.openstack.org/482938 | 12:06 |
| *** jtomasek_ has joined #openstack-infra | 12:07 | |
| *** lucas-hungry is now known as lucasagomes | 12:09 | |
| *** shardy_lunch is now known as shardy | 12:10 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Add support for a parallel set of secrets for connections https://review.openstack.org/482941 | 12:14 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Support fedora-26 for nodepool dsvm job https://review.openstack.org/482942 | 12:14 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder master: Bump fedora-minimal DIB_RELEASE 26 https://review.openstack.org/482570 | 12:15 |
| *** slaweq_ has joined #openstack-infra | 12:16 | |
| *** kjackal_ has joined #openstack-infra | 12:18 | |
| *** kjackal__ has joined #openstack-infra | 12:20 | |
| *** rlandy has joined #openstack-infra | 12:20 | |
| *** kjackal___ has quit IRC | 12:20 | |
| *** slaweq_ has quit IRC | 12:21 | |
| *** jamesden_ has joined #openstack-infra | 12:21 | |
| *** kjackal_ has quit IRC | 12:22 | |
| *** krtaylor has joined #openstack-infra | 12:23 | |
| *** jamesdenton has quit IRC | 12:24 | |
| *** jtomasek_ has quit IRC | 12:25 | |
| pabelanger | fungi: I think we might need to restore mysql.qcow2 for a day or so. Dealing with job failured on older branched of devstack currently | 12:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add MySQL and Github connection information for Zuul v3 https://review.openstack.org/482949 | 12:27 |
| pabelanger | fungi: gate-grenade-dsvm-multinode seems tobe failing on newton | 12:28 |
| fungi | pabelanger: okay, is there a revert? | 12:28 |
| pabelanger | fungi: no revert yet, I was thinking we could just add the qcow2 image back to tarballs.o.o for now do we can get DIBs building again | 12:29 |
| mordred | cmurphy: if you don't mind- would you look at https://review.openstack.org/482941 ? I did more templating in that than I normally do | 12:29 |
| pabelanger | at 3 days of failures because we are still attempting to download it | 12:29 |
| cmurphy | mordred: sure | 12:29 |
| mordred | cmurphy: there is an example of the data in https://review.openstack.org/482949 - the data in zuul_connections_secret is the same format | 12:30 |
| fungi | pabelanger: oh, now i get what you mean | 12:30 |
| cmurphy | mordred: the syntax job is already helping you | 12:30 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config master: Update ACLs for stable/pike in prep for release https://review.openstack.org/482951 | 12:31 |
| fungi | pabelanger: i believe the changes will merge fine if you depends-on in sequence all the way back to the one for the oldest branch | 12:31 |
| *** zzzeek_ has joined #openstack-infra | 12:32 | |
| mordred | cmurphy: woot! | 12:32 |
| fungi | pabelanger: but i can also just restore it for now | 12:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul master: Add support for a parallel set of secrets for connections https://review.openstack.org/482941 | 12:33 |
| pabelanger | fungi: I'm not sure I follow on your depends-on comment | 12:33 |
| pabelanger | also: http://logs.openstack.org/03/482603/2/check/gate-grenade-dsvm-multinode/76487d1/logs/grenade.sh.txt.gz#_2017-07-12_11_02_41_587 is the failure atm | 12:33 |
| pabelanger | mtreinish: ^ does that sound familiar for devstack failure? | 12:34 |
| pabelanger | this is on newton branch | 12:34 |
| fungi | pabelanger: i've temporarily restored that one image to it's original location | 12:35 |
| *** pgadiya has quit IRC | 12:35 | |
| pabelanger | fungi: great, ty | 12:35 |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config master: Update ACLs for stable/pike in prep for release https://review.openstack.org/482951 | 12:35 |
| fungi | as for depends-on between branches, this is common for sequencing upgrade fixes for grenade jobs | 12:35 |
| openstackgerrit | Akihiro Motoki proposed openstack-dev/hacking master: rearrange existing documentation to fit the new standard layout https://review.openstack.org/482537 | 12:35 |
| openstackgerrit | Akihiro Motoki proposed openstack-dev/hacking master: Switch from oslosphinx to openstackdocstheme https://review.openstack.org/482536 | 12:35 |
| openstackgerrit | Akihiro Motoki proposed openstack-dev/hacking master: Update the doc URL in the documents https://review.openstack.org/482538 | 12:35 |
| fungi | pabelanger: make the stable/newton change depends-on the stable/mitaka change (or maybe it's time to drop grenade jobs for newton?) | 12:36 |
| amotoki | is there any way to know which repository is specified in Depends-On? | 12:37 |
| pabelanger | fungi: right, I'll have to see what openstack-qa would like to do | 12:37 |
| amotoki | I am wondering how we can send a patch to neutron which depends on libraries like neutron-lib. | 12:37 |
| fungi | since the issue is that the newton grenade job starts by installing mitaka devstack which still wants to download that file, unless you depends-on the change which makes that no longer happen | 12:38 |
| pabelanger | fungi: okay, I think I see what you are saying | 12:38 |
| mordred | amotoki: you can just put the change-id in the depends-on | 12:39 |
| fungi | amotoki: not sure what you mean. a change can declare multiple depends-on change-ids, but also a single change-id can refer to multiple changes in different repositories | 12:39 |
| mordred | amotoki: it totally works across repos | 12:39 |
| *** gtmanfred has quit IRC | 12:40 | |
| *** zhenguo_ has joined #openstack-infra | 12:40 | |
| *** v1k0d3n_ has joined #openstack-infra | 12:40 | |
| *** persia__ has joined #openstack-infra | 12:40 | |
| *** hrybacki_ has joined #openstack-infra | 12:40 | |
| *** fmccrthy_ has joined #openstack-infra | 12:40 | |
| *** szaher has quit IRC | 12:40 | |
| *** squid has joined #openstack-infra | 12:40 | |
| fungi | amotoki: are you asking how to switch behavior between installing a dependent change from source vs installing the same library from a release? | 12:40 |
| *** zerick has quit IRC | 12:41 | |
| *** chandankumar has quit IRC | 12:41 | |
| amotoki | fungi: mordred: if we send a patch which depends on a library patch, upper-constraints enforces to use a release version. I woudl like to run edit-constraints if we have a lib patch in Depends-On . | 12:41 |
| *** yamamoto has quit IRC | 12:41 | |
| *** tbarron_ has joined #openstack-infra | 12:41 | |
| pabelanger | mordred: fungi: do you mind adding https://review.openstack.org/#/c/482592/ to your review pipeline. It brings fedora-26 mirrors online. I've confirmed we have the HDD space in afs | 12:41 |
| *** zbitter has joined #openstack-infra | 12:41 | |
| *** fmccrthy has quit IRC | 12:41 | |
| *** v1k0d3n has quit IRC | 12:41 | |
| *** DinaBelova has quit IRC | 12:41 | |
| *** rfolco has quit IRC | 12:41 | |
| *** vipul has quit IRC | 12:41 | |
| *** hrybacki has quit IRC | 12:41 | |
| *** zhenguo has quit IRC | 12:41 | |
| *** persia has quit IRC | 12:41 | |
| *** calebb has quit IRC | 12:41 | |
| *** fbouliane has quit IRC | 12:41 | |
| *** dprince has quit IRC | 12:41 | |
| *** zaneb has quit IRC | 12:41 | |
| *** numans has quit IRC | 12:41 | |
| *** eglute_s has quit IRC | 12:41 | |
| *** tbarron has quit IRC | 12:41 | |
| *** vaidy has quit IRC | 12:41 | |
| *** kuromagi has quit IRC | 12:41 | |
| *** loquacities has quit IRC | 12:41 | |
| *** melwitt has quit IRC | 12:41 | |
| *** rhagarty has quit IRC | 12:41 | |
| *** vkmc has quit IRC | 12:41 | |
| *** fmccrthy_ is now known as fmccrthy | 12:41 | |
| *** v1k0d3n_ is now known as v1k0d3n | 12:41 | |
| *** hrybacki_ is now known as hrybacki | 12:41 | |
| *** zhenguo_ is now known as zhenguo | 12:41 | |
| *** zerick has joined #openstack-infra | 12:41 | |
| *** chandankumar has joined #openstack-infra | 12:41 | |
| *** melwitt has joined #openstack-infra | 12:41 | |
| fungi | if so, you don't do that. you have different jobs for from source or from release packages and run both | 12:41 |
| *** vipul has joined #openstack-infra | 12:41 | |
| *** mrunge has quit IRC | 12:41 | |
| *** tbarron_ is now known as tbarron | 12:41 | |
| pabelanger | mordred: fungi: once online and things migrated to fedora-26, I'll remove the fedora-25 mirroring | 12:41 |
| mordred | amotoki: ah - that'sa whole different thing. you'll need to have a job that ^^ yah, what fungi said | 12:41 |
| *** numans has joined #openstack-infra | 12:41 | |
| *** melwitt is now known as Guest86208 | 12:41 | |
| *** eglute has joined #openstack-infra | 12:42 | |
| *** mrunge has joined #openstack-infra | 12:42 | |
| *** kuromagi has joined #openstack-infra | 12:42 | |
| *** DinaBelova has joined #openstack-infra | 12:42 | |
| *** gtmanfred has joined #openstack-infra | 12:42 | |
| *** fbouliane has joined #openstack-infra | 12:42 | |
| *** rfolco has joined #openstack-infra | 12:42 | |
| *** vkmc has joined #openstack-infra | 12:42 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Remove legacy and unused code from tripleo-ci https://review.openstack.org/482929 | 12:42 |
| amotoki | fungi: mordred: that solves the issue, but it wastes our infra resources in most cases. | 12:42 |
| *** rhagarty has joined #openstack-infra | 12:42 | |
| *** loquacities has joined #openstack-infra | 12:43 | |
| *** vaidy has joined #openstack-infra | 12:44 | |
| *** notmyname has quit IRC | 12:44 | |
| *** timburke has quit IRC | 12:44 | |
| fungi | amotoki: having a job selectively install proposed changes on branch tip sometimes but release versions other times can deadlock your repository | 12:44 |
| *** szaher has joined #openstack-infra | 12:44 | |
| *** dgonzalez has quit IRC | 12:44 | |
| fungi | amotoki: also you would require some fairly intractable logic to determine recursive dependencies in a series of changes | 12:45 |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: WIP: Adjust MTUs in multinode workflow on RDO Cloud https://review.openstack.org/481222 | 12:45 |
| andreas_s | pabelanger, clarkb Seems like the issue with the trove image still exists. The cache-devstack element code iterates over all branches https://github.com/openstack-infra/project-config/blob/master/nodepool/elements/cache-devstack/extra-data.d/55-cache-devstack-repos#L124 | 12:45 |
| *** dgonzalez has joined #openstack-infra | 12:46 | |
| *** timburke has joined #openstack-infra | 12:46 | |
| * amrith tunes in when he hears the word trove | 12:46 | |
| amrith | what did we do now? | 12:46 |
| amotoki | fungi: ah I see. it seems more complicated than I thought. | 12:46 |
| fungi | amotoki: zuul precalculates all that for the job and presents a consistently-named git revision, but to make use of that the job would need to install each repository you're concerned about through devstack-gate or zuul-cloner | 12:46 |
| pabelanger | andreas_s: yes, still working on patches to devstack, for now the image is back online so we can build images | 12:47 |
| *** notmyname has joined #openstack-infra | 12:47 | |
| * amrith wanders away ... that trove ... | 12:47 | |
| pabelanger | andreas_s: once I can confirm we can build DIBs with the image still on tarballs.o.o, I'll ask fungi to remove it again | 12:47 |
| *** dprince has joined #openstack-infra | 12:47 | |
| *** eharney has joined #openstack-infra | 12:47 | |
| fungi | amrith: stable branches of devstack were still insisting on downloading trove images | 12:47 |
| amrith | uh oh, sorry about that | 12:48 |
| andreas_s | pabelanger, ah great - I still was seeing the issue in my thirdparty ci - but let me check again | 12:48 |
| amrith | do you want to revert them in a different place? | 12:48 |
| pabelanger | andreas_s: Ya, we just restored in about 20mins go | 12:48 |
| pabelanger | amrith: we currently need to land: https://review.openstack.org/#/c/482603/ | 12:48 |
| fungi | amrith: i temporarily moved the one it uses back where devstack is looking for it until we can get devstack reviewers to approve the stable branch changes to rip it out | 12:49 |
| pabelanger | amrith: trying to determine the best path forward on grenade job | 12:49 |
| amrith | pabelanger fungi just restore the file on tarballs.o.o | 12:49 |
| andreas_s | pabelanger, yep, seems to work again... | 12:49 |
| amrith | and land the patch | 12:49 |
| amrith | we now know what falls apart when we remove t.o.o | 12:49 |
| amrith | we an remove it at a later date | 12:49 |
| amrith | sorry for the confusion, I didn't know that things outside trove used our turds | 12:50 |
| * amrith grins ... sorry, images | 12:50 | |
| fungi | amrith: i think devstack used to unconditionally download it in case trove got enabled | 12:50 |
| andreas_s | pabelanger, I guess we also need to fix that for the liberty kilo and mitaka branches (not sure why we still have them...) | 12:51 |
| fungi | i'll be surprised if those can even still run jobs | 12:52 |
| pabelanger | Ya, noticed devstack-cache was still hitting those branches | 12:52 |
| pabelanger | need to see if they actually add images | 12:53 |
| *** jaypipes has joined #openstack-infra | 12:53 | |
| pabelanger | or just EOL them? | 12:53 |
| *** dougs2 has left #openstack-infra | 12:54 | |
| *** sanket_ has quit IRC | 12:54 | |
| openstackgerrit | Attila Darazs proposed openstack-infra/puppet-openstackci master: Add a static footer for TripleO Quickstart logs https://review.openstack.org/482210 | 12:55 |
| pabelanger | fungi: https://review.openstack.org/#/c/450818/ seems to be related | 12:56 |
| *** xinliang has quit IRC | 12:56 | |
| *** d0ugal has joined #openstack-infra | 12:56 | |
| pabelanger | fungi: looking to see if it applies to stable branches | 12:56 |
| *** xinliang has joined #openstack-infra | 12:57 | |
| *** kgiusti has joined #openstack-infra | 12:57 | |
| tosky | uhm, are there connection issues for tarballs.o.o? ( Failed to connect to tarballs.openstack.org port 80: Connection timed out) | 12:57 |
| tosky | oh, no, sorry | 12:57 |
| tosky | ignore me | 12:57 |
| tosky | old logs | 12:57 |
| *** tushar has joined #openstack-infra | 12:58 | |
| pabelanger | tosky: we've been having issues recently but fungi is working to fix them | 12:58 |
| tosky | pabelanger: yes, and the logs tricked me | 12:58 |
| tosky | sorry for the confusion | 12:58 |
| *** tushar has quit IRC | 12:58 | |
| *** tushar has joined #openstack-infra | 12:58 | |
| *** ralonsoh has quit IRC | 12:58 | |
| *** ralonsoh_ has joined #openstack-infra | 12:58 | |
| *** tushar has quit IRC | 12:58 | |
| *** tushar has joined #openstack-infra | 12:59 | |
| *** LindaWang has joined #openstack-infra | 12:59 | |
| *** tushar has quit IRC | 12:59 | |
| *** Goneri has joined #openstack-infra | 12:59 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Cleanup components doc https://review.openstack.org/482636 | 12:59 |
| *** ralonsoh_ is now known as ralonsoh | 13:00 | |
| fungi | fwiw, i'm still keeping an eye on http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=311&rra_id=all until we can cut jobs over to using the local tarballs proxy caches | 13:01 |
| pabelanger | fungi: question, how does devstack install stable/mitake of keystone, when I don't see any branch on git.o.o? http://logs.openstack.org/03/482603/2/check/gate-grenade-dsvm-multinode/76487d1/logs/old/devstacklog.txt.gz#_2017-07-12_11_01_25_737 | 13:01 |
| fungi | pabelanger: i don't think it can, unless someone has been updating devstack to switch from branch names to eol tags (or unless dib isn't deleting cached branches when they disappear from the remote?) | 13:03 |
| pabelanger | fungi: http://logs.openstack.org/03/482603/2/check/gate-grenade-dsvm-multinode/76487d1/logs/pip2-freeze.txt.gz seems to imply to used the master branch | 13:04 |
| fungi | pabelanger: yep! that's the fallback behavior when it can't find the requested branch | 13:05 |
| *** Julien-zte has joined #openstack-infra | 13:05 | |
| pabelanger | fungi: okay, ya so it looks like grenade job is indeed broken on newton. I guess it is time to remove or update devstack to use eol tags? | 13:05 |
| fungi | basically there's no point in keeping stable/mitaka around for devstack once the branch is eol on the services it's using | 13:05 |
| pabelanger | fungi: okay, let me remove job and have mtreinish review | 13:06 |
| fungi | while it may run, it won't be testing what you expect | 13:06 |
| pabelanger | ya | 13:06 |
| *** kjackal__ has quit IRC | 13:06 | |
| pabelanger | seems to be that | 13:06 |
| pabelanger | infact, I think gate-grenade-dsvm-multinode can now be deleted, seems to only run on newton | 13:08 |
| *** yamamoto has joined #openstack-infra | 13:08 | |
| *** mdrabe has joined #openstack-infra | 13:10 | |
| *** zzzeek_ has quit IRC | 13:12 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove gate-grenade-dsvm-multinode job https://review.openstack.org/482962 | 13:13 |
| *** zzzeek_ has joined #openstack-infra | 13:13 | |
| pabelanger | fungi: mtreinish: ^ When you have time to review | 13:13 |
| *** yamamoto has quit IRC | 13:13 | |
| *** zhurong has joined #openstack-infra | 13:14 | |
| pabelanger | andreaf: ^ might be interested as well | 13:14 |
| *** slaweq_ has joined #openstack-infra | 13:17 | |
| pabelanger | fungi: clarkb: 15.184.66.172 is our new mirror in infracloud-vanilla | 13:17 |
| pabelanger | fungi: clarkb: I'm going to start the process of updating DNS records to point to it | 13:18 |
| *** d0ugal has quit IRC | 13:19 | |
| *** isaacb has quit IRC | 13:19 | |
| pabelanger | #status log DNS for mirror.regionone.infracloud-vanilla.openstack.org changed to 15.184.66.172, 60min TTL | 13:22 |
| openstackstatus | pabelanger: finished logging | 13:22 |
| *** slaweq_ has quit IRC | 13:22 | |
| fungi | pabelanger: awesome, thanks! disk space lgtm | 13:23 |
| *** rossella_s has quit IRC | 13:23 | |
| pabelanger | fungi: i'll wait until I see some traffic to new server before working on infracloud-chocolate | 13:23 |
| fungi | sounds great | 13:23 |
| *** michaelxin has joined #openstack-infra | 13:25 | |
| pabelanger | starting to see traffic to the new server now | 13:27 |
| pabelanger | nc 15.184.66.182 19885 | 13:27 |
| *** isaacb has joined #openstack-infra | 13:28 | |
| *** baoli has joined #openstack-infra | 13:28 | |
| pabelanger | k, switching to infracloud-chocolate now | 13:28 |
| *** zzzeek_ has quit IRC | 13:29 | |
| *** frickler has quit IRC | 13:33 | |
| *** zzzeek_ has joined #openstack-infra | 13:34 | |
| *** bobh has joined #openstack-infra | 13:37 | |
| *** rossella_s has joined #openstack-infra | 13:38 | |
| *** armax has joined #openstack-infra | 13:39 | |
| *** tushar has joined #openstack-infra | 13:40 | |
| tushar | Hi All | 13:40 |
| *** frickler has joined #openstack-infra | 13:41 | |
| tushar | I have setup a new third party CI | 13:41 |
| tushar | facing some issue in jenkins slave | 13:41 |
| tushar | I am not able to see any jenkins slave on jenkins UI | 13:41 |
| tushar | can anybody tell what I am missing here ? | 13:41 |
| pabelanger | tushar: nodepool debug.log should help explain what is going on. | 13:42 |
| pabelanger | is your nodes being launched in your cloud properly? | 13:42 |
| *** trown is now known as trown|brb | 13:42 | |
| tushar | Pabelanger : node is launched and is in ready state | 13:42 |
| pabelanger | tushar: okay, so you should see nodepool handing off the node to jenkins | 13:43 |
| pabelanger | tushar: then, jenkins will attempt to SSH into said node, using SSH key that has been setup in jenkins | 13:43 |
| pabelanger | but logs in nodepool will show this process and help debug where the failure is | 13:44 |
| tushar | ok..checking the nodepool logs | 13:44 |
| *** coolsvap has quit IRC | 13:45 | |
| fungi | worth noting, this sounds a lot like bkero's issue where he needed python-jenkins master branch to use nodepool with a recent (in the last few months) jenkins release | 13:46 |
| *** mriedem has joined #openstack-infra | 13:46 | |
| fungi | jenkins changed some api calls in backward-incompatible ways recently | 13:46 |
| pabelanger | agree, that is what I am thinking too | 13:46 |
| pabelanger | IIRC, is was due to a security issue | 13:47 |
| *** trown|brb is now known as trown | 13:47 | |
| *** jascott1 has joined #openstack-infra | 13:47 | |
| clarkb | fungi: I think the issue ended up being fixed by using latest nodepool which uses python-jenkins rather than the old nodepool vode wrapped around python-jenkins that eventually got into python-jenkins and was improves there | 13:51 |
| *** yamamoto has joined #openstack-infra | 13:51 | |
| clarkb | tldr use latest nodepool | 13:51 |
| *** mat128 has joined #openstack-infra | 13:51 | |
| *** jascott1 has quit IRC | 13:52 | |
| *** mat128 has quit IRC | 13:54 | |
| *** adarazs has joined #openstack-infra | 13:55 | |
| *** xyang1 has joined #openstack-infra | 13:56 | |
| *** frickler has quit IRC | 13:57 | |
| *** ZZelle has quit IRC | 13:57 | |
| adarazs | hi folks, I'm trying to get this change to pass the gates: https://review.openstack.org/482210 (openstack-infra/puppet-openstackci project) and it seems "gate-openstackci-beaker-centos-7" is failing with something not relevant to my change, can somebody help me here? | 13:58 |
| adarazs | "Gem::RemoteFetcher::UnknownHostError: no such name" -> http://logs.openstack.org/10/482210/2/check/gate-openstackci-beaker-centos-7/6f445f7/console.html#_2017-07-12_13_30_21_570246 | 13:58 |
| openstackgerrit | Rabi Mishra proposed openstack-infra/project-config master: Modify some heat related projects to use standard templates https://review.openstack.org/482973 | 14:01 |
| *** jcoufal has joined #openstack-infra | 14:02 | |
| *** frickler has joined #openstack-infra | 14:03 | |
| adarazs | hm, it passed before, so it's probably some temporary issue, I'll recheck and hope for the best. | 14:04 |
| *** gcb has joined #openstack-infra | 14:04 | |
| *** ykarel__ is now known as ykarel | 14:05 | |
| *** hongbin has joined #openstack-infra | 14:06 | |
| fungi | adarazs: yes, sometimes (like anywhere else on the internet) dns lookups fail | 14:08 |
| fungi | so it was probably temporary | 14:08 |
| *** rbrndt has joined #openstack-infra | 14:09 | |
| *** ykarel is now known as ykarel|afk | 14:10 | |
| tushar | pabelanger : nothing much I found in nodepool and jenkins log | 14:12 |
| tushar | in nodepool logs I can see " node 1234 is ready " | 14:12 |
| *** mat128 has joined #openstack-infra | 14:13 | |
| pabelanger | tushar: do you see nodepool telling jenkins to add the node as a slave? | 14:13 |
| tushar | after this i should see " Node id: 12345 added to jenkins " | 14:13 |
| tushar | but its missing | 14:13 |
| tushar | and there is no error message as well | 14:14 |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: WIP: Adjust MTUs in multinode workflow on RDO Cloud https://review.openstack.org/481222 | 14:14 |
| pabelanger | tushar: okay, so this sounds like the issue clarkb and fungi were just talking about. Likely need to update nodepool per clarkb suggestion | 14:14 |
| *** ykarel|afk has quit IRC | 14:14 | |
| tushar | Pabelanger : you mean need to upgrade nodepool version? | 14:15 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Correct run job parameter documentation https://review.openstack.org/482977 | 14:15 |
| pabelanger | tushar: ya | 14:15 |
| tushar | Pabelnger : oky | 14:16 |
| tushar | but I am using old versions of nodepool as well as jenkins | 14:16 |
| *** makowals has quit IRC | 14:16 | |
| tushar | anyway I will try by upgrading nodepool | 14:16 |
| *** arxcruz has quit IRC | 14:17 | |
| clarkb | ya new nodepool uses python-jenkins a bit more whovh handles jenkins behavior better | 14:18 |
| *** arxcruz has joined #openstack-infra | 14:18 | |
| *** slaweq_ has joined #openstack-infra | 14:18 | |
| *** makowals has joined #openstack-infra | 14:19 | |
| tushar | clarkb : new nodepool means 0.3.1 version? | 14:20 |
| *** tjones has joined #openstack-infra | 14:20 | |
| openstackgerrit | zhurong proposed openstack-infra/project-config master: Remove the outdate murano-ci Third-Party CI https://review.openstack.org/482980 | 14:20 |
| clarkb | tushar: 0.4.1 I think | 14:20 |
| tushar | clarkb: with this version of nodepool ...which version of jnkins is compatible? | 14:21 |
| clarkb | tushar: whatever versions python-jenkins supports | 14:22 |
| tushar | ok | 14:22 |
| tushar | clarkb : will try this...thanks | 14:22 |
| *** wolverineav has quit IRC | 14:22 | |
| *** slaweq_ has quit IRC | 14:23 | |
| *** zzzeek has quit IRC | 14:24 | |
| cmurphy | if anyone's interested i did some beaker refactoring that i'm hoping will make overall puppet reviews easier https://review.openstack.org/#/q/topic:refactor-infra-beaker | 14:25 |
| fungi | clarkb: though nodepool 0.4.1 _does_ require also installing zookeeper, right? | 14:26 |
| fungi | cmurphy: thanks! | 14:26 |
| *** camunoz has joined #openstack-infra | 14:26 | |
| clarkb | fungi: yes it does | 14:28 |
| *** dizquierdo has quit IRC | 14:29 | |
| mnaser | really throwing this out here but would zuulv3 be ready enough to use for simple periodic jobs? (we are puppet-izing all the things and automation for jenkins has been hell) | 14:33 |
| mnaser | the jobs are pretty much just poll a url to check if new images are released by upstream (ex: ubuntu cloud images) and then download them, test them and publish them | 14:33 |
| *** Julien-zte has quit IRC | 14:34 | |
| fungi | mnaser: maybe, but asking in #zuul is probably better. biggest issues are the syntax is still shifting sands to some extent and security isolation for workers is still being hammered out | 14:34 |
| mnaser | oh, didnt know there was a channel for it now, thought that was still in here | 14:34 |
| *** gouthamr has joined #openstack-infra | 14:35 | |
| fungi | here is good for general operational discussions but zuul v3 development state is probably more on topic in #zuul until it's released | 14:35 |
| *** Julien-zte has joined #openstack-infra | 14:35 | |
| *** Julien-zte has quit IRC | 14:35 | |
| *** wolverineav has joined #openstack-infra | 14:36 | |
| fungi | you're likely to get more complete info in there about what's still missing/broken/changing | 14:36 |
| pabelanger | mnaser: I think it is ready for experimenting, but not sure about production just yet. We've been getting good fixes for zuulv3 from people running it locally for testing so far | 14:36 |
| *** Julien-zte has joined #openstack-infra | 14:36 | |
| *** Julien-zte has quit IRC | 14:36 | |
| fungi | we have it running today in an advisory capacity self-testing changes proposed to the zuul repo and some zuulv3-specific configuration repos so it _does_ run, but we're a couple months out from where we expect to be able to reasonably switch our load and use cases to it | 14:37 |
| *** Julien-zte has joined #openstack-infra | 14:37 | |
| *** Julien-zte has quit IRC | 14:37 | |
| *** zbitter is now known as zaneb | 14:37 | |
| *** Julien-zte has joined #openstack-infra | 14:38 | |
| mnaser | fungi i've figured that if it can self-test at this point, it's probably more than enough to do what we're looking to do out of it | 14:38 |
| *** Julien-zte has quit IRC | 14:38 | |
| * mnaser will do some more reading | 14:38 | |
| *** ykarel has joined #openstack-infra | 14:38 | |
| *** Julien-zte has joined #openstack-infra | 14:38 | |
| *** Julien-zte has quit IRC | 14:39 | |
| openstackgerrit | Tomasz Trębski proposed openstack-infra/project-config master: Make Py35 gate votable for monasca-notification https://review.openstack.org/482892 | 14:39 |
| fungi | for example, i couldn't tell you if periodic triggers are working yet but the people who are paying close attention to #zuul likely know | 14:39 |
| *** Julien-zte has joined #openstack-infra | 14:39 | |
| *** Julien-zte has quit IRC | 14:39 | |
| pabelanger | Ya, we don't actually have a periodic job running on zuulv3.o.o yet | 14:40 |
| *** Julien-zte has joined #openstack-infra | 14:40 | |
| *** Julien-zte has quit IRC | 14:40 | |
| *** Julien-zte has joined #openstack-infra | 14:41 | |
| *** Julien-zte has quit IRC | 14:41 | |
| *** dingyichen has quit IRC | 14:42 | |
| *** Julien-zte has joined #openstack-infra | 14:42 | |
| *** Julien-zte has quit IRC | 14:42 | |
| *** dizquierdo has joined #openstack-infra | 14:44 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Updated Eventbrite permission model https://review.openstack.org/482730 | 14:45 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Updated Eventbrite permission model https://review.openstack.org/482730 | 14:46 |
| pabelanger | #status log DNS for mirror.regionone.infracloud-chocolate.openstack.org changed to 15.184.69.112, 60min TTL | 14:46 |
| openstackstatus | pabelanger: finished logging | 14:46 |
| pabelanger | fungi: clarkb: new infracloud-chocolate mirror online, just waiting for traffic to start hitting it | 14:46 |
| ianychoi | clarkb, fungi: Hello! I wanted to attend to last infra team meeting but it was too difficult to wake up at 4am unfortunately. If you have time, would you see http://lists.openstack.org/pipermail/openstack-infra/2017-July/005494.html and support with higher priority? | 14:49 |
| *** nicolasbock has quit IRC | 14:50 | |
| pabelanger | nc 15.184.69.142 19885 | 14:52 |
| pabelanger | is now hitting new mirror | 14:52 |
| *** zhurong has quit IRC | 14:53 | |
| *** isaacb has quit IRC | 14:54 | |
| *** psachin has quit IRC | 14:54 | |
| fungi | pabelanger: disk space looks good there now too. i'll get the new apache cache locations set up and warmed on them and then un-wip the bottom of the change stack | 14:54 |
| pabelanger | fungi: great, I think we can now delete the original vanilla mirror, I haven't see any traffic for a bit now | 14:55 |
| pabelanger | chocolate will need another 60mins or so | 14:55 |
| *** dtantsur is now known as dtantsur|afk | 14:55 | |
| pabelanger | and deleted | 14:57 |
| inc0 | good morning | 14:57 |
| *** david-lyle has quit IRC | 14:58 | |
| *** david-lyle has joined #openstack-infra | 14:58 | |
| *** prateek has quit IRC | 14:59 | |
| *** marst has joined #openstack-infra | 15:00 | |
| fungi | pabelanger: i've approved https://review.openstack.org/482228 and https://review.openstack.org/482229 now that the caches are all ready | 15:01 |
| fungi | rather, cache locations | 15:01 |
| pabelanger | fungi: Oh, I still need to SSH into new servers from puppetmaster | 15:01 |
| *** rcernin has quit IRC | 15:01 | |
| pabelanger | let me do that now | 15:01 |
| openstackgerrit | Kien Nguyen proposed openstack-infra/project-config master: Setup multinode CI for Zun https://review.openstack.org/482877 | 15:01 |
| fungi | pabelanger: thanks, the changes definitely won't take effect until that happens | 15:02 |
| *** zzzeek_ has quit IRC | 15:02 | |
| pabelanger | fungi: done, new host keys accepted | 15:02 |
| fungi | approved https://review.openstack.org/482203 just now too, so once those are confirmed working inc0 should be able to recheck his change to use the proxy for tarballs access | 15:03 |
| inc0 | fungi: just let me know | 15:04 |
| *** zzzeek has joined #openstack-infra | 15:04 | |
| fungi | inc0: will do. we should be very close now | 15:04 |
| *** makowals has quit IRC | 15:04 | |
| inc0 | cool, thanks! | 15:04 |
| *** felipemonteiro has joined #openstack-infra | 15:07 | |
| *** felipemonteiro_ has joined #openstack-infra | 15:08 | |
| pabelanger | fungi: did we update /etc/mirrors.sh with new proxy info? | 15:08 |
| *** squid is now known as calebb | 15:09 | |
| fungi | pabelanger: oh, i guess we could. i didn't think to add anything to it | 15:09 |
| pabelanger | fungi: ya, we have both docker / rdo proxies listed today | 15:09 |
| *** nicolasbock has joined #openstack-infra | 15:10 | |
| pabelanger | eg: NODEPOOL_RDO_PROXY | 15:10 |
| pabelanger | we should likely add NODEPOOL_TARBALLS_PROXY | 15:10 |
| pabelanger | I can propose that now | 15:10 |
| *** makowals has joined #openstack-infra | 15:10 | |
| fungi | yeah, i just found where we're adding that | 15:10 |
| openstackgerrit | Merged openstack-infra/system-config master: Move the per-region proxy caches into /var/cache https://review.openstack.org/482228 | 15:10 |
| pabelanger | fungi: cool | 15:10 |
| fungi | go for it, though inc0's patch is just hardcoding the reference to http://$NODEPOOL_MIRROR_HOST:8080/tarballs | 15:11 |
| fungi | so that kolla won't need to wait for new images | 15:11 |
| fungi | (new nodepool images, i should clarify) | 15:11 |
| pabelanger | right, they should be able to update once we get images online in a day or so | 15:11 |
| openstackgerrit | Merged openstack-infra/system-config master: Increase per-region caching proxy capacity https://review.openstack.org/482229 | 15:12 |
| *** felipemonteiro has quit IRC | 15:12 | |
| openstackgerrit | Merged openstack-infra/system-config master: Caching proxies for tarballs.openstack.org https://review.openstack.org/482203 | 15:13 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add NODEPOOL_TARBALLS_PROXY to configure_mirror.sh https://review.openstack.org/483006 | 15:14 |
| *** isaacb has joined #openstack-infra | 15:16 | |
| *** slaweq_ has joined #openstack-infra | 15:19 | |
| *** prateek has joined #openstack-infra | 15:20 | |
| *** slaweq_ has quit IRC | 15:24 | |
| *** alexchadin has quit IRC | 15:25 | |
| *** makowals has quit IRC | 15:32 | |
| *** mdrabe has quit IRC | 15:33 | |
| *** ldnunes has quit IRC | 15:33 | |
| calebb | pabelanger: hey, I was taking a look at updating the puppet install script to know how to install puppet 4, and was trying to update the fedora stuff but wanted to ask if the fedora stuff is still used/needed | 15:34 |
| *** coolsvap has joined #openstack-infra | 15:34 | |
| *** Sukhdev has joined #openstack-infra | 15:34 | |
| *** yamahata has joined #openstack-infra | 15:35 | |
| pabelanger | calebb: ya, we can likely drop that now, since we don't use fedora on control plane servers. The funny thing, is that is already setup to use puppet 4 | 15:36 |
| pabelanger | calebb: ubuntu and centos-7 are the operating systems we have | 15:36 |
| calebb | ahh, heh, ok thanks! | 15:37 |
| *** makowals has joined #openstack-infra | 15:37 | |
| *** LindaWang has quit IRC | 15:37 | |
| *** andreas_s has quit IRC | 15:38 | |
| *** mdrabe has joined #openstack-infra | 15:38 | |
| smcginnis | A normal cinder stable core pointed out to me that they don't have +2 on Ocata branches. | 15:39 |
| smcginnis | I think I remember we do a limited group during RC. Was something missed after final cut? | 15:39 |
| fungi | smcginnis: looks possible http://git.openstack.org/cgit/openstack-infra/project-config/tree/gerrit/acls/openstack/cinder.config#n19 | 15:41 |
| *** ccamacho has left #openstack-infra | 15:41 | |
| fungi | smcginnis: oh, though that's consistent across other projects' acls | 15:41 |
| *** liusheng has quit IRC | 15:42 | |
| fungi | i think it has more to do with the release team's scripts to update group memberships on cinder-release-branch | 15:42 |
| smcginnis | fungi: Hmm. Does the */ocata* section just need to be removed? | 15:42 |
| openstackgerrit | Caleb Boylan proposed openstack-infra/system-config master: (WIP) Get install_puppet.sh to install puppet 4 https://review.openstack.org/481955 | 15:42 |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: WIP: Adjust MTUs in multinode workflow on RDO Cloud https://review.openstack.org/481222 | 15:43 |
| smcginnis | Should now fall under the normal ref/heads/stable/* where the cinder-stable-maint group is. | 15:43 |
| *** liusheng has joined #openstack-infra | 15:43 | |
| fungi | smcginnis: i think that happens when the stable/pike section is created, but maybe ask in #openstack-release once the release team meeting concludes | 15:43 |
| fungi | https://review.openstack.org/#/admin/groups/cinder-release-branch looks similar to https://review.openstack.org/#/admin/groups/nova-release-branch anyway | 15:44 |
| smcginnis | fungi: OK, will do. Thanks for the pointers. | 15:44 |
| *** ldnunes has joined #openstack-infra | 15:45 | |
| fungi | smcginnis: oh! this showed up in my review queue today... https://review.openstack.org/482951 | 15:46 |
| *** jtomasek_ has joined #openstack-infra | 15:47 | |
| *** yamamoto has quit IRC | 15:48 | |
| *** yamamoto has joined #openstack-infra | 15:49 | |
| *** jcoufal has quit IRC | 15:49 | |
| *** yamamoto has quit IRC | 15:51 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix race in test_shadow https://review.openstack.org/483016 | 15:51 |
| *** iyamahat has joined #openstack-infra | 15:51 | |
| *** lucasagomes is now known as lucas-afk | 15:52 | |
| *** jtomasek_ has quit IRC | 15:53 | |
| *** ihrachys has quit IRC | 15:54 | |
| *** ihrachys has joined #openstack-infra | 15:55 | |
| *** amotoki is now known as amotoki_away | 15:56 | |
| *** egonzalez has quit IRC | 15:57 | |
| *** amotoki_away is now known as amotoki | 15:59 | |
| *** amotoki is now known as amotoki_away | 15:59 | |
| *** Rockyg has joined #openstack-infra | 15:59 | |
| *** jamesden_ has quit IRC | 15:59 | |
| *** jascott1 has joined #openstack-infra | 16:00 | |
| *** jamesdenton has joined #openstack-infra | 16:01 | |
| *** esberglu has quit IRC | 16:04 | |
| *** jcoufal has joined #openstack-infra | 16:04 | |
| *** tesseract has quit IRC | 16:05 | |
| *** isaacb has quit IRC | 16:07 | |
| *** gcb has quit IRC | 16:09 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-roles master: Create executor-info role https://review.openstack.org/483022 | 16:09 |
| fungi | any available project-config-core reviewers for a second +2 and approval on https://review.openstack.org/482951 ? it fixes a bit of a stable/ocata branch review emergency since a couple steps in the release management process were performed too close together | 16:11 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add debug info about which executor a job is run from https://review.openstack.org/483023 | 16:11 |
| *** vhosakot has joined #openstack-infra | 16:12 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-roles master: Create executor-info role https://review.openstack.org/483022 | 16:13 |
| *** trown is now known as trown|lunch | 16:13 | |
| *** annegentle has joined #openstack-infra | 16:14 | |
| *** esberglu has joined #openstack-infra | 16:18 | |
| openstackgerrit | Thierry Carrez proposed openstack-infra/project-config master: Remove stable/ocata ACL for os-apply-config https://review.openstack.org/483024 | 16:18 |
| *** slaweq_ has joined #openstack-infra | 16:21 | |
| *** mat128 has quit IRC | 16:21 | |
| *** esberglu has quit IRC | 16:22 | |
| *** shardy has quit IRC | 16:25 | |
| *** slaweq_ has quit IRC | 16:25 | |
| *** rossella_s has quit IRC | 16:26 | |
| bkero | clarkb: so it looks like the SSHLauncher jenkins plugin added a default behavior of deny while connecting to hosts without known ssh host keys. They added a toggle for that, and I think I can add that as a launcher param, but it doesn't seem to be working. http://paset.openstack.org/show/615185/ | 16:26 |
| *** esberglu has joined #openstack-infra | 16:27 | |
| *** jpich has quit IRC | 16:27 | |
| bkero | jeblair: would you still happen to remember enough jenkins_manager.py to know how to reference a non-string/int param? | 16:27 |
| pabelanger | #status log both mirrors in infracloud-chocolate and infracloud-vanilla replaced with 250GB HDD mirror flavors now. | 16:28 |
| openstackstatus | pabelanger: finished logging | 16:28 |
| pabelanger | fungi: clarkb: finished with mirror migration | 16:28 |
| *** ihrachys has quit IRC | 16:28 | |
| pabelanger | old server deleted too | 16:28 |
| *** ihrachys has joined #openstack-infra | 16:29 | |
| fungi | pabelanger: thanks! i'm checking the proxy now | 16:29 |
| pabelanger | fungi: great | 16:30 |
| fungi | clarkb: is mirror.la1.citycloud in active use? i don't see it in cacti | 16:30 |
| fungi | i should probably double-check our ansible inventory for mirrors i missed before we go further | 16:30 |
| clarkb | fungi: yes it should be | 16:31 |
| clarkb | la1 kna1 sto2 and lon1 | 16:31 |
| *** ralonsoh has quit IRC | 16:32 | |
| jeblair | bkero: no, but you might be able to use web developer tools to look at the web page to see what it does | 16:33 |
| *** annegentle has quit IRC | 16:33 | |
| *** dizquierdo has quit IRC | 16:33 | |
| bkero | jeblair: ah, fair point. I'll look at the outbound json packet | 16:34 |
| *** annegentle has joined #openstack-infra | 16:34 | |
| *** Guest86208 is now known as melwitt | 16:36 | |
| bkero | +"sshHostKeyVerificationStrategy":+{"stapler-class":+"hudson.plugins.sshslaves.verifiers.KnownHostsFileKeyVerificationStrategy" | 16:37 |
| bkero | +"$class":+"hudson.plugins.sshslaves.verifiers.KnownHostsFileKeyVerificationStrategy"} | 16:37 |
| bkero | I'm guessing that's it. I need to make it a dict with a stapler class key | 16:37 |
| fungi | anybody happen to know why `sudo ansible '*' --list-hosts` on puppetmaster.o.o doesn't list the infra-cloud mirrors? | 16:38 |
| fungi | do we need to flush the dynamic inventory cache after replacing those instances? | 16:38 |
| *** mdrabe has quit IRC | 16:39 | |
| *** raissa_ has quit IRC | 16:39 | |
| *** mdrabe_ has joined #openstack-infra | 16:39 | |
| *** Sukhdev has quit IRC | 16:41 | |
| *** claudiub has quit IRC | 16:43 | |
| fungi | deleting /var/cache/ansible-inventory/ansible-inventory.cache seems to have solved that | 16:44 |
| *** mdrabe_ has quit IRC | 16:46 | |
| *** mdrabe_ has joined #openstack-infra | 16:46 | |
| *** jamesden_ has joined #openstack-infra | 16:48 | |
| *** jamesdenton has quit IRC | 16:50 | |
| *** yamamoto has joined #openstack-infra | 16:51 | |
| fungi | looks like mirror.sto2.citycloud wasn't getting puppet updates due to an invalid cached hostkey on the puppetmaster | 16:53 |
| fungi | fixing | 16:53 |
| *** tosky_ has joined #openstack-infra | 16:53 | |
| *** jcoufal has quit IRC | 16:53 | |
| *** tosky is now known as Guest45177 | 16:53 | |
| *** tosky_ is now known as tosky | 16:53 | |
| fungi | okay, so luckily the only ones i missed were the citycloud mirrors and they all have a sufficiently large rootfs as well | 16:55 |
| *** Guest45177 has quit IRC | 16:55 | |
| *** pcaruana has quit IRC | 16:56 | |
| *** jtomasek_ has joined #openstack-infra | 16:58 | |
| *** yamamoto has quit IRC | 17:02 | |
| fungi | pabelanger: i realized i should probably add space on mirror.regionone.tripleo-test-cloud-rh1 too, but openstack volume create is leaving me with a cinder volume in an error state | 17:02 |
| fungi | pabelanger: any idea whether (1) this is supposed to work or (2) we should even care to make ample space for an apache proxy cache there? | 17:03 |
| dhellmann | fungi : do you remember how the git config is set up for the signing node? I'm seeing an error with the release jobs because gerrit doesn't like the email address being used by the proposal step http://logs.openstack.org/74/74b983ec535eb2939edf772a2237e16466cf7367/release-post/tag-releases/aa9abad/console.html | 17:06 |
| dhellmann | I don't remember if that's system data somewhere or if the script does it | 17:06 |
| pabelanger | fungi: ya, I don't think they have the space currently. However, I don't think we should care as I am helping tripleo deprecate that cloud in favor of thirdpary CI in rdocloud | 17:07 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add job's project as implicit role project https://review.openstack.org/482726 | 17:07 |
| fungi | normally the job would set the git configuration if memory serves... checking now | 17:07 |
| fungi | dhellmann: ^ | 17:07 |
| fungi | pabelanger: okay, i'll just ignore it for now | 17:07 |
| *** e0ne has quit IRC | 17:07 | |
| pabelanger | fungi: ++ | 17:07 |
| *** iyamahat has quit IRC | 17:09 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add networking-ovn job to ovsdbapp gate https://review.openstack.org/481650 | 17:11 |
| fungi | dhellmann: nevermind. it's preconfigured on that node by puppet: https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/signing_node.pp | 17:11 |
| dhellmann | ok | 17:12 |
| dhellmann | so, I guess I need to change the email address used there? | 17:12 |
| *** eharney has quit IRC | 17:12 | |
| fungi | dhellmann: yeah, well the job needs to _not_ change the e-mail address used to create the tag being pushed | 17:12 |
| dhellmann | oh, well, the error message says infra-root@openstack.org is registered and that's the email used | 17:12 |
| *** eharney has joined #openstack-infra | 17:13 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Pass result data back from ansible https://review.openstack.org/479442 | 17:13 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Support relative urls in success-url https://review.openstack.org/481342 | 17:13 |
| dhellmann | hrm | 17:13 |
| *** mat128 has joined #openstack-infra | 17:13 | |
| *** ramishra has quit IRC | 17:13 | |
| dhellmann | fungi : I see no signs in the log for that job that it did try to change the email address | 17:13 |
| *** mat128 has quit IRC | 17:14 | |
| dmsimard | Hey, asking on behalf of another user -- anyone know if it's possible to send a review over http (as opposed to over ssh) ? | 17:15 |
| fungi | dhellmann: aha! the script is trying to push this commit to gerrit: https://git.openstack.org/cgit/openstack/manila/commit/?id=ee0e5d71ee385d2a6ecc26bc42c2958d427b650c | 17:15 |
| dmsimard | Network has egress ssh on lockdown | 17:15 |
| fungi | dmsimard: yes, it should work. i think we even have directions for configuring git-review to use the https interface | 17:16 |
| dmsimard | fungi: okay I'll try and find them in the docs, thanks | 17:16 |
| dhellmann | fungi : oh, weird | 17:16 |
| fungi | dmsimard: i'm trying to find it | 17:16 |
| fungi | dhellmann: i wonder if the script is incorrectly trying to push the branch as a review? | 17:17 |
| clarkb | dmsimard: fungi there is old ml thread on how to do it to bypass great firewall of china | 17:17 |
| dhellmann | fungi : it does this: git push gerrit driverfixes/mitaka | 17:17 |
| dhellmann | after creating the branch | 17:17 |
| dhellmann | that has been working, but maybe not in this case? | 17:18 |
| fungi | clarkb: dmsimard: i think we incorporated a synopsis of it into one of our documents. thought it might be in the git-review docs bit not finding it | 17:18 |
| *** trown|lunch is now known as trown | 17:18 | |
| *** eranrom has quit IRC | 17:19 | |
| dhellmann | fungi : ee0e5d71ee385d2a6ecc26bc42c2958d427b650c does not exist in gerrit. I wonder how it got into the repo. | 17:19 |
| fungi | clarkb: dmsimard: aha! https://docs.openstack.org/infra/manual/developers.html#code-review | 17:19 |
| dhellmann | it's a merge commit, could that be why? | 17:19 |
| *** weshay is now known as weshay|lunch | 17:20 | |
| openstackgerrit | Merged openstack-infra/project-config master: Update ACLs for stable/pike in prep for release https://review.openstack.org/482951 | 17:20 |
| *** weshay|lunch is now known as weshay | 17:20 | |
| dmsimard | fungi: that html anchor is bad but I found what you meant to link to :) | 17:21 |
| dmsimard | https://docs.openstack.org/infra/manual/developers.html#accessing-gerrit-over-https | 17:21 |
| *** slaweq_ has joined #openstack-infra | 17:21 | |
| fungi | dmsimard: yep, sorry, i copied the wrong anchor | 17:21 |
| fungi | dhellmann: i think i might know what's going on here | 17:22 |
| *** claudiub has joined #openstack-infra | 17:22 | |
| dhellmann | fungi : good; I'm stumped. | 17:22 |
| *** Apoorva has joined #openstack-infra | 17:25 | |
| *** yamahata has quit IRC | 17:25 | |
| *** kjackal_ has joined #openstack-infra | 17:25 | |
| *** slaweq_ has quit IRC | 17:26 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add spacing lines and formatting to result dicts https://review.openstack.org/483042 | 17:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add play recap back in to the end of plays https://review.openstack.org/483043 | 17:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Log execution phase and include information https://review.openstack.org/483044 | 17:27 |
| dhellmann | fungi : I'll revert the patch to create the branches, so I can re-revert it and run the jobs again after we've fixed whatever it is. https://review.openstack.org/#/c/483040 | 17:30 |
| *** ykarel has quit IRC | 17:30 | |
| fungi | dhellmann: still trying to run this down... when i git remote update in openstack/manila i don't see a mitaka-eol tag. something's screwy here | 17:30 |
| dhellmann | oh, weird | 17:31 |
| dhellmann | I have that tag on 211f691d5cc82b32c5d83b5ca4646523167a4e86 fwiw | 17:31 |
| dhellmann | do we have a git repo sync issue? | 17:31 |
| AJaeger | fungi, http://git.openstack.org/cgit/openstack/manila/tag/?h=mitaka-eol | 17:31 |
| *** mat128 has joined #openstack-infra | 17:31 | |
| fungi | okay, once i did a git pull --tags i got it, but git remote update wasn't for some strange reason | 17:31 |
| dhellmann | odd | 17:32 |
| fungi | now i have the object in question | 17:32 |
| *** sambetts is now known as sambetts|afk | 17:32 | |
| fungi | dhellmann: okay, so i _think_ the issue at hand is that it's trying to create a new branch which includes commits not in any existing branch | 17:33 |
| fungi | thus gerrit believes the merge commits in there are new commits | 17:33 |
| dhellmann | hmm | 17:33 |
| dhellmann | it creates the branch using 211f691d5cc82b32c5d83b5ca4646523167a4e86 | 17:34 |
| dhellmann | and git branch -a --contains 211f691d5cc82b32c5d83b5ca4646523167a4e86 shows me origin/stable/mitaka | 17:34 |
| dhellmann | is there an origin/stable/mitaka still? | 17:34 |
| fungi | dhellmann: only because you haven't pruned old (now deleted) branches | 17:34 |
| dhellmann | aha | 17:35 |
| fungi | anyway, continuing my explanation, the release managers group doesn't have the required forgeCommitter permission to be able to push commits it wasn't the committer on | 17:35 |
| dhellmann | does that mean I couldn't even do this through the web UI? | 17:35 |
| fungi | you probably could because that's not a git push operation | 17:35 |
| dhellmann | we could change the eol process to ensure the driverfixes branch comes before the stable branch deletion (and we could validate that in the releases repo check job) | 17:36 |
| dhellmann | ok, let me try to do it by hand and see if that works better | 17:36 |
| fungi | as for forgeCommitter, normally that's only needed by users importing commits from outside gerrit (i.e. those in the project bootstrappers group like our new repo import automation) | 17:36 |
| dhellmann | yeah, if this is a sequencing error, I'm fine with addressing that through docs | 17:37 |
| dhellmann | and not changing the acls | 17:37 |
| fungi | gerrit has special handling when you're pushing commits so that if your history contains commits you're not a committer on but they're already part of a change in gerrit or in the history of an existing branch it won't reject them | 17:38 |
| dhellmann | the UI did let me create the branch | 17:38 |
| fungi | however it doesn't seem to do the same when the commits are only in the history of tags | 17:38 |
| fungi | and yeah, i expected the webui would work here | 17:38 |
| *** tosky has quit IRC | 17:38 | |
| fungi | so if we can switch to api calls for this step later (maybe after the gerrit upgrade? can't remember if you can do that yet in 2.11) then the sequencing issue goes away | 17:39 |
| *** adam_g has quit IRC | 17:39 | |
| dhellmann | I think I didn't switch to API calls because it had to be able to submit a patch anyway, so it seemed like we'd be dealing with extra credentials for no real reason. it sounds like that was not a fully informed decision. | 17:40 |
| dhellmann | I will add that to the future list for queens | 17:40 |
| *** adam_g has joined #openstack-infra | 17:41 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Add new line between zuul_stream messages https://review.openstack.org/483048 | 17:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add play recap back in to the end of plays https://review.openstack.org/483043 | 17:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add spacing lines and formatting to result dicts https://review.openstack.org/483042 | 17:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Log execution phase and include information https://review.openstack.org/483044 | 17:42 |
| *** iyamahat has joined #openstack-infra | 17:43 | |
| gouthamr | interesting scrollback. learnt a thing about these scripts. thanks for resolving this dhellmann fungi , i was able to cherry-pick a change to the new branch | 17:44 |
| dhellmann | gouthamr : you'll want to update the .gitreview file in those branches, too | 17:45 |
| dhellmann | I may be able to run the script to do that, let me see | 17:45 |
| gouthamr | oh, sure thing.. | 17:46 |
| *** annegentle has quit IRC | 17:47 | |
| dhellmann | gouthamr : you'll want to update https://review.openstack.org/#/c/483052/ with a valid url for the constraints file | 17:49 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix race in test_shadow https://review.openstack.org/483016 | 17:49 |
| dhellmann | gouthamr : and https://review.openstack.org/483054 may need to be updated, too | 17:50 |
| gouthamr | dhellmann: yep thank you.. i'll update these.. I think jenkins will attempt to run all the jobs, i proposed a project-config change to limit those: https://review.openstack.org/#/c/482785/ | 17:51 |
| dhellmann | gouthamr : sounds good. I'll leave it to you to take those over, but let me know if you run into trouble and I can help. | 17:51 |
| gouthamr | dhellmann: will do, thanks! :) | 17:52 |
| *** gyee has joined #openstack-infra | 17:53 | |
| *** jaosorior has joined #openstack-infra | 17:54 | |
| *** yamahata has joined #openstack-infra | 17:54 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove stable/ocata ACL for os-apply-config https://review.openstack.org/483024 | 17:55 |
| *** jascott1 has quit IRC | 17:57 | |
| *** electrofelix has quit IRC | 17:57 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: WIP: Adjust MTUs in multinode workflow on RDO Cloud https://review.openstack.org/481222 | 17:58 |
| *** rbrndt has quit IRC | 17:59 | |
| *** jamesden_ has quit IRC | 18:00 | |
| *** tosky has joined #openstack-infra | 18:01 | |
| *** jcoufal has joined #openstack-infra | 18:01 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Create configure-mirrors role https://review.openstack.org/482593 | 18:01 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove default value for mirror_host https://review.openstack.org/482915 | 18:01 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Replace hand-written slug with ansible_managed https://review.openstack.org/482916 | 18:01 |
| *** jamesdenton has joined #openstack-infra | 18:01 | |
| dhellmann | fungi : thanks for your help with that branching issue; I think we're all set now. | 18:02 |
| *** jascott1 has joined #openstack-infra | 18:04 | |
| *** e0ne has joined #openstack-infra | 18:04 | |
| *** yamamoto has joined #openstack-infra | 18:04 | |
| *** jascott1 has quit IRC | 18:04 | |
| *** ykarel has joined #openstack-infra | 18:06 | |
| fungi | dhellmann: excellent! | 18:07 |
| *** annegentle has joined #openstack-infra | 18:08 | |
| fungi | inc0: i'm testing retrieving /tarballs/kolla/images/centos-binary-registry-master.tar.gz from http://mirror.iad.rax.openstack.org:8080 right now to make sure it retrieves and caches the entire file (it's your largest at the moment, weighing in at ~4.2g) | 18:09 |
| *** eranrom has joined #openstack-infra | 18:10 | |
| fungi | inc0: if that works, then you're probably safe to try rechecking your change | 18:10 |
| fungi | i'll let you know shortly | 18:10 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/project-config master: ironic: Add DEVSTACK_GATE_IRONIC_INSPECTOR=1 for inspector jobs https://review.openstack.org/483058 | 18:12 |
| *** xarses has joined #openstack-infra | 18:13 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate master: Add ironic-inspector services to features.yaml https://review.openstack.org/483059 | 18:14 |
| jlvillal | ^^ I am hoping that those two patches will fix the ironic-inspector grenade gate. Reviews appreciated. | 18:16 |
| *** xarses_ has quit IRC | 18:16 | |
| jlvillal | Error we are seeing now: http://logs.openstack.org/85/480185/4/check/gate-grenade-dsvm-ironic-inspector-ubuntu-xenial/9790ec4/logs/grenade.sh.txt.gz#_2017-07-11_23_55_12_301 | 18:16 |
| *** ykarel has quit IRC | 18:19 | |
| jeblair | pabelanger, fungi: i've seen several instances of chocolate nodes being very slow | 18:20 |
| *** yamamoto has quit IRC | 18:20 | |
| jeblair | pabelanger, fungi: a bunch of them were on zuulv3 jobs, so we wrote it off to our ongoing work to use mirrors in v3 jobs, but i just saw a zuulv2 job fail | 18:20 |
| *** yamamoto has joined #openstack-infra | 18:21 | |
| jeblair | http://logs.openstack.org/36/482636/3/gate/gate-zuul-python35/f4102c2/console.html | 18:21 |
| *** slaweq_ has joined #openstack-infra | 18:22 | |
| *** Rockyg has quit IRC | 18:23 | |
| *** rlandy is now known as rlandy|brb | 18:24 | |
| *** yamamoto has quit IRC | 18:26 | |
| *** slaweq_ has quit IRC | 18:27 | |
| pabelanger | jeblair: we did replace mirrors this morning, I wonder if that is impacting jobs | 18:30 |
| pabelanger | or AFS cache is not warmed up yet? | 18:30 |
| jeblair | pabelanger: that would mean the v3 jobs are slow because we're using upstream ubuntu, and the v2 jobs are slow because our mirror caches are cold. that's a significant coincidence. | 18:31 |
| pabelanger | okay, infracloud-chocolate mirror is under a large amount of load | 18:32 |
| pabelanger | jeblair: agree, let me see if I can determine if we have a bad compute node | 18:32 |
| jeblair | pabelanger: want me to dig up some instance uuids? | 18:34 |
| pabelanger | http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=553&page=2 seems a little spotty | 18:34 |
| pabelanger | jeblair: sure, that would be helpful | 18:34 |
| *** kjackal_ has quit IRC | 18:35 | |
| jeblair | pabelanger: some of the v3 timeouts were in vanilla | 18:36 |
| *** jascott1 has joined #openstack-infra | 18:37 | |
| *** ldnunes has quit IRC | 18:37 | |
| pabelanger | chocolate mirror is under more load then vanilla, that is for sure | 18:38 |
| jeblair | pabelanger: hrm, i wonder if we're just oversubscribed? | 18:39 |
| jeblair | zuul is running at full capacity | 18:40 |
| *** efoley has quit IRC | 18:40 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Pass result data back from ansible https://review.openstack.org/479442 | 18:40 |
| pabelanger | jeblair: I think you might be correct, http://grafana.openstack.org/dashboard/db/nodepool-infra-cloud | 18:40 |
| pabelanger | we are also getting a large amount of launch failures too | 18:41 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: TEST: Use v2 net-iso templates in ci https://review.openstack.org/483064 | 18:41 |
| pabelanger | jeblair: looks like we are up to 4+ hours on all pipeline now. We look to be a full capacity on all our clouds | 18:42 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate master: WIP: Test patch to see if fixes ironic-inspector grenade https://review.openstack.org/483065 | 18:43 |
| *** Sukhdev has joined #openstack-infra | 18:45 | |
| *** makowals has quit IRC | 18:46 | |
| *** kjackal_ has joined #openstack-infra | 18:46 | |
| *** kjackal_ has quit IRC | 18:47 | |
| fungi | inc0: my download test through mirror.rax.iad resulted in the correct size and checksum, and seems to have increased the cache there by roughly the expected amount, so i thing you should be all set to move forward | 18:47 |
| inc0 | fungi: recheck is triggered https://review.openstack.org/#/c/482217/5 | 18:49 |
| jeblair | pabelanger: i looked into why zuul's job queue is >15k. it looks like it's mostly set_description jobs, which just aren't getting picked up because the launchers are so busy | 18:49 |
| jeblair | (that should be pretty harmless and we can ignore it) | 18:49 |
| *** Swami has joined #openstack-infra | 18:50 | |
| pabelanger | Ya, guess everybody is trying land patches for pike | 18:50 |
| pabelanger | its been a while since I've see zuul this busy | 18:50 |
| jeblair | that accounts for 12187 out of 16890 jobs, leaving 4703 actual jobs | 18:50 |
| *** ldnunes has joined #openstack-infra | 18:50 | |
| fungi | yikes, what happened around 16:00 utc to spike job activity so high? | 18:51 |
| fungi | looks like it just came up out of nowhere | 18:51 |
| pabelanger | think we had a gate reset, I can see a large amount of nodes deleted just before it | 18:51 |
| pabelanger | but ya, we launched about 650 nodes | 18:52 |
| fungi | yeah, there are several resets in the integrated queue | 18:52 |
| jeblair | a gate reset doesn't change the overall number of jobs in the system (or even nodes, since deleted==building) | 18:52 |
| fungi | that top change failed on a job timeout in infracloud-chocolate | 18:53 |
| jeblair | we may want to dial back max-servers for those two clouds | 18:53 |
| pabelanger | ya, I think that is a good idea | 18:53 |
| fungi | agreed, wondering if we have a cascade/chain reaction sort of situation going on | 18:53 |
| pabelanger | well, we have been losing nodes over time in infracloud, I don't think we ever adjusted our max-server when we turned some of the servers off | 18:54 |
| *** mdrabe_ has quit IRC | 18:54 | |
| pabelanger | so, we likely need to re-calculate our capacity again | 18:54 |
| fungi | well, the nova change causing the more recent reset in the integrated gate looks like it timed out on an ansible task after the main job completed | 18:55 |
| fungi | and happened in ovh-bhs1 | 18:55 |
| jeblair | about 1000 jobs were queued between 15:30 and 15:45 | 18:56 |
| *** mdrabe has joined #openstack-infra | 18:56 | |
| *** annegentle has quit IRC | 18:57 | |
| *** rlandy|brb is now known as rlandy | 18:57 | |
| *** rbrndt has joined #openstack-infra | 18:57 | |
| pabelanger | are we okay with reducing max-servers by 10 nodes for each infracloud region? | 18:58 |
| *** baoli has quit IRC | 18:59 | |
| fungi | sounds like a fine place to start | 19:00 |
| *** baoli has joined #openstack-infra | 19:00 | |
| jeblair | pabelanger: how many online compute hosts do we have? | 19:00 |
| *** baoli has quit IRC | 19:00 | |
| pabelanger | jeblair: let me check | 19:01 |
| fungi | looking at http://logs.openstack.org/97/482497/1/gate/gate-nova-pep8-ubuntu-xenial/c8c6067/_zuul_ansible/ansible_log.txt all i can guess is the launcher was so congested it took ~35 minutes to start the post playbook after the job playbook was done | 19:01 |
| fungi | load average on the launchers is pretty high, though that's not unusual | 19:01 |
| *** baoli has joined #openstack-infra | 19:01 | |
| *** annegentle has joined #openstack-infra | 19:02 | |
| pabelanger | jeblair: infracloud-vanilla is reporting 40 compute nodes ATM | 19:03 |
| *** coolsvap has quit IRC | 19:03 | |
| fungi | actually, some of the launcher load averages look like they're starting to come down (05 and 06 in particular) | 19:04 |
| pabelanger | jeblair: of that, 38 are functional | 19:05 |
| pabelanger | let me see if I can find out why that is | 19:05 |
| fungi | similarly, looks like the zuul job queue and nodes in use are starting to come down as well | 19:06 |
| *** annegentle has quit IRC | 19:06 | |
| jeblair | oh, so it looks like we're neither ram or disk constrained there. only cpu constrained? | 19:06 |
| jeblair | dropping by 10 servers (approx 10% of our current) sounds good | 19:07 |
| fungi | so guessing we had a major job spike around 15:30 and 15:45 as jeblair indicates, and by the time we started looking into it whatever caused it was mostly complete and so we were looking at the aftermath/backlog? | 19:07 |
| pabelanger | Ya, 5 compute nodes in infracloud-vanilla appear to be unreachable via networking. I'll make a note to try to access them via ilo and see if they can be recovered | 19:08 |
| pabelanger | otherwise, we likely need feet on the ground to help troubleshoot | 19:08 |
| jeblair | fungi: yes. the set_description jobs make it hard to tell what the actual queue level is at this point, but looks like they should dissipate soon. | 19:09 |
| AJaeger | mriedem: could you review https://review.openstack.org/#/c/482962 , please? | 19:10 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Reduce infracloud choclate / vanilla max-servers by 10 https://review.openstack.org/483071 | 19:11 |
| pabelanger | jeblair: fungi: ^ | 19:11 |
| mriedem | ok | 19:11 |
| jeblair | fungi, pabelanger: spot checking jobs launched in that timeframe, i don't see any smoking guns | 19:11 |
| fungi | jeblair: maybe we had a lag in zuul acting on new events which was long enough that the event queue pileup resulted in the utilization spike we see on the graph? | 19:12 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add some indexes to allow efficient dashboard sql https://review.openstack.org/481614 | 19:14 |
| *** e0ne has quit IRC | 19:14 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove function_cache variables from executor client https://review.openstack.org/482890 | 19:14 |
| openstackgerrit | Patrick East proposed openstack-infra/devstack-gate master: Set cinder image cache limits https://review.openstack.org/483073 | 19:15 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Cleanup components doc https://review.openstack.org/482636 | 19:15 |
| AJaeger | team, would appreciate a review on https://review.openstack.org/475300 (project-config changes for omni) - the change should work but it's sooo unusual that I'd like some of you review it | 19:15 |
| jeblair | fungi: maybe -- zuul was still active in the preceding 30 minutes, but not as active as normal. | 19:16 |
| *** yamamoto has joined #openstack-infra | 19:16 | |
| *** nicolasbock has quit IRC | 19:16 | |
| pabelanger | TIL: infracloud.io | 19:17 |
| *** harlowja has quit IRC | 19:17 | |
| *** david-lyle has quit IRC | 19:17 | |
| jeblair | AJaeger: is that an official project? | 19:17 |
| *** david-lyle has joined #openstack-infra | 19:17 | |
| AJaeger | jeblair: no | 19:18 |
| jeblair | pabelanger, fungi: we're now under capacity | 19:18 |
| fungi | yeah, it sloped off very quickly | 19:19 |
| AJaeger | jeblair: omni = "Multi and hybrid cloud management with OpenStack API." - they allow using Amazon with OpenStack API | 19:19 |
| jeblair | fungi: and yeah, i think we may have overloaded the launchers a bit, based on the playbook delay you found | 19:19 |
| *** adisky__ has quit IRC | 19:20 | |
| jeblair | fungi: the math says we might run up to 187 jobs on a launcher now, which may be a bit too much. we may want to throw another 2-4 launchers at it. | 19:20 |
| pabelanger | Ya, I think we could stand to use atleast another 2 for sure | 19:21 |
| fungi | jeblair: bumping up to 10 dynamic launchers seems prudent | 19:21 |
| fungi | or thereabouts | 19:22 |
| jeblair | AJaeger: okay. i think most of the stuff we'd be concerned about (zuul-cloner) has been handled in earlier patchsets. if they ever wanted to become an official project, this is something to be aware of. | 19:22 |
| openstackgerrit | Eric Kao proposed openstack-infra/project-config master: enable py35 gate for congress-dashboard https://review.openstack.org/483076 | 19:22 |
| inc0 | fungi: cache seems to be working | 19:23 |
| inc0 | what's invalidation period for this? | 19:23 |
| fungi | jeblair: what's our target peak? not still 100 jobs per launcher i guess? | 19:23 |
| *** slaweq_ has joined #openstack-infra | 19:23 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: WIP Testing changes required to move 3nodes job to quickstart https://review.openstack.org/483077 | 19:23 |
| inc0 | technically we could invalidate when file changes on source | 19:23 |
| jeblair | fungi: probably more. drat. we really need a statsd metric for currently running jobs on each launcher. :( | 19:23 |
| AJaeger | thanks, jeblair | 19:23 |
| clarkb | inc0: the cache itself is 24 hours. | 19:25 |
| fungi | inc0: whatever the default in apache 2.4.7 default is; vhost config here: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/mirror.vhost.erb | 19:25 |
| clarkb | fungi: 24 houra iic | 19:25 |
| fungi | cool. i woulda had to look that up in the apache docs ;) | 19:25 |
| inc0 | yeah that's totally fine | 19:25 |
| pabelanger | jeblair: I think I had a patch up for that a while back, but think it got -2. I can see if I can find it again | 19:25 |
| inc0 | since we'll be moving to 24h updates of files anyway | 19:26 |
| fungi | jeblair: zuul v4 will have nodepool elastically scaling nodepool launchers, right? ;) | 19:26 |
| *** slaweq_ has quit IRC | 19:26 | |
| pabelanger | fungi: yes please :D | 19:26 |
| jeblair | fungi: quite possibly. :) something something containers something. | 19:26 |
| *** slaweq_ has joined #openstack-infra | 19:26 | |
| fungi | er, nodepool elastically scaling zuul launchers i mean | 19:26 |
| fungi | what, you want another build when there are already 1000 going? you're queued until an 11th launcher node is built and brought into service | 19:28 |
| openstackgerrit | Merged openstack-infra/project-config master: Reduce infracloud choclate / vanilla max-servers by 10 https://review.openstack.org/483071 | 19:28 |
| openstackgerrit | Eric Kao proposed openstack-infra/project-config master: enable py35 gate for congress-dashboard https://review.openstack.org/483076 | 19:29 |
| *** e0ne has joined #openstack-infra | 19:29 | |
| jeblair | fungi, pabelanger: so if we add 2 more, our ceiling will be 150/launcher. that seems like a good nominal target for now. | 19:29 |
| *** annegentle has joined #openstack-infra | 19:30 | |
| fungi | a fine place to start and then see if similar lags occur between playbooks or not | 19:30 |
| fungi | at least we have a fingerprint of sorts for misbehavior under excess load | 19:30 |
| pabelanger | agree, 150 seems fine to me | 19:31 |
| fungi | so we should be able to tell if it happens again | 19:31 |
| jeblair | the ptg conflicts with The Linux Foundation's Open Source Summit North America which is being keynoted by Joseph Gordon-Levitt. | 19:31 |
| fungi | yep, unfortunate. they announced their dates after we'd nailed down our contracts aiui | 19:32 |
| pabelanger | fungi: jeblair: do either of you want to add the new launchers? I don't might turning them up if needed | 19:32 |
| jeblair | pabelanger: i was about to try to find some food, so if you're up for it, that'd be great | 19:32 |
| fungi | similar to oscon announcing their date on top of our boston summit after we'd already booked the venue | 19:32 |
| fungi | pabelanger: go for it! | 19:32 |
| pabelanger | k, let me top up my coffee first | 19:34 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Correct run job parameter documentation https://review.openstack.org/482977 | 19:34 |
| openstackgerrit | Eric Kao proposed openstack-infra/project-config master: enable py35 gate for congress-dashboard https://review.openstack.org/483076 | 19:34 |
| zara_the_lemur__ | (heh, just saw that open source summit email) | 19:34 |
| *** camunoz has quit IRC | 19:35 | |
| AJaeger | team, change 482067 passes http proxy variables in tox.ini for project-config - is that something we should take? | 19:37 |
| jeblair | yeah, i mean, we're going to need to book a real a-lister for the ptg keynote if we want to compete. :) | 19:37 |
| zara_the_lemur__ | I actually had no idea who he was but could tell the email thought I did. | 19:38 |
| jeblair | AJaeger: i think it's ok. hopefully will help some folks run tox before submitted project-config changes. probably not many, but hey. | 19:38 |
| jeblair | zara_the_lemur__: third rock from the sun? | 19:39 |
| fungi | apparently he played snowden in the 2016 movie of the same name | 19:40 |
| fungi | which is presumably the draw there | 19:40 |
| zara_the_lemur__ | I think I have actually seen none of the films on his imdb | 19:41 |
| fungi | we got a keynote from the real edward snowden, they probably had to settle for the fake one | 19:41 |
| jeblair | yeah. not a bad likeness. | 19:41 |
| zara_the_lemur__ | (I guess I see about 1 film a year or something...) | 19:41 |
| zara_the_lemur__ | haha | 19:41 |
| zara_the_lemur__ | well the email had a big photo of him | 19:41 |
| zara_the_lemur__ | so now I know what he looks like | 19:42 |
| jeblair | zara_the_lemur__: emails have photos? | 19:42 |
| fungi | zara_the_lemur__: that's about how often i go out to watch a talkie anyway | 19:42 |
| jeblair | i go to the local movie palace whenever they show a 70mm print. which, actually, is twice this month. so kind of awesome. | 19:42 |
| *** slaweq_ has quit IRC | 19:43 | |
| zara_the_lemur__ | I can take a screenshot of this email photo for those who have html such things firmly turned off | 19:43 |
| zara_the_lemur__ | it's quite something | 19:43 |
| *** wolverineav has quit IRC | 19:43 | |
| *** slaweq_ has joined #openstack-infra | 19:43 | |
| jeblair | okay, not always. i passed on batman vs superman. | 19:43 |
| *** wolverineav has joined #openstack-infra | 19:44 | |
| zara_the_lemur__ | from what I've heard, that was the right decision | 19:44 |
| zara_the_lemur__ | (and, well, also from the title) | 19:44 |
| fungi | i figured it was just a godzilla vs mothra remake at heart | 19:44 |
| zara_the_lemur__ | http://imgur.com/a/3cshz | 19:45 |
| jeblair | fungi: that i would see | 19:45 |
| jeblair | zara_the_lemur__, fungi: no glasses, so he must not be doing his snowden impression | 19:46 |
| fungi | i think i have it on dvd, but yes a picture show like that really needs a bigger screen than my living room can hold | 19:46 |
| zara_the_lemur__ | :D | 19:46 |
| fungi | sorry, mothra vs godzilla (the '64 film). g vs m was the '92 one | 19:48 |
| fungi | it's unfortunately all too easy to confuse the two | 19:48 |
| *** slaweq_ has quit IRC | 19:48 | |
| *** slaweq_ has joined #openstack-infra | 19:49 | |
| zara_the_lemur__ | I found a poster: http://vignette4.wikia.nocookie.net/godzilla/images/d/da/Mothra_vs._Godzilla_Poster_A.png/revision/latest?cb=20140514004544 | 19:49 |
| fungi | nice! | 19:50 |
| *** wolverineav has quit IRC | 19:50 | |
| *** mdrabe_ has joined #openstack-infra | 19:53 | |
| *** mdrabe_ has left #openstack-infra | 19:53 | |
| *** yamamoto has quit IRC | 19:54 | |
| *** florianf has quit IRC | 19:55 | |
| *** baoli has quit IRC | 19:56 | |
| pabelanger | okay, starting on zuul-launchers | 19:56 |
| fungi | i'm rolling forward with ptgbot changes now that things seem to have calmed down for a few minutes | 19:57 |
| fungi | hopefully with a couple hours to focus i can get this squared away | 19:57 |
| openstackgerrit | Merged openstack-infra/puppet-ptgbot master: Add instructions related to the PTGbot IRC bot https://review.openstack.org/480938 | 20:00 |
| *** baoli has joined #openstack-infra | 20:00 | |
| *** baoli has quit IRC | 20:00 | |
| *** harlowja has joined #openstack-infra | 20:02 | |
| dmsimard | pabelanger: hey, having problems bootstrapping a new nodepool DIB image. It looks like the image is taking slightly more than 30 minutes to build and it doesn't get the time to finish before nodepool starts another image build :/ | 20:03 |
| dmsimard | pabelanger: can this be configured somewhere ? | 20:04 |
| mtreinish | pabelanger: that patch lgtm | 20:05 |
| pabelanger | dmsimard: do you have a build log? a new DIB only will get created if previous failed. How often are you setup to rebuild images? | 20:06 |
| openstackgerrit | Merged openstack-infra/puppet-ptgbot master: Serve Web site content https://review.openstack.org/481230 | 20:07 |
| *** baoli has joined #openstack-infra | 20:07 | |
| *** makowals has joined #openstack-infra | 20:08 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add zl08.o.o and zl09.o.o https://review.openstack.org/483088 | 20:12 |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Add environment file for multinode tests on RDO Cloud https://review.openstack.org/481222 | 20:18 |
| dhellmann | fungi : checking in to see if you have any feedback on that script to move the old content around on the AFS servers? | 20:19 |
| pabelanger | dmsimard: which version of nodepool are you using again? | 20:22 |
| openstackgerrit | Merged openstack-infra/system-config master: Install ptgbot on eavesdrop.o.o https://review.openstack.org/481201 | 20:22 |
| dmsimard | pabelanger: I'm trying something first, let you know if I figure out the issue | 20:22 |
| fungi | dhellmann: that's https://review.openstack.org/481710 right? did you ever manage to get any feedback from the docs team on it? | 20:23 |
| fungi | on the end result i mean | 20:23 |
| *** jerryz has joined #openstack-infra | 20:24 | |
| *** jkilpatr has quit IRC | 20:25 | |
| pabelanger | dmsimard: the old way was setting image-update value for crontab to trigger builds, it is possible you have it setup incorrectly | 20:25 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate master: Add ironic-inspector services to features.yaml https://review.openstack.org/483059 | 20:25 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate master: WIP: Test patch to see if fixes ironic-inspector grenade https://review.openstack.org/483065 | 20:25 |
| jerryz | clarkb: Hi, could you please add me in fuel-plugin-fortinet-core group? thanks! | 20:27 |
| fungi | dhellmann: ignore my earlier question about docs team reviewers. for some reason i'd cached that change at a time prior to when they'd left comments/votes on it | 20:27 |
| jerryz | clarkb: my gerrit id is 6671 xyzjerry@gmail.com | 20:28 |
| dhellmann | fungi : I can adjust the path thing in the script, though I didn't necessarily intend for us to actually merge that change -- it was just an easy way to share the script | 20:28 |
| fungi | dhellmann: oh, in that case it's good enough | 20:29 |
| dhellmann | fungi : ok, cool | 20:29 |
| dhellmann | fungi: what would be a good time to sit down and run that? | 20:29 |
| sbezverk | fungi: ping | 20:29 |
| fungi | i can take a stab at it tomorrow... i'm in the middle of ptgbot things right at the moment | 20:29 |
| dhellmann | and then to run the script it produces, of course | 20:29 |
| clarkb | jerryz: another infra-root will need to do it. I am currently in transit | 20:29 |
| *** e0ne has quit IRC | 20:29 | |
| dhellmann | oh, sure, I didn't mean right now, just looking to set a rough time | 20:30 |
| jerryz | clarkb: ok. thanks. | 20:30 |
| dhellmann | fungi : perhaps after lunch tomorrow? unless earlier is better for you | 20:30 |
| fungi | after lunch is fine | 20:31 |
| sbezverk | fungi: we have kolla-kubernetes gates broken and it seems related to constraints gate-kolla-kubernetes-python27-ubuntu-xenial/upper-constraints.txt | 20:31 |
| dhellmann | ok. AJaeger wanted me to remove some of the older releases, so I'll do that and post a new version | 20:31 |
| dhellmann | fungi : thanks | 20:31 |
| fungi | i'll want to remember to incorporate jeblair's suggestion of disabling the vos release for it and then vetting the read-write volume prior to accepting the results | 20:31 |
| pabelanger | fungi: jeblair: both zl08.o.o and zl09.o.o are online now. Do you mind reviewing https://review.openstack.org/#/c/483088/ to add them into system-config (firewall, cacti) | 20:32 |
| jeblair | pabelanger: we didn't have a zl08 before? | 20:32 |
| pabelanger | no, just zl01-zl07 | 20:33 |
| fungi | jeblair: we had zlstatic01 and zl01-zl07 i thought | 20:33 |
| jeblair | wow my math was all wrong then. | 20:33 |
| jeblair | we were at 214/zl | 20:33 |
| *** e0ne has joined #openstack-infra | 20:33 | |
| jeblair | going down to 166 with this | 20:33 |
| jeblair | so neat. better than we thought. :) | 20:33 |
| sbezverk | all jobs are failing here http://logs.openstack.org/82/483082/1/check/gate-kolla-kubernetes-python27-ubuntu-xenial/32616ff/console.html#_2017-07-12_20_00_25_252101, appreciate if somebody could check this.. | 20:33 |
| *** makowals has quit IRC | 20:34 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: WIP Testing changes required to move 3nodes job to quickstart https://review.openstack.org/483077 | 20:34 |
| fungi | sbezverk: we're not the requirements team. however i agree that looks like it _could_ be the result of a recent cliff release | 20:35 |
| *** wolverineav has joined #openstack-infra | 20:35 | |
| jeblair | sbezverk: http://git.openstack.org/cgit/openstack/cliff/ says there was one 2 days ago | 20:35 |
| pabelanger | Ya, I see a new version on pypi from 2 days ago | 20:35 |
| sbezverk | fungi: any suggestion for my next step? | 20:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add play recap back in to the end of plays https://review.openstack.org/483043 | 20:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add spacing lines and formatting to result dicts https://review.openstack.org/483042 | 20:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Log execution phase and include information https://review.openstack.org/483044 | 20:36 |
| sbezverk | right it started a couple of days | 20:36 |
| sbezverk | ago | 20:36 |
| *** wolverineav has quit IRC | 20:37 | |
| *** wolverineav has joined #openstack-infra | 20:37 | |
| fungi | sbezverk: maybe dtroyer has a suggestion since he was the one to propose that release | 20:38 |
| fungi | https://git.openstack.org/cgit/openstack/cliff/tag/?h=2.8.0 | 20:38 |
| *** kgiusti has quit IRC | 20:38 | |
| dhellmann | sbezverk : I'll try to help you. Let's move to #openstack-sdks | 20:39 |
| *** jkilpatr has joined #openstack-infra | 20:41 | |
| sbezverk | dhellmann : thanks a lot | 20:43 |
| inc0 | fungi: change merged | 20:44 |
| inc0 | let me know if you get more issues with tarballs | 20:45 |
| *** Goneri has quit IRC | 20:50 | |
| *** e0ne has quit IRC | 20:53 | |
| *** trown is now known as trown|outtypewww | 20:54 | |
| fungi | inc0: thanks, i'll keep a close eye on it | 20:54 |
| *** yamamoto has joined #openstack-infra | 20:54 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: WIP Testing changes required to move 3nodes job to quickstart https://review.openstack.org/483077 | 20:56 |
| *** yamamoto has quit IRC | 21:01 | |
| *** vhosakot has quit IRC | 21:01 | |
| *** jtomasek_ has quit IRC | 21:02 | |
| *** vhosakot has joined #openstack-infra | 21:02 | |
| *** jcoufal has quit IRC | 21:03 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Log execution phase and include information https://review.openstack.org/483044 | 21:03 |
| *** iyamahat_ has joined #openstack-infra | 21:03 | |
| *** iyamahat has quit IRC | 21:04 | |
| *** Sukhdev has quit IRC | 21:04 | |
| *** iyamahat_ has quit IRC | 21:07 | |
| *** iyamahat has joined #openstack-infra | 21:07 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-openstackci master: Add support for zuul v3 job-output files https://review.openstack.org/483101 | 21:08 |
| *** dave-mccowan has quit IRC | 21:09 | |
| *** bramwelt has quit IRC | 21:13 | |
| *** lennyb has quit IRC | 21:15 | |
| *** mat128 has quit IRC | 21:17 | |
| *** dprince has quit IRC | 21:20 | |
| *** thorst has quit IRC | 21:23 | |
| *** yland has joined #openstack-infra | 21:25 | |
| *** lennyb has joined #openstack-infra | 21:27 | |
| *** iyamahat_ has joined #openstack-infra | 21:31 | |
| *** iyamahat has quit IRC | 21:32 | |
| *** mat128 has joined #openstack-infra | 21:33 | |
| *** wolverineav has quit IRC | 21:36 | |
| *** wolverineav has joined #openstack-infra | 21:37 | |
| openstackgerrit | Merged openstack-infra/shade master: Improve doc formatting a bit https://review.openstack.org/482819 | 21:38 |
| *** zzzeek_ has joined #openstack-infra | 21:42 | |
| *** yland has quit IRC | 21:44 | |
| *** Goneri has joined #openstack-infra | 21:46 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-ptgbot master: Pass vhost_name into the vhost resource https://review.openstack.org/483107 | 21:49 |
| fungi | any infra-puppet-core up for a quick fix? ^ | 21:49 |
| *** armax has quit IRC | 21:51 | |
| *** felipemonteiro_ has quit IRC | 21:52 | |
| *** armax has joined #openstack-infra | 21:52 | |
| *** mattmceuen has joined #openstack-infra | 21:53 | |
| mordred | fungi: looking | 21:54 |
| *** eharney has quit IRC | 21:54 | |
| *** ihrachys has quit IRC | 21:54 | |
| mordred | +3 | 21:54 |
| fungi | thanks cmurphy and mordred! | 21:54 |
| mordred | spekaing of small fixes ... https://review.openstack.org/483101 is easy if anyone wants it :) | 21:54 |
| fungi | lgtm | 21:55 |
| *** armax has quit IRC | 21:56 | |
| pabelanger | fungi: mordred: jeblair: clarkb: sorry for ping again, but 483088 is last step for new zuul-launchers. If you want to review today great, otherwise, likely hold off until morning to start new zuul-launchers | 21:58 |
| *** Apoorva has quit IRC | 21:58 | |
| jeblair | pabelanger: oh sorry, thanks for the reping | 21:59 |
| fungi | yep, same. juggling too many torches | 21:59 |
| fungi | approved now | 21:59 |
| jeblair | mordred: why don't we gzip job-output.txt? | 21:59 |
| * fungi needs to go juggle the dinner torch for a bit now | 21:59 | |
| jeblair | mordred: maybe we can do that in the post-playbook? | 22:00 |
| mordred | jeblair: I thought about that too - we run in to that lovely chicken-and-egg of not being able to log our actions of gzipping the log file :) | 22:00 |
| mordred | jeblair: also - last week one of the files DID get gzipped - whichconfuses me | 22:01 |
| jeblair | mordred: cron gzipped it probably | 22:02 |
| mordred | jeblair: ah. that's why | 22:02 |
| mordred | http://logs.openstack.org/14/481014/5/check/9ce8405/ | 22:02 |
| mordred | jeblair: that's the one with a gzipped log | 22:02 |
| jeblair | mordred: still, there's no escaping the fact that at some point, we have to stop logging. i'm okay with the gzip being on the other side of that line. | 22:03 |
| mordred | kk | 22:03 |
| jeblair | mordred: (i think an advantage of zuulv3 is that we can actually put the gzip steps in the base post playbooks and maybe even get rid of the cron) | 22:03 |
| *** jaosorior has quit IRC | 22:03 | |
| *** Apoorva has joined #openstack-infra | 22:04 | |
| jeblair | (we can also enforce a max log size) | 22:04 |
| openstackgerrit | Merged openstack-infra/puppet-ptgbot master: Pass vhost_name into the vhost resource https://review.openstack.org/483107 | 22:04 |
| pabelanger | nice, 481014 is multi-node! | 22:04 |
| *** Apoorva has quit IRC | 22:04 | |
| *** ldnunes has quit IRC | 22:05 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add zl08.o.o and zl09.o.o https://review.openstack.org/483088 | 22:09 |
| *** Apoorva has joined #openstack-infra | 22:10 | |
| *** wolverineav has quit IRC | 22:10 | |
| *** esberglu has quit IRC | 22:11 | |
| *** wolverineav has joined #openstack-infra | 22:13 | |
| *** mat128 has joined #openstack-infra | 22:13 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove gate-grenade-dsvm-multinode job https://review.openstack.org/482962 | 22:13 |
| *** jamesdenton has quit IRC | 22:14 | |
| *** makowals has joined #openstack-infra | 22:14 | |
| *** jamesdenton has joined #openstack-infra | 22:14 | |
| *** zzzeek_ has quit IRC | 22:15 | |
| *** dprince has joined #openstack-infra | 22:16 | |
| *** zzzeek_ has joined #openstack-infra | 22:17 | |
| *** zzzeek has quit IRC | 22:18 | |
| *** jamesdenton has quit IRC | 22:19 | |
| *** wolverineav has quit IRC | 22:21 | |
| *** dprince has quit IRC | 22:22 | |
| *** slaweq_ has quit IRC | 22:23 | |
| *** wolverineav has joined #openstack-infra | 22:24 | |
| *** makowals has quit IRC | 22:28 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Correct run job parameter documentation https://review.openstack.org/482977 | 22:28 |
| *** Goneri has quit IRC | 22:32 | |
| *** d34dh0r53 has quit IRC | 22:32 | |
| *** puiterwijk has quit IRC | 22:32 | |
| *** puiterwi1 has joined #openstack-infra | 22:32 | |
| *** tosky has quit IRC | 22:32 | |
| *** watersoul has quit IRC | 22:32 | |
| *** d34dh0r53 has joined #openstack-infra | 22:32 | |
| *** rm_work has quit IRC | 22:32 | |
| *** puiterwi1 is now known as puiterwijk | 22:32 | |
| *** tosky has joined #openstack-infra | 22:32 | |
| *** bobh has quit IRC | 22:32 | |
| *** rm_work has joined #openstack-infra | 22:32 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/os-loganalyze master: Add processing for zuul phases https://review.openstack.org/483117 | 22:34 |
| *** rhagarty has quit IRC | 22:35 | |
| *** mpjetta has quit IRC | 22:35 | |
| *** raginbajin has quit IRC | 22:35 | |
| *** yuval has quit IRC | 22:35 | |
| *** slagle has quit IRC | 22:35 | |
| mordred | jeblair, pabelanger: ^^ there's a stab at the first part of consuming those phases from the zuul_stream output | 22:35 |
| *** slagle has joined #openstack-infra | 22:35 | |
| *** sambetts|afk has quit IRC | 22:36 | |
| *** mpjetta has joined #openstack-infra | 22:37 | |
| *** szaher has quit IRC | 22:37 | |
| *** szaher has joined #openstack-infra | 22:37 | |
| *** yuval has joined #openstack-infra | 22:37 | |
| *** rhagarty has joined #openstack-infra | 22:37 | |
| *** sambetts_ has joined #openstack-infra | 22:37 | |
| *** watersoul has joined #openstack-infra | 22:38 | |
| *** claudiub has quit IRC | 22:38 | |
| *** mat128 has quit IRC | 22:38 | |
| *** mat128 has joined #openstack-infra | 22:39 | |
| *** raginbajin has joined #openstack-infra | 22:42 | |
| *** mat128 has quit IRC | 22:43 | |
| *** makowals has joined #openstack-infra | 22:44 | |
| *** thorst has joined #openstack-infra | 22:45 | |
| *** thorst has quit IRC | 22:45 | |
| *** rbrndt has quit IRC | 22:45 | |
| openstackgerrit | Brian Haley proposed openstack-infra/project-config master: Split Neutron check queue grafana dashboard https://review.openstack.org/483119 | 22:46 |
| *** rlandy is now known as rlandy|bbl | 22:46 | |
| *** ihrachys has joined #openstack-infra | 22:46 | |
| jeblair | mordred: couple of things inline on that. | 22:47 |
| jeblair | mordred: and flake8 caught a real thing too | 22:48 |
| openstackgerrit | Jeremy Stanley proposed openstack/ptgbot master: List requests as a requirement https://review.openstack.org/483120 | 22:48 |
| *** makowals has quit IRC | 22:50 | |
| *** yamamoto has joined #openstack-infra | 22:52 | |
| *** wolverineav has quit IRC | 22:55 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-ptgbot master: Correct the db_filename path https://review.openstack.org/483124 | 22:55 |
| *** tosky has quit IRC | 22:57 | |
| *** armax has joined #openstack-infra | 22:58 | |
| mordred | jeblair: I like it when it does that1 | 22:59 |
| *** mattmceuen has quit IRC | 23:00 | |
| *** annegentle has quit IRC | 23:01 | |
| *** wolverineav has joined #openstack-infra | 23:02 | |
| *** gordc has quit IRC | 23:02 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-ptgbot master: Make HTML and Javascript files world readable https://review.openstack.org/483125 | 23:10 |
| *** d0ugal has joined #openstack-infra | 23:11 | |
| *** baoli_ has joined #openstack-infra | 23:16 | |
| *** baoli has quit IRC | 23:18 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/os-loganalyze master: Add processing for zuul phases https://review.openstack.org/483117 | 23:20 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate master: Add ironic-inspector services to features.yaml https://review.openstack.org/483059 | 23:22 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate master: WIP: Test patch to see if fixes ironic-inspector grenade https://review.openstack.org/483065 | 23:22 |
| *** wolverineav has quit IRC | 23:23 | |
| *** slaweq_ has joined #openstack-infra | 23:23 | |
| *** wolverineav has joined #openstack-infra | 23:23 | |
| *** mat128 has joined #openstack-infra | 23:26 | |
| *** baoli_ has quit IRC | 23:26 | |
| pabelanger | wow, something odd with nodepool | 23:27 |
| pabelanger | 300 ready nodes | 23:27 |
| pabelanger | odd | 23:28 |
| pabelanger | Oh, I think I know | 23:28 |
| pabelanger | because zl08 and zl09 haven't started | 23:28 |
| pabelanger | oh, maybe not | 23:28 |
| pabelanger | Hmm | 23:28 |
| *** slaweq_ has quit IRC | 23:28 | |
| pabelanger | ugh, I forgot to add them to nodepool.yaml | 23:30 |
| pabelanger | k, playing them in shutdown mode | 23:30 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add zl08 and zl09 to nodepool.yaml https://review.openstack.org/483128 | 23:31 |
| pabelanger | infra-root: does anybody mind +3 on ^, I forgot to update nodepool with the new zuul-launchers | 23:32 |
| mordred | pabelanger: done | 23:34 |
| pabelanger | ty | 23:35 |
| *** dhajare has quit IRC | 23:37 | |
| *** stakeda has joined #openstack-infra | 23:38 | |
| *** mattmceuen has joined #openstack-infra | 23:39 | |
| *** hongbin has quit IRC | 23:40 | |
| *** marst has quit IRC | 23:41 | |
| *** wolverineav has quit IRC | 23:44 | |
| *** wolverineav has joined #openstack-infra | 23:46 | |
| pabelanger | okay, both zl08 and zl09 stopped. zuul rescheduled the jobs on those launchers, I'm cleaning up nodepool now | 23:48 |
| jeblair | pabelanger: what's to clean up? | 23:51 |
| openstackgerrit | Merged openstack-infra/project-config master: Add zl08 and zl09 to nodepool.yaml https://review.openstack.org/483128 | 23:51 |
| pabelanger | jeblair: between both new launchers, we had approx 300 nodes in ready state. Which was resulting in no other launchers getting nodes assigned to them | 23:52 |
| pabelanger | and wasn't able to get new nodes assigned to a good launcher to land ^ | 23:52 |
| jeblair | pabelanger: how did you stop zuul-launcher? | 23:53 |
| pabelanger | I used zuul-launcher stop | 23:53 |
| jeblair | oh :( that's like the one time you don't want to do that. | 23:53 |
| jeblair | pabelanger: either 'zuul-launcher graceful' or 'zuul-launcher release' would have returned all the nodes to nodepool | 23:54 |
| *** Swami has quit IRC | 23:54 | |
| *** s-shiono has joined #openstack-infra | 23:54 | |
| pabelanger | Ah | 23:54 |
| jeblair | perhaps we could have added release to stop as well | 23:54 |
| pabelanger | I was going to do graceful, but wasn't sure how long it would have taken | 23:54 |
| jeblair | pabelanger: no time at all if it wasn't running any jobs | 23:55 |
| pabelanger | jeblair: true | 23:55 |
| pabelanger | is there a downside not to release for stop? | 23:55 |
| jeblair | pabelanger: oh... i misunderstood | 23:55 |
| jeblair | pabelanger: they were running jobs but not getting marked used | 23:55 |
| pabelanger | right, some nodes were used, but didn't update nodepool properly | 23:56 |
| jeblair | nevermind then :) | 23:56 |
| *** bobh has joined #openstack-infra | 23:56 | |
| pabelanger | k | 23:56 |
| *** mat128 has quit IRC | 23:57 | |
| *** Julien-zte has joined #openstack-infra | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
