| *** yamamoto has joined #openstack-infra | 00:03 | |
| *** heyongli has quit IRC | 00:07 | |
| *** heyongli has joined #openstack-infra | 00:08 | |
| *** jesslampe has quit IRC | 00:09 | |
| *** yamamoto has quit IRC | 00:09 | |
| *** bobh has quit IRC | 00:11 | |
| *** dingyichen has joined #openstack-infra | 00:12 | |
| *** spsurya has joined #openstack-infra | 00:16 | |
| *** heyongli has quit IRC | 00:18 | |
| *** heyongli has joined #openstack-infra | 00:18 | |
| *** jesslampe has joined #openstack-infra | 00:19 | |
| *** jesslampe has quit IRC | 00:20 | |
| *** heyongli has quit IRC | 00:28 | |
| *** heyongli has joined #openstack-infra | 00:28 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add MariaDB to cache https://review.openstack.org/577325 | 00:33 |
|---|---|---|
| mnaser | ianw: i want to make a patch to switch up NODEPOOL_MARIADB_MIRROR in project-config | 00:35 |
| mnaser | can i just do it alone or do i have to depends-on the removal of the other one | 00:35 |
| mnaser | so you know, we make sure it doesn't sit there forgotten | 00:35 |
| openstackgerrit | Merged openstack-infra/system-config master: adding #openstack-mentoring to meetbot channel https://review.openstack.org/577243 | 00:36 |
| ianw | mnaser: well i kind of like depends-on as a forcing measure, even if not strictly required :) | 00:36 |
| mnaser | ianw: haha, i'll do it for the sakes of making sure we don't miss it :) | 00:37 |
| ianw | i wouldn't mind one more set of eyes on it, to make sure didn't forget anything | 00:37 |
| mnaser | also my change shouldn't merge until puppet runs once at least | 00:38 |
| *** heyongli has quit IRC | 00:38 | |
| *** heyongli has joined #openstack-infra | 00:39 | |
| *** hongbin has joined #openstack-infra | 00:40 | |
| *** aspiers has quit IRC | 00:41 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/openstack-zuul-jobs master: Add NODEPOOL_MARIADB_PROXY and drop NODEPOOL_MARIADB_MIRROR https://review.openstack.org/577677 | 00:42 |
| mnaser | ianw: hope i don't end up holding my change hostage cause no one wants to review a 117 line change patch :P | 00:42 |
| ianw | right now is probably globally the worst time to get attention anything :) but the nice thing is I usually have the gate pretty much to myself | 00:44 |
| *** heyongli has quit IRC | 00:48 | |
| *** heyongli has joined #openstack-infra | 00:49 | |
| mnaser | ianw: oh yes, over the weekend i can pretty much get so much done and iterate so much quicker lol | 00:49 |
| openstackgerrit | Anita Kuno proposed openstack-infra/system-config master: Remove Infra-Cloud docs https://review.openstack.org/574881 | 00:49 |
| *** rh-jelabarre has joined #openstack-infra | 00:52 | |
| *** heyongli has quit IRC | 00:59 | |
| *** heyongli has joined #openstack-infra | 00:59 | |
| *** yamamoto has joined #openstack-infra | 01:05 | |
| *** heyongli has quit IRC | 01:09 | |
| *** heyongli has joined #openstack-infra | 01:09 | |
| *** yamamoto has quit IRC | 01:10 | |
| *** sthussey has quit IRC | 01:15 | |
| *** openstack has joined #openstack-infra | 01:18 | |
| *** ChanServ sets mode: +o openstack | 01:18 | |
| *** heyongli has quit IRC | 01:19 | |
| *** heyongli has joined #openstack-infra | 01:20 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Update to openstackdocstheme https://review.openstack.org/577678 | 01:21 |
| *** heyongli has quit IRC | 01:29 | |
| *** zhangfei has joined #openstack-infra | 01:30 | |
| *** heyongli has joined #openstack-infra | 01:30 | |
| ianw | oh maybe i finally caught a host with one of these traceroute issues | 01:37 |
| * ianw goes host fishing | 01:37 | |
| *** bobh has joined #openstack-infra | 01:38 | |
| ianw | oh no, bummer. my hold was only on the single-node job, and it failed on one of the hosts of the multi-node jobs. the host that got away | 01:39 |
| *** heyongli has quit IRC | 01:40 | |
| *** heyongli has joined #openstack-infra | 01:40 | |
| openstackgerrit | Nguyen Van Trung proposed openstack/diskimage-builder master: Add iscsi-boot element for CentOS images https://review.openstack.org/542708 | 01:42 |
| *** zhongjun__ has joined #openstack-infra | 01:44 | |
| *** zhangfei has quit IRC | 01:47 | |
| *** heyongli has quit IRC | 01:50 | |
| *** heyongli has joined #openstack-infra | 01:50 | |
| *** gongysh has joined #openstack-infra | 01:50 | |
| openstackgerrit | Merged openstack-infra/system-config master: Don't use snakeoil files with real certs https://review.openstack.org/577571 | 01:53 |
| *** ramishra has joined #openstack-infra | 01:54 | |
| *** heyongli has quit IRC | 02:00 | |
| *** heyongli has joined #openstack-infra | 02:01 | |
| *** yamamoto has joined #openstack-infra | 02:07 | |
| *** heyongli has quit IRC | 02:10 | |
| *** heyongli has joined #openstack-infra | 02:11 | |
| *** yamamoto has quit IRC | 02:12 | |
| *** heyongli has quit IRC | 02:21 | |
| *** heyongli has joined #openstack-infra | 02:21 | |
| *** heyongli has quit IRC | 02:31 | |
| *** heyongli has joined #openstack-infra | 02:31 | |
| *** annp has joined #openstack-infra | 02:31 | |
| *** bobh has quit IRC | 02:35 | |
| *** rh-jelabarre has quit IRC | 02:37 | |
| *** heyongli has quit IRC | 02:41 | |
| *** heyongli has joined #openstack-infra | 02:42 | |
| *** psachin` has joined #openstack-infra | 02:42 | |
| *** __holden__ has quit IRC | 02:46 | |
| *** heyongli has quit IRC | 02:51 | |
| *** heyongli has joined #openstack-infra | 02:52 | |
| *** psachin` has quit IRC | 02:53 | |
| *** heyongli has quit IRC | 03:02 | |
| *** heyongli has joined #openstack-infra | 03:02 | |
| portdirect | hey - is storyboard ok atm? | 03:04 |
| *** germs has quit IRC | 03:04 | |
| portdirect | I'm having difficulty commenting on stories | 03:04 |
| *** germs has joined #openstack-infra | 03:04 | |
| *** germs has quit IRC | 03:04 | |
| *** germs has joined #openstack-infra | 03:04 | |
| portdirect | `500: POST /api/v1/stories/2002658/comments: (InternalError) (1205, u'Lock wait timeout exceeded; try restarting transaction') u'UPDATE stories SET updated_at=%(updated_at)s WHERE stories.id = %(stories_id)s' {'updated_at': datetime.datetime(2018, 6, 25, 3, 3, 25, 259728), 'stories_id': 2002658}` | 03:04 |
| *** germs has quit IRC | 03:06 | |
| *** aeng_ has joined #openstack-infra | 03:07 | |
| *** germs has joined #openstack-infra | 03:08 | |
| *** germs has quit IRC | 03:08 | |
| *** germs has joined #openstack-infra | 03:08 | |
| *** yamamoto has joined #openstack-infra | 03:09 | |
| ianw | hmm i was poking around a little ago and not seeing issues, but didn't comment | 03:09 |
| *** slaweq has joined #openstack-infra | 03:11 | |
| ianw | portdirect: ok, yeah, there are quite a few similar errors in the log | 03:11 |
| *** heyongli has quit IRC | 03:12 | |
| *** heyongli has joined #openstack-infra | 03:12 | |
| *** yamamoto has quit IRC | 03:14 | |
| ianw | fungi: i think the errors actually started with -> http://paste.openstack.org/show/724194/ | 03:15 |
| *** slaweq has quit IRC | 03:16 | |
| ianw | some suggestions that looks like a threading error, possibly | 03:17 |
| *** heyongli has quit IRC | 03:22 | |
| *** heyongli has joined #openstack-infra | 03:23 | |
| ianw | yeah, errors before that about closed connections. definitely seems the rabbitmq reconnection logic in storyboard can get itself stuck | 03:26 |
| *** heyongli has quit IRC | 03:32 | |
| *** heyongli has joined #openstack-infra | 03:33 | |
| ianw | #status log storyboard.o.o works & apache restarted after persistent errors appearing to occur after rabbitmq disconnection. See log around Sun Jun 24 08:33:43.058712 for original error (http://paste.openstack.org/show/724194/) | 03:35 |
| openstackstatus | ianw: finished logging | 03:35 |
| ianw | workers even | 03:35 |
| mnaser | ianw: I’m really sorry for asking this, but something has come up and I’m being asked to temporarily drop max-servers to 0 because of an issue | 03:36 |
| mnaser | ianw: I don’t have a laptop to push the change from. Would you mind doing that and I can review? Sorry for asking .. | 03:37 |
| mnaser | (IRCCloud on my phone) | 03:37 |
| ianw | mnaser: ok, is it dire emergency, i can hand-edit? | 03:37 |
| mnaser | ianw: hand edit would be even better | 03:37 |
| mnaser | It should be resolved by tomorrow am for revert | 03:38 |
| ianw | ok give me a tick and we can work this out | 03:38 |
| *** tojuvone has quit IRC | 03:40 | |
| ianw | #status log nl03.o.o in emergency and vexxhost max-severs turned to 0 temporarily | 03:41 |
| openstackstatus | ianw: finished logging | 03:41 |
| *** heyongli has quit IRC | 03:43 | |
| *** heyongli has joined #openstack-infra | 03:43 | |
| mnaser | ianw: thank you. | 03:45 |
| *** udesale has joined #openstack-infra | 03:46 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Disable vexxhost https://review.openstack.org/577718 | 03:46 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Revert "Disable vexxhost" https://review.openstack.org/577719 | 03:46 |
| *** lpetrut has joined #openstack-infra | 03:50 | |
| *** heyongli has quit IRC | 03:53 | |
| *** heyongli has joined #openstack-infra | 03:53 | |
| *** jesslampe has joined #openstack-infra | 03:55 | |
| *** tojuvone has joined #openstack-infra | 04:00 | |
| *** heyongli has quit IRC | 04:03 | |
| *** heyongli has joined #openstack-infra | 04:04 | |
| *** yamamoto has joined #openstack-infra | 04:10 | |
| *** heyongli has quit IRC | 04:13 | |
| *** heyongli has joined #openstack-infra | 04:14 | |
| *** d0ugal_ has joined #openstack-infra | 04:15 | |
| *** cshastri has joined #openstack-infra | 04:15 | |
| *** germs has quit IRC | 04:15 | |
| *** d0ugal has quit IRC | 04:15 | |
| *** yamamoto has quit IRC | 04:15 | |
| mnaser | ianw: reviewed above thanks | 04:16 |
| *** ykarel has joined #openstack-infra | 04:21 | |
| openstackgerrit | Merged openstack-infra/project-config master: Disable vexxhost https://review.openstack.org/577718 | 04:21 |
| *** hongbin has quit IRC | 04:22 | |
| *** heyongli has quit IRC | 04:24 | |
| *** heyongli has joined #openstack-infra | 04:24 | |
| *** sree_ has joined #openstack-infra | 04:27 | |
| *** sree_ is now known as Guest36681 | 04:27 | |
| *** lpetrut has quit IRC | 04:28 | |
| *** heyongli has quit IRC | 04:34 | |
| *** heyongli has joined #openstack-infra | 04:34 | |
| *** psachin` has joined #openstack-infra | 04:34 | |
| *** germs has joined #openstack-infra | 04:40 | |
| *** noama has joined #openstack-infra | 04:41 | |
| *** heyongli has quit IRC | 04:44 | |
| *** heyongli has joined #openstack-infra | 04:45 | |
| *** dhajare has joined #openstack-infra | 04:45 | |
| *** heyongli has quit IRC | 04:54 | |
| *** heyongli has joined #openstack-infra | 04:55 | |
| *** s-shiono has joined #openstack-infra | 05:02 | |
| *** heyongli has quit IRC | 05:05 | |
| *** heyongli has joined #openstack-infra | 05:05 | |
| *** iranzo has joined #openstack-infra | 05:10 | |
| *** aspiers has joined #openstack-infra | 05:11 | |
| *** yamamoto has joined #openstack-infra | 05:12 | |
| *** heyongli has quit IRC | 05:15 | |
| *** janki has joined #openstack-infra | 05:15 | |
| *** heyongli has joined #openstack-infra | 05:15 | |
| *** gongysh has quit IRC | 05:15 | |
| *** e0ne has joined #openstack-infra | 05:17 | |
| *** yamamoto has quit IRC | 05:18 | |
| *** nicolasbock has joined #openstack-infra | 05:21 | |
| *** heyongli has quit IRC | 05:25 | |
| *** heyongli has joined #openstack-infra | 05:26 | |
| *** jesslampe has quit IRC | 05:28 | |
| *** armaan has joined #openstack-infra | 05:29 | |
| *** yamamoto has joined #openstack-infra | 05:30 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: log-inventory: add missing zuul_info_dir prep https://review.openstack.org/577674 | 05:33 |
| *** heyongli has quit IRC | 05:35 | |
| *** hamzy_ has quit IRC | 05:36 | |
| *** heyongli has joined #openstack-infra | 05:36 | |
| *** Bhujay has joined #openstack-infra | 05:37 | |
| *** hamzy has joined #openstack-infra | 05:38 | |
| *** hamzy has quit IRC | 05:40 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Improve logging of GithubTriggerEvents https://review.openstack.org/575014 | 05:40 |
| *** gongysh has joined #openstack-infra | 05:41 | |
| *** hamzy has joined #openstack-infra | 05:42 | |
| *** Bhujay has quit IRC | 05:44 | |
| *** Bhujay has joined #openstack-infra | 05:44 | |
| *** heyongli has quit IRC | 05:46 | |
| *** heyongli has joined #openstack-infra | 05:46 | |
| *** e0ne has quit IRC | 05:46 | |
| *** e0ne has joined #openstack-infra | 05:48 | |
| *** germs_ has joined #openstack-infra | 05:51 | |
| *** Bhujay has quit IRC | 05:53 | |
| *** yolanda has joined #openstack-infra | 05:53 | |
| *** germs has quit IRC | 05:54 | |
| *** heyongli has quit IRC | 05:56 | |
| *** heyongli has joined #openstack-infra | 05:56 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a Kubernetes driver https://review.openstack.org/535557 | 05:59 |
| *** aeng_ has quit IRC | 06:00 | |
| *** pcichy has quit IRC | 06:02 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a Runc driver https://review.openstack.org/535556 | 06:03 |
| *** heyongli has quit IRC | 06:06 | |
| *** heyongli has joined #openstack-infra | 06:07 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenShift resource provider https://review.openstack.org/570667 | 06:09 |
| *** florianf has joined #openstack-infra | 06:10 | |
| *** Guest7592 has quit IRC | 06:14 | |
| *** jcoufal has joined #openstack-infra | 06:16 | |
| *** jcoufal has quit IRC | 06:16 | |
| *** heyongli has quit IRC | 06:16 | |
| *** heyongli has joined #openstack-infra | 06:17 | |
| *** jcoufal has joined #openstack-infra | 06:17 | |
| *** andreas_s has joined #openstack-infra | 06:21 | |
| *** dhajare has quit IRC | 06:23 | |
| *** heyongli has quit IRC | 06:27 | |
| *** heyongli has joined #openstack-infra | 06:27 | |
| *** kazsh_ has quit IRC | 06:28 | |
| *** kazsh has joined #openstack-infra | 06:29 | |
| *** jbadiapa has joined #openstack-infra | 06:35 | |
| *** e0ne has quit IRC | 06:37 | |
| *** heyongli has quit IRC | 06:37 | |
| *** heyongli has joined #openstack-infra | 06:37 | |
| *** Bhujay has joined #openstack-infra | 06:41 | |
| *** jtomasek has joined #openstack-infra | 06:43 | |
| *** heyongli has quit IRC | 06:47 | |
| *** heyongli has joined #openstack-infra | 06:48 | |
| *** lifeless has quit IRC | 06:49 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: add support for resource connection type https://review.openstack.org/570668 | 06:56 |
| *** heyongli has quit IRC | 06:57 | |
| *** heyongli has joined #openstack-infra | 06:58 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-base-jobs master: Add openshift-base job https://review.openstack.org/570669 | 07:01 |
| *** armaan has quit IRC | 07:05 | |
| *** armaan has joined #openstack-infra | 07:06 | |
| *** pcaruana has joined #openstack-infra | 07:07 | |
| *** heyongli has quit IRC | 07:08 | |
| *** heyongli has joined #openstack-infra | 07:08 | |
| *** dhajare has joined #openstack-infra | 07:11 | |
| *** armaan has quit IRC | 07:12 | |
| *** zoli|gone is now known as zoli | 07:13 | |
| *** heyongli has quit IRC | 07:18 | |
| *** heyongli has joined #openstack-infra | 07:18 | |
| *** tesseract has joined #openstack-infra | 07:20 | |
| *** dpawlik has joined #openstack-infra | 07:21 | |
| *** lpetrut has joined #openstack-infra | 07:25 | |
| *** heyongli has quit IRC | 07:28 | |
| *** kjackal has joined #openstack-infra | 07:29 | |
| *** heyongli has joined #openstack-infra | 07:29 | |
| *** lpetrut has quit IRC | 07:29 | |
| *** hashar has joined #openstack-infra | 07:34 | |
| *** Hal has joined #openstack-infra | 07:35 | |
| *** Hal is now known as Guest72489 | 07:35 | |
| *** heyongli has quit IRC | 07:38 | |
| *** heyongli has joined #openstack-infra | 07:39 | |
| *** jpena|off is now known as jpena | 07:41 | |
| *** amoralej|off is now known as amoralej | 07:42 | |
| *** heyongli has quit IRC | 07:49 | |
| *** heyongli has joined #openstack-infra | 07:49 | |
| *** armaan has joined #openstack-infra | 07:51 | |
| *** jpich has joined #openstack-infra | 07:55 | |
| *** toabctl has quit IRC | 07:58 | |
| *** pgadiya has joined #openstack-infra | 07:59 | |
| *** heyongli has quit IRC | 07:59 | |
| *** pgadiya has quit IRC | 07:59 | |
| *** heyongli has joined #openstack-infra | 07:59 | |
| *** kjackal_ has joined #openstack-infra | 07:59 | |
| *** andreas_s has quit IRC | 08:01 | |
| *** kjackal has quit IRC | 08:04 | |
| *** pcichy has joined #openstack-infra | 08:06 | |
| *** roman_g has joined #openstack-infra | 08:09 | |
| *** heyongli has quit IRC | 08:09 | |
| *** heyongli has joined #openstack-infra | 08:10 | |
| *** andreas_s has joined #openstack-infra | 08:10 | |
| *** snapiri- has joined #openstack-infra | 08:11 | |
| *** ccamacho has joined #openstack-infra | 08:11 | |
| *** snapiri has quit IRC | 08:13 | |
| *** kjackal has joined #openstack-infra | 08:13 | |
| *** jistr|off is now known as jistr | 08:14 | |
| *** d0ugal_ has quit IRC | 08:16 | |
| *** dbecker has joined #openstack-infra | 08:16 | |
| *** kjackal_ has quit IRC | 08:17 | |
| *** d0ugal has joined #openstack-infra | 08:17 | |
| *** d0ugal has quit IRC | 08:17 | |
| *** d0ugal has joined #openstack-infra | 08:17 | |
| *** andreas_s has quit IRC | 08:19 | |
| *** ykarel_ has joined #openstack-infra | 08:19 | |
| *** heyongli has quit IRC | 08:19 | |
| *** heyongli has joined #openstack-infra | 08:20 | |
| *** andreas_s has joined #openstack-infra | 08:20 | |
| *** ykarel_ is now known as ykarel|lunch | 08:21 | |
| *** armaan has quit IRC | 08:21 | |
| *** ykarel has quit IRC | 08:22 | |
| *** shardy has joined #openstack-infra | 08:24 | |
| *** alexchadin has joined #openstack-infra | 08:25 | |
| *** lpetrut has joined #openstack-infra | 08:25 | |
| *** arxcruz|off is now known as arxcruz | 08:26 | |
| *** kjackal_ has joined #openstack-infra | 08:27 | |
| *** andreas_s has quit IRC | 08:28 | |
| *** andreas_s has joined #openstack-infra | 08:28 | |
| *** andreas_s has quit IRC | 08:29 | |
| *** toabctl has joined #openstack-infra | 08:29 | |
| *** andreas_s has joined #openstack-infra | 08:29 | |
| *** heyongli has quit IRC | 08:30 | |
| *** derekh has joined #openstack-infra | 08:30 | |
| *** heyongli has joined #openstack-infra | 08:30 | |
| *** kjackal has quit IRC | 08:31 | |
| *** e0ne has joined #openstack-infra | 08:34 | |
| *** sshnaidm|off is now known as sshnaidm | 08:39 | |
| *** heyongli has quit IRC | 08:40 | |
| *** heyongli has joined #openstack-infra | 08:40 | |
| *** pjrusak has joined #openstack-infra | 08:43 | |
| *** jaosorior has joined #openstack-infra | 08:43 | |
| *** namnh has joined #openstack-infra | 08:43 | |
| *** kjackal has joined #openstack-infra | 08:44 | |
| *** alexchad_ has joined #openstack-infra | 08:45 | |
| *** dingyichen has quit IRC | 08:47 | |
| *** alexchadin has quit IRC | 08:48 | |
| *** kjackal_ has quit IRC | 08:49 | |
| *** pblaho has joined #openstack-infra | 08:50 | |
| *** heyongli has quit IRC | 08:50 | |
| *** heyongli has joined #openstack-infra | 08:51 | |
| *** electrofelix has joined #openstack-infra | 08:58 | |
| *** snapiri- has quit IRC | 08:59 | |
| *** snapiri has joined #openstack-infra | 08:59 | |
| *** gfidente has joined #openstack-infra | 08:59 | |
| *** gfidente has joined #openstack-infra | 08:59 | |
| pjrusak | I have a question regarding projects in Zuulv3. I would like to add to checkout repositories project from github and I'm not sure how to do this. Should I put link to repo as project.name? | 08:59 |
| *** heyongli has quit IRC | 09:00 | |
| *** heyongli has joined #openstack-infra | 09:01 | |
| *** alexchad_ is now known as alexchadin | 09:01 | |
| *** dtantsur|afk is now known as dtantsur | 09:03 | |
| *** r-mibu has joined #openstack-infra | 09:04 | |
| *** jcoufal has quit IRC | 09:10 | |
| *** heyongli has quit IRC | 09:11 | |
| *** heyongli has joined #openstack-infra | 09:11 | |
| *** kjackal_ has joined #openstack-infra | 09:11 | |
| *** jcoufal has joined #openstack-infra | 09:13 | |
| *** jcoufal has quit IRC | 09:13 | |
| *** jcoufal has joined #openstack-infra | 09:14 | |
| *** kjackal has quit IRC | 09:15 | |
| *** kjackal has joined #openstack-infra | 09:17 | |
| *** heyongli has quit IRC | 09:21 | |
| *** heyongli has joined #openstack-infra | 09:21 | |
| *** kjackal_ has quit IRC | 09:21 | |
| *** andreas_s has quit IRC | 09:24 | |
| *** andreas_s has joined #openstack-infra | 09:25 | |
| *** sree_ has joined #openstack-infra | 09:25 | |
| *** sree_ is now known as Guest93496 | 09:25 | |
| *** zoli is now known as zoli|lunch | 09:28 | |
| AJaeger | pjrusak: check http://git.openstack.org/cgit/openstack/openstack-ansible/tree/zuul.d/jobs.yaml#n52 for an example | 09:28 |
| *** Guest36681 has quit IRC | 09:29 | |
| *** ykarel|lunch is now known as ykarel | 09:29 | |
| *** Guest93496 has quit IRC | 09:30 | |
| *** heyongli has quit IRC | 09:31 | |
| *** heyongli has joined #openstack-infra | 09:32 | |
| *** andreas_s has quit IRC | 09:34 | |
| *** andreas_s has joined #openstack-infra | 09:34 | |
| *** andreas_s has quit IRC | 09:35 | |
| *** kjackal_ has joined #openstack-infra | 09:36 | |
| *** andreas_s has joined #openstack-infra | 09:36 | |
| *** namnh has quit IRC | 09:36 | |
| *** kjackal__ has joined #openstack-infra | 09:37 | |
| *** kjackal has quit IRC | 09:38 | |
| *** kjackal_ has quit IRC | 09:41 | |
| *** andreas_s has quit IRC | 09:41 | |
| *** namnh has joined #openstack-infra | 09:41 | |
| *** heyongli has quit IRC | 09:41 | |
| *** andreas_s has joined #openstack-infra | 09:42 | |
| *** heyongli has joined #openstack-infra | 09:42 | |
| *** andreas_s has quit IRC | 09:43 | |
| *** andreas_s has joined #openstack-infra | 09:43 | |
| pjrusak | thanks AJaeger | 09:44 |
| pjrusak | So if I add repo under job definition like in linked openstack-ansible example do I have to create project in projects.yaml with matching name? | 09:47 |
| *** andreas_s has quit IRC | 09:48 | |
| *** andreas_s has joined #openstack-infra | 09:48 | |
| AJaeger | pjrusak: if you test an openstack project and add one repo from github to check it out - that you can do in-tree. | 09:51 |
| AJaeger | pjrusak: if you want to test a github repo on its own, that is a different story... | 09:51 |
| *** heyongli has quit IRC | 09:52 | |
| *** heyongli has joined #openstack-infra | 09:52 | |
| *** sree_ has joined #openstack-infra | 09:52 | |
| *** sree_ is now known as Guest9127 | 09:53 | |
| *** rosmaita has joined #openstack-infra | 09:55 | |
| *** peereb has joined #openstack-infra | 09:59 | |
| *** Guest9127 has quit IRC | 10:01 | |
| *** peereb has quit IRC | 10:01 | |
| *** andreas_s has quit IRC | 10:01 | |
| *** peereb has joined #openstack-infra | 10:01 | |
| *** andreas_s has joined #openstack-infra | 10:01 | |
| *** andreas_s has quit IRC | 10:01 | |
| *** andreas_s has joined #openstack-infra | 10:02 | |
| *** heyongli has quit IRC | 10:02 | |
| *** peereb has quit IRC | 10:02 | |
| *** heyongli has joined #openstack-infra | 10:02 | |
| *** peereb has joined #openstack-infra | 10:03 | |
| *** peereb has quit IRC | 10:04 | |
| *** peereb has joined #openstack-infra | 10:04 | |
| *** peereb has quit IRC | 10:05 | |
| *** lpetrut has quit IRC | 10:06 | |
| *** alexchad_ has joined #openstack-infra | 10:07 | |
| *** quiquell|rover has joined #openstack-infra | 10:08 | |
| quiquell|rover | Hello do you know of any issuw with register-1.docker.io ? | 10:09 |
| quiquell|rover | we are hitting 503 issues | 10:09 |
| quiquell|rover | https://bugs.launchpad.net/tripleo/+bug/1778472 | 10:09 |
| openstack | Launchpad bug 1778472 in tripleo "docker pull failed: Get https://registry-1.docker.io/v2/tripleomaster/centos-binary-rsyslog-base/manifests/current-tripleo: received unexpected HTTP status: 503 Service Unavailable" [Critical,Triaged] - Assigned to Quique Llorente (quiquell) | 10:09 |
| *** lpetrut has joined #openstack-infra | 10:09 | |
| *** alexchadin has quit IRC | 10:10 | |
| *** cshastri_ has joined #openstack-infra | 10:11 | |
| *** alexchad_ has quit IRC | 10:11 | |
| *** cshastri has quit IRC | 10:11 | |
| pjrusak | AJaeger: It's first case | 10:11 |
| *** heyongli has quit IRC | 10:12 | |
| *** heyongli has joined #openstack-infra | 10:13 | |
| *** kjackal__ has quit IRC | 10:14 | |
| *** namnh_ has joined #openstack-infra | 10:21 | |
| *** namnh_ has quit IRC | 10:21 | |
| *** heyongli has quit IRC | 10:22 | |
| *** heyongli has joined #openstack-infra | 10:23 | |
| *** gfidente has quit IRC | 10:23 | |
| *** namnh has quit IRC | 10:24 | |
| *** caphrim007_ has quit IRC | 10:26 | |
| *** alexchadin has joined #openstack-infra | 10:26 | |
| *** caphrim007 has joined #openstack-infra | 10:26 | |
| *** kjackal has joined #openstack-infra | 10:27 | |
| ianw | clarkb / fungi : now we've settled on those projects, i've committed them to git hiera, thanks | 10:29 |
| ianw | re previous discussion on uncommitted packethost changes within | 10:29 |
| *** heyongli has quit IRC | 10:33 | |
| *** heyongli has joined #openstack-infra | 10:33 | |
| AJaeger | pjrusak: then go ahead ;) | 10:37 |
| *** alexchadin has quit IRC | 10:38 | |
| quiquell|rover | jpena: How long takes to test the rdoinfo changes ? | 10:38 |
| *** alexchadin has joined #openstack-infra | 10:38 | |
| quiquell|rover | https://review.rdoproject.org/r/#/c/14467/ no results yet | 10:38 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add uuid for vfat https://review.openstack.org/576765 | 10:38 |
| *** alexchadin has quit IRC | 10:38 | |
| jpena | quiquell|rover: it can take a while, I see it is still running 1 test in https://softwarefactory-project.io/zuul/t/rdoproject.org/status.html | 10:39 |
| *** alexchadin has joined #openstack-infra | 10:39 | |
| *** alexchadin has quit IRC | 10:39 | |
| quiquell|rover | jpena: We are already migrated ? | 10:39 |
| jpena | quiquell|rover: for rdoinfo jobs, yes | 10:40 |
| *** alexchadin has joined #openstack-infra | 10:40 | |
| quiquell|rover | jpena: \o/ | 10:40 |
| *** alexchadin has quit IRC | 10:40 | |
| *** alexchadin has joined #openstack-infra | 10:40 | |
| *** alexchadin has quit IRC | 10:41 | |
| *** heyongli has quit IRC | 10:43 | |
| *** heyongli has joined #openstack-infra | 10:43 | |
| *** s-shiono has quit IRC | 10:44 | |
| *** shardy has quit IRC | 10:44 | |
| *** edmondsw has joined #openstack-infra | 10:45 | |
| *** edmondsw_ has joined #openstack-infra | 10:48 | |
| *** edmondsw has quit IRC | 10:49 | |
| *** dhill_ has quit IRC | 10:53 | |
| *** heyongli has quit IRC | 10:53 | |
| *** heyongli has joined #openstack-infra | 10:54 | |
| quiquell|rover | jpena: This is related to my change ? https://logs.rdoproject.org/67/14467/3/check/legacy-rdoinfo-tripleo-pike-testing-centos-7-multinode-1ctlr-featureset006/0209252/job-output.txt.gz#_2018-06-25_09_45_03_379935 | 10:56 |
| quiquell|rover | chown: cannot access '/opt/git': No such file or directory | 10:56 |
| *** gfidente has joined #openstack-infra | 10:57 | |
| jpena | quiquell|rover: this is probably related to the migration to zuul v3. amoralej does ^^ ring any bell? | 10:58 |
| amoralej | yes | 10:58 |
| amoralej | seems related tov3 | 10:58 |
| amoralej | let me check how that works in oooq jobs for tripleo-ci | 10:59 |
| amoralej | quiquell|rover, aren't yo useen a similar issue in other jobs for non rdoinfo reviews? | 10:59 |
| *** jpena is now known as jpena|lunch | 11:00 | |
| *** heyongli has quit IRC | 11:03 | |
| *** zoli|lunch is now known as zoli | 11:04 | |
| *** heyongli has joined #openstack-infra | 11:04 | |
| *** slaweq has joined #openstack-infra | 11:04 | |
| *** quiquell|rover has quit IRC | 11:06 | |
| *** jcoufal has quit IRC | 11:08 | |
| *** gongysh has quit IRC | 11:08 | |
| *** alexchadin has joined #openstack-infra | 11:11 | |
| *** heyongli has quit IRC | 11:14 | |
| *** pgadiya has joined #openstack-infra | 11:14 | |
| *** pgadiya has quit IRC | 11:14 | |
| *** heyongli has joined #openstack-infra | 11:14 | |
| *** mriedem has joined #openstack-infra | 11:16 | |
| *** alexchadin has quit IRC | 11:17 | |
| *** cshastri_ has quit IRC | 11:18 | |
| *** alexchadin has joined #openstack-infra | 11:20 | |
| *** cshastri_ has joined #openstack-infra | 11:23 | |
| *** slaweq has quit IRC | 11:23 | |
| *** heyongli has quit IRC | 11:24 | |
| *** heyongli has joined #openstack-infra | 11:24 | |
| *** cshastri_ has quit IRC | 11:25 | |
| *** cshastri has joined #openstack-infra | 11:25 | |
| *** udesale has quit IRC | 11:27 | |
| *** tosky has joined #openstack-infra | 11:28 | |
| *** ldnunes has joined #openstack-infra | 11:29 | |
| *** amoralej is now known as amoralej|lunch | 11:32 | |
| *** heyongli has quit IRC | 11:34 | |
| *** heyongli has joined #openstack-infra | 11:35 | |
| *** cshastri has quit IRC | 11:35 | |
| *** cshastri has joined #openstack-infra | 11:36 | |
| *** noama has quit IRC | 11:37 | |
| *** cshastri has quit IRC | 11:41 | |
| *** annp has quit IRC | 11:44 | |
| *** heyongli has quit IRC | 11:44 | |
| *** heyongli has joined #openstack-infra | 11:45 | |
| *** edmondsw_ has quit IRC | 11:53 | |
| *** elefrancois has joined #openstack-infra | 11:53 | |
| *** heyongli has quit IRC | 11:55 | |
| *** heyongli has joined #openstack-infra | 11:55 | |
| *** alexchadin has quit IRC | 11:56 | |
| *** alexchadin has joined #openstack-infra | 11:58 | |
| *** dhill_ has joined #openstack-infra | 11:58 | |
| *** jpena|lunch is now known as jpena | 12:00 | |
| *** sree_ has joined #openstack-infra | 12:00 | |
| *** sree_ is now known as Guest2793 | 12:01 | |
| openstackgerrit | Nguyen Van Trung proposed openstack/diskimage-builder master: Add iscsi-boot element for CentOS images https://review.openstack.org/542708 | 12:02 |
| *** alexchadin has quit IRC | 12:02 | |
| *** alexchadin has joined #openstack-infra | 12:03 | |
| *** heyongli has quit IRC | 12:05 | |
| *** heyongli has joined #openstack-infra | 12:05 | |
| *** shardy has joined #openstack-infra | 12:12 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: build-python-release: ensure LC_ALL is set to en_US.utf8 https://review.openstack.org/577800 | 12:13 |
| *** boden has joined #openstack-infra | 12:14 | |
| *** trown|outtypewww has quit IRC | 12:15 | |
| *** heyongli has quit IRC | 12:15 | |
| *** rh-jelabarre has joined #openstack-infra | 12:15 | |
| *** heyongli has joined #openstack-infra | 12:16 | |
| boden | AJaeger ping | 12:19 |
| *** r-mibu has quit IRC | 12:19 | |
| *** ykarel_ has joined #openstack-infra | 12:20 | |
| AJaeger | boden: https://blogs.gnome.org/markmc/2014/02/20/naked-pings/ - please leave a message so that others can help | 12:21 |
| *** bobh has joined #openstack-infra | 12:21 | |
| *** weshay_ is now known as weshay|ruck | 12:22 | |
| *** ykarel has quit IRC | 12:22 | |
| boden | in regards to the stable/queens breakage in networking projects due to https://review.openstack.org/#/c/576526/ I’ve updated the blacklist in queens/requirements but had to exclude lbaas (see https://review.openstack.org/#/c/576988/)… still having breakage due to lbaas not in project requirements http://logs.openstack.org/25/576925/5/check/requirements-check/d1c73c3/job-output.txt.gz#_2018-06-25_10_52_29_193147 is the prope | 12:24 |
| boden | thing to do here just to not list neutron-lbaas in the proj requirements? | 12:24 |
| *** ykarel_ is now known as ykarel | 12:24 | |
| AJaeger | boden: taht looks fine to me... | 12:25 |
| boden | AJeager: looks fine == exclude lbaas from the proj requirements? | 12:25 |
| *** heyongli has quit IRC | 12:25 | |
| *** heyongli has joined #openstack-infra | 12:26 | |
| AJaeger | boden: yeah | 12:26 |
| *** edmondsw has joined #openstack-infra | 12:30 | |
| *** amoralej|lunch is now known as amoralej | 12:31 | |
| *** rlandy has joined #openstack-infra | 12:32 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix missing dequeue on synchronized pull request variant one https://review.openstack.org/577354 | 12:32 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix missing dequeue on synchronized pull request variant two https://review.openstack.org/577355 | 12:32 |
| *** germs_ has quit IRC | 12:35 | |
| *** trown has joined #openstack-infra | 12:35 | |
| *** heyongli has quit IRC | 12:36 | |
| *** heyongli has joined #openstack-infra | 12:36 | |
| *** tpsilva has joined #openstack-infra | 12:38 | |
| *** rosmaita has quit IRC | 12:40 | |
| *** heyongli has quit IRC | 12:46 | |
| *** heyongli has joined #openstack-infra | 12:47 | |
| *** evrardjp_ has joined #openstack-infra | 12:47 | |
| *** evrardjp has quit IRC | 12:50 | |
| *** evrardjp_ has quit IRC | 12:53 | |
| *** rosmaita has joined #openstack-infra | 12:53 | |
| *** bobh has quit IRC | 12:56 | |
| *** efried_pto is now known as efried | 12:56 | |
| *** heyongli has quit IRC | 12:56 | |
| *** heyongli has joined #openstack-infra | 12:57 | |
| *** efried has left #openstack-infra | 12:57 | |
| *** Guest2793 has quit IRC | 13:00 | |
| *** sree has joined #openstack-infra | 13:01 | |
| *** ramishra has quit IRC | 13:04 | |
| *** sree has quit IRC | 13:06 | |
| *** ramishra has joined #openstack-infra | 13:06 | |
| *** heyongli has quit IRC | 13:06 | |
| *** Guest71763 is now known as honza | 13:07 | |
| *** heyongli has joined #openstack-infra | 13:07 | |
| *** caphrim007 has quit IRC | 13:07 | |
| *** evrardjp_ has joined #openstack-infra | 13:07 | |
| openstackgerrit | Ilya Etingof proposed openstack-infra/project-config master: Enable doc and release notes for virtualbmc https://review.openstack.org/577811 | 13:08 |
| boden | AJeager that does not seem to work http://logs.openstack.org/25/576925/6/check/openstack-tox-py27/b09f26c/job-output.txt.gz#_2018-06-25_12_35_19_145187 | 13:09 |
| boden | lbaas is not installed now | 13:09 |
| boden | AJaeger ^ sorry I spelled your handle wrong | 13:10 |
| *** fmccrthy_ is now known as fmccrthy | 13:11 | |
| *** fmccrthy is now known as fmccarthy | 13:12 | |
| *** bdodd_ has joined #openstack-infra | 13:14 | |
| *** fmccarthy is now known as rtamalin | 13:14 | |
| *** kgiusti has joined #openstack-infra | 13:15 | |
| *** agopi has joined #openstack-infra | 13:15 | |
| *** katkapilatova has joined #openstack-infra | 13:15 | |
| *** evrardjp_ has quit IRC | 13:16 | |
| *** bdodd has quit IRC | 13:16 | |
| *** rtamalin is now known as fmccarthy | 13:16 | |
| *** heyongli has quit IRC | 13:17 | |
| *** heyongli has joined #openstack-infra | 13:17 | |
| *** evrardjp_ has joined #openstack-infra | 13:17 | |
| AJaeger | boden: http://logs.openstack.org/25/576925/6/check/openstack-tox-py27/b09f26c/job-output.txt.gz#_2018-06-25_12_29_01_286994 sets it up | 13:17 |
| AJaeger | boden: also http://logs.openstack.org/25/576925/6/check/openstack-tox-py27/b09f26c/job-output.txt.gz#_2018-06-25_12_35_04_467827 | 13:18 |
| AJaeger | boden, looks like neutron_lbaas exists but not ImportError: No module named neutron_lbaas.services.loadbalancer . Wsa it renamed? | 13:18 |
| boden | AJaeger: seems to be there https://github.com/openstack/neutron-lbaas/tree/stable/queens/neutron_lbaas/services/loadbalancer | 13:19 |
| AJaeger | boden: then I'm lost - hope somebody else can help... | 13:20 |
| boden | AJaeger I’ve been lost/broken for days on this stuff… | 13:21 |
| AJaeger | boden: I'm confused why lbaas is not in http://logs.openstack.org/25/576925/6/check/openstack-tox-py27/b09f26c/job-output.txt.gz#_2018-06-25_12_33_52_114306 | 13:21 |
| AJaeger | mordred: any ideas ? ^ | 13:22 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Static driver: fix node registration https://review.openstack.org/577464 | 13:25 |
| boden | AJaeger: should the project’s queens zuul.yaml perhaps be a copy of what we have in project-config/master? e.g. https://review.openstack.org/#/c/576925/6/.zuul.yaml should be the full list of jobs | 13:25 |
| *** heyongli has quit IRC | 13:27 | |
| *** heyongli has joined #openstack-infra | 13:27 | |
| AJaeger | boden: project-config is branchless and applies for all your branches | 13:28 |
| AJaeger | the jobs look setup correctly - but let's see whether mordred has an idea... | 13:28 |
| *** agopi has quit IRC | 13:29 | |
| *** bobh has joined #openstack-infra | 13:29 | |
| *** agopi has joined #openstack-infra | 13:30 | |
| *** sthussey has joined #openstack-infra | 13:31 | |
| *** agopi_ has joined #openstack-infra | 13:32 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Add tenant yaml validation option to scheduler https://review.openstack.org/574265 | 13:32 |
| *** caphrim007 has joined #openstack-infra | 13:33 | |
| *** bdodd_ has quit IRC | 13:33 | |
| *** bdodd has joined #openstack-infra | 13:34 | |
| *** agopi has quit IRC | 13:34 | |
| *** slaweq has joined #openstack-infra | 13:36 | |
| *** heyongli has quit IRC | 13:37 | |
| *** heyongli has joined #openstack-infra | 13:38 | |
| *** caphrim007 has quit IRC | 13:38 | |
| mordred | AJaeger: looking | 13:39 |
| *** slaweq has quit IRC | 13:41 | |
| mordred | AJaeger: yeah - from this: http://logs.openstack.org/25/576925/6/check/openstack-tox-py27/b09f26c/tox/py27-siblings.txt | 13:42 |
| mordred | it looks like the lbaas repo is in place and found, but lbaas is not installed in the tox env in the first place | 13:42 |
| *** agopi_ is now known as agopi | 13:44 | |
| mordred | https://review.openstack.org/#/c/576925/6/requirements.txt has lbaas commented out | 13:44 |
| boden | mordred yes because of http://logs.openstack.org/25/576925/5/check/requirements-check/d1c73c3/job-output.txt.gz#_2018-06-25_10_52_29_193147 if it’s not | 13:45 |
| *** germs has joined #openstack-infra | 13:45 | |
| *** germs has quit IRC | 13:45 | |
| *** germs has joined #openstack-infra | 13:45 | |
| boden | mordred that was my original question today; how to address http://logs.openstack.org/25/576925/5/check/requirements-check/d1c73c3/job-output.txt.gz#_2018-06-25_10_52_29_193147 so I commented out lbaas in the requiremetns.txt and now the results of lbaas not getting installed | 13:46 |
| boden | so I’m lost as how to address | 13:46 |
| mordred | well, you got neutron-lbaas into upper-constraints now with https://review.openstack.org/#/c/576526/2 yeah? | 13:47 |
| *** zoli is now known as zoli|afk | 13:47 | |
| boden | mordred I didn’t but yes.. are you suggesting to just constrain it in the requirements.txt? | 13:47 |
| *** heyongli has quit IRC | 13:47 | |
| *** heyongli has joined #openstack-infra | 13:48 | |
| mordred | well, I think we need to add neutron-lbaas to global-requirements in stable/queens | 13:48 |
| boden | mordred I tried that in https://review.openstack.org/#/c/576988/2 and it fails | 13:49 |
| boden | mordred looking now at the logs for ^, maybe it will work to add to global-requirements now… it appears the failures http://logs.openstack.org/88/576988/2/check/openstack-tox-validate/49ac035/job-output.txt.gz#_2018-06-21_12_41_01_701527 are not realted to lbaas | 13:51 |
| boden | I can try | 13:52 |
| *** eharney has joined #openstack-infra | 13:52 | |
| mordred | yeah - let's try that - because we need to be able to uncomment that neutron-lbaas line from the requirements.txt | 13:53 |
| boden | mordred ack | 13:53 |
| *** anteaya has joined #openstack-infra | 13:56 | |
| boden | mordred https://review.openstack.org/#/c/577825/ | 13:58 |
| *** heyongli has quit IRC | 13:58 | |
| *** heyongli has joined #openstack-infra | 13:58 | |
| mordred | sweet. | 13:58 |
| *** jcoufal has joined #openstack-infra | 14:00 | |
| *** salv-orlando has joined #openstack-infra | 14:03 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Add pause function to executor https://review.openstack.org/577827 | 14:04 |
| *** felipemonteiro__ has joined #openstack-infra | 14:07 | |
| *** salv-orlando has quit IRC | 14:08 | |
| *** heyongli has quit IRC | 14:08 | |
| *** heyongli has joined #openstack-infra | 14:08 | |
| *** felipemonteiro_ has joined #openstack-infra | 14:10 | |
| *** ihar has joined #openstack-infra | 14:12 | |
| *** felipemonteiro__ has quit IRC | 14:14 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Add pause function to executor https://review.openstack.org/577827 | 14:15 |
| *** heyongli has quit IRC | 14:18 | |
| *** heyongli has joined #openstack-infra | 14:19 | |
| *** janki has quit IRC | 14:22 | |
| *** rpioso|afk is now known as rpioso | 14:22 | |
| *** germs has quit IRC | 14:22 | |
| *** pjrusak has quit IRC | 14:24 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add pause function to executor https://review.openstack.org/577827 | 14:25 |
| *** yamamoto has quit IRC | 14:26 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: better handle existing keywords files/directories https://review.openstack.org/577633 | 14:27 |
| *** heyongli has quit IRC | 14:29 | |
| *** heyongli has joined #openstack-infra | 14:29 | |
| *** hongbin has joined #openstack-infra | 14:30 | |
| *** andreas_s has quit IRC | 14:30 | |
| *** andreas_s has joined #openstack-infra | 14:31 | |
| *** andreas_s has quit IRC | 14:32 | |
| *** andreas_s has joined #openstack-infra | 14:33 | |
| *** andreas_s has quit IRC | 14:34 | |
| *** andreas_s has joined #openstack-infra | 14:34 | |
| *** ykarel has quit IRC | 14:36 | |
| *** panda|off is now known as panda | 14:36 | |
| *** ykarel has joined #openstack-infra | 14:38 | |
| *** felipemonteiro_ has quit IRC | 14:39 | |
| *** felipemonteiro__ has joined #openstack-infra | 14:39 | |
| *** heyongli has quit IRC | 14:39 | |
| *** heyongli has joined #openstack-infra | 14:39 | |
| *** andreas_s has quit IRC | 14:39 | |
| *** jamesmcarthur has joined #openstack-infra | 14:40 | |
| *** andreas_s has joined #openstack-infra | 14:40 | |
| clarkb | ianw: thank you! I'm going to see if I can manage some hiera cleanups today | 14:43 |
| *** udesale has joined #openstack-infra | 14:45 | |
| *** heyongli has quit IRC | 14:49 | |
| *** heyongli has joined #openstack-infra | 14:49 | |
| *** zoli|afk is now known as zoli | 14:50 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Static driver: fix node registration https://review.openstack.org/577464 | 14:51 |
| clarkb | also the ssl certcheck warnings are gone. Confirming that my cert renewals appear to have worked as expected | 14:52 |
| mnaser | https://review.openstack.org/#/c/577719/1 | 14:52 |
| mnaser | we can merge this again to add capacity | 14:52 |
| clarkb | mnaser: noted, reviewing now | 14:53 |
| clarkb | done, I'll double check that puppet is enabled on that host now too | 14:53 |
| mnaser | clarkb: the kata container arch call is soon.. do you want to hop on to talk about a quick status update of things or got other stuff? i can do a little update too | 14:53 |
| *** andreas_s has quit IRC | 14:54 | |
| clarkb | mnaser: I can if you think that would be helpful. I think the biggest thing we are running into now is general test reliability. Since xenial crio is unhappy and fedora crashes | 14:54 |
| *** janki has joined #openstack-infra | 14:55 | |
| clarkb | I found the call details. I'll call in | 14:55 |
| mnaser | clarkb: i think it would be very helpful, the arch. calls usually has great coverage of devs/etc so they'll be able to chime in | 14:55 |
| mnaser | cool, i added it to the agenda here: https://etherpad.openstack.org/p/katacontainers-2018-architecture-committee-mtgs | 14:55 |
| * mnaser is trying to avoid openstack infra | 14:55 | |
| mnaser | :p | 14:55 |
| mnaser | (or using that term rather) | 14:55 |
| *** ykarel is now known as ykarel|away | 14:55 | |
| clarkb | sometimes I feel like I'm the one person that doesn't like zoom for these calls (I find the quality to be poor in particular) | 14:56 |
| *** felipemonteiro__ has quit IRC | 14:59 | |
| *** heyongli has quit IRC | 14:59 | |
| *** heyongli has joined #openstack-infra | 15:00 | |
| *** e0ne has quit IRC | 15:01 | |
| *** alexchadin has quit IRC | 15:02 | |
| *** andreas_s has joined #openstack-infra | 15:02 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Disable vexxhost" https://review.openstack.org/577719 | 15:02 |
| *** salv-orlando has joined #openstack-infra | 15:04 | |
| mordred | clarkb: I find zoom very hard to deal with | 15:04 |
| *** alexchadin has joined #openstack-infra | 15:05 | |
| *** rh-jelabarre has quit IRC | 15:05 | |
| anteaya | I don't like zoom | 15:06 |
| anteaya | if we are taking a poll | 15:06 |
| anteaya | chat doesn't work at all for me | 15:06 |
| prometheanfire | zoom is annoying in how they package things | 15:07 |
| anteaya | and if the group is just doing audio I don't understand why voip isn't being used | 15:07 |
| prometheanfire | they don't include all the needed qt libs from what I've seen | 15:07 |
| prometheanfire | they also leak their build env into the build | 15:07 |
| *** e0ne has joined #openstack-infra | 15:08 | |
| fungi | yeah, there are some zoom calls i need to join regularly, so i just use their dial-in number | 15:08 |
| *** salv-orlando has quit IRC | 15:08 | |
| fungi | and pretend it's nothing more than a phone bridge | 15:09 |
| anteaya | I didn't know zoom had dial-in | 15:09 |
| *** kukacz_ has quit IRC | 15:09 | |
| *** heyongli has quit IRC | 15:10 | |
| *** heyongli has joined #openstack-infra | 15:10 | |
| *** kukacz_ has joined #openstack-infra | 15:10 | |
| *** Bhujay has quit IRC | 15:12 | |
| *** pcaruana has quit IRC | 15:13 | |
| *** andreas_s has quit IRC | 15:14 | |
| *** ykarel|away has quit IRC | 15:14 | |
| *** andreas_s has joined #openstack-infra | 15:14 | |
| *** yamamoto has joined #openstack-infra | 15:16 | |
| clarkb | mnaser: as soon as the vexxhost revert change is applied I will recheck with a hold in place and get you the instance uuid for the fedora VM | 15:16 |
| mnaser | clarkb: cool, thanks | 15:16 |
| *** rh-jelabarre has joined #openstack-infra | 15:18 | |
| *** hemna_ has joined #openstack-infra | 15:19 | |
| *** heyongli has quit IRC | 15:20 | |
| *** heyongli has joined #openstack-infra | 15:20 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Add tenant yaml validation option to scheduler https://review.openstack.org/574265 | 15:27 |
| clarkb | mnaser: c9c168a6-919a-418f-9bbc-b7e433fc28b9 is the instance that is running the job. It just started though, no crash yet | 15:27 |
| mnaser | clarkb: ok lets keep an eye | 15:30 |
| clarkb | mnaser: http://zuul.openstack.org/stream.html?uuid=d133ebbfbbce4177bc04aa05f37ec34a&logfile=console.log is the console log stream | 15:30 |
| *** heyongli has quit IRC | 15:30 | |
| *** heyongli has joined #openstack-infra | 15:30 | |
| clarkb | mordred: anteaya: prometheanfire: I bottle it up on my phone. But that comes with drawbacks like chat being not great | 15:30 |
| prometheanfire | I looked at packaging it, but ran away | 15:31 |
| prometheanfire | I just extract the deb now | 15:31 |
| *** e0ne has quit IRC | 15:32 | |
| clarkb | one feature I've decied that all conference tooling like this needs is individual volume controls for each attendee on my side | 15:34 |
| mordred | clarkb: ++ | 15:34 |
| clarkb | that way I can turn up the person in their car on speakerphone and turn down the person with mic too close to mouth | 15:34 |
| mordred | clarkb: I've been much more pleased with bluejeans as a phone and linux user | 15:34 |
| clarkb | discord apparently supports this but its another not libre tool | 15:34 |
| clarkb | is free as in beer though | 15:35 |
| mordred | which is also a not-libre tool | 15:35 |
| mordred | but so far it's the least sucky non-free conferencing system i've had to interact with | 15:36 |
| clarkb | I had trouble with bluejeans in dublin, but that may have been conference networking more than anything else | 15:37 |
| *** sree has joined #openstack-infra | 15:37 | |
| *** andreas_s has quit IRC | 15:38 | |
| *** andreas_s has joined #openstack-infra | 15:38 | |
| prometheanfire | ya, I've used discord for stuff (Elite) | 15:38 |
| *** heyongli has quit IRC | 15:40 | |
| *** lpetrut has quit IRC | 15:40 | |
| *** heyongli has joined #openstack-infra | 15:41 | |
| *** andreas_s has quit IRC | 15:41 | |
| *** andreas_s has joined #openstack-infra | 15:41 | |
| *** sree has quit IRC | 15:41 | |
| *** apetrich has quit IRC | 15:42 | |
| *** andreas_s has quit IRC | 15:43 | |
| *** andreas_s has joined #openstack-infra | 15:43 | |
| *** electrofelix has quit IRC | 15:44 | |
| *** Guest72489 has quit IRC | 15:44 | |
| *** caphrim007 has joined #openstack-infra | 15:44 | |
| *** andreas_s has quit IRC | 15:44 | |
| *** apetrich has joined #openstack-infra | 15:45 | |
| *** andreas_s has joined #openstack-infra | 15:45 | |
| *** pbourke has quit IRC | 15:45 | |
| *** caphrim007 has quit IRC | 15:45 | |
| *** caphrim007 has joined #openstack-infra | 15:45 | |
| *** caphrim007 has quit IRC | 15:46 | |
| *** caphrim007 has joined #openstack-infra | 15:46 | |
| *** ramishra has quit IRC | 15:47 | |
| *** pbourke has joined #openstack-infra | 15:47 | |
| fungi | does anyone still use mumble? | 15:50 |
| prometheanfire | no, but they should | 15:50 |
| clarkb | fungi: some people do, but discord seems ot have really pushed into that space | 15:50 |
| *** heyongli has quit IRC | 15:51 | |
| *** heyongli has joined #openstack-infra | 15:51 | |
| prometheanfire | https://github.com/pi0/mangler | 15:51 |
| prometheanfire | Our project’s motto is: “No one should use our software… ever” | 15:51 |
| prometheanfire | the other byline was 'use mumble' :D | 15:51 |
| fungi | a lot of the projects which were big on sticking to free/open tools tended to use mumble for years, i just didn't know if it had gained any momentum | 15:52 |
| *** camunoz has joined #openstack-infra | 15:52 | |
| prometheanfire | the best thing about ventrilo was that you could overload the send buffer and those that recieved the audio would either crash or have the audio play, even if they disconnect it'd continue | 15:52 |
| prometheanfire | very fun for joining and playing silence then something trolly (that makes noise after you disconnect) | 15:53 |
| prometheanfire | also, tcp for voip, wtf | 15:53 |
| *** lpetrut has joined #openstack-infra | 15:54 | |
| *** andreas_s has quit IRC | 15:54 | |
| *** iyamahat has joined #openstack-infra | 15:55 | |
| clarkb | mnaser: it didn't crash this time, but did fail tests :/ | 15:56 |
| mnaser | clarkb: of course it would.. i dunno if kata ci is broken orrrr | 15:58 |
| fungi | maybe it didn't get far enough to trigger the crash | 15:58 |
| mnaser | that's my guess too fungi | 15:58 |
| clarkb | looks like some of the docker commands timed out | 15:58 |
| clarkb | maybe they crashed more gracefully this time | 15:59 |
| clarkb | the good news is this generates service logs people can look at | 15:59 |
| *** dtantsur is now known as dtantsur|afl | 15:59 | |
| *** dtantsur|afl is now known as dtantsur|afk | 15:59 | |
| pabelanger | so, if we ran the roles use-cached-repos and mirror-workspace-git-repos in a base job, then proceeded to run prepare-workspace role, for projects that mirrored using the /opt/git cache, they would be an noop for rsync in prepare-workspace right? | 16:00 |
| clarkb | pabelanger: not if rsync is used, git repos don't have to look the same on disk | 16:00 |
| clarkb | pabelanger: they can have different packfiles in particular | 16:01 |
| pabelanger | Hmm, okay | 16:01 |
| *** heyongli has quit IRC | 16:01 | |
| *** heyongli has joined #openstack-infra | 16:01 | |
| mnaser | i'm noticing a lot more jobs timing out at packethost | 16:01 |
| clarkb | the contents overall are the same, just written to disk in a different organizational structure | 16:01 |
| pabelanger | trying to come up with a solution to only using use-cached-repos and mirror-workspace-git-repos for git.o.o projects but prepare-workspace for everything else. Guess I'll need to add some condition checks to zuul.projects info | 16:02 |
| clarkb | mnaser: e-r at least doesn't seem to show a big spike | 16:02 |
| pabelanger | otherwise takes about 5 mins for prepare-workspace to run for dsvm jobs | 16:03 |
| mnaser | wonder if it's OSA specific.. every time i see a timed out job it's been on packethost :X | 16:03 |
| *** zoli is now known as zoli|gone | 16:03 | |
| *** zoli|gone is now known as zoli | 16:03 | |
| fungi | possible we need to switch to the flavor with 8vcpus instead of 4? | 16:04 |
| *** salv-orlando has joined #openstack-infra | 16:04 | |
| mnaser | this is 8 vcpus i think | 16:06 |
| mnaser | oh | 16:06 |
| mnaser | on packet | 16:06 |
| fungi | yeah | 16:06 |
| mnaser | let me see if it's hanging or running slower in the timeouts ive seen | 16:06 |
| fungi | i think we started out using a 4vcpu flavor thinking it might be sufficient | 16:06 |
| clarkb | depends on how well is parallelized | 16:06 |
| mnaser | http://logs.openstack.org/32/576932/1/gate/openstack-ansible-functional-opensuse-423/93e3a94/job-output.txt.gz#_2018-06-25_14_21_13_222754 | 16:07 |
| mnaser | it doesnt have to do with vcpus | 16:07 |
| mnaser | i'll try to track down more info if i see | 16:07 |
| clarkb | mnaser: thats weird, is it possible some wheel is being built that makes things go to lunch? | 16:08 |
| *** gfidente has quit IRC | 16:08 | |
| mnaser | it is but i don't have much more info beyond that :( | 16:08 |
| *** alexchadin has quit IRC | 16:08 | |
| mnaser | i've been trying to drag mirrors and make sure we're using mirrors in our job so i found a lot of cases | 16:08 |
| mnaser | https://review.openstack.org/#/c/577677/ if anyone doesnt mind doing that and the dependency would help :> | 16:09 |
| *** ykarel|away has joined #openstack-infra | 16:09 | |
| clarkb | mnaser: is only suse affected too would be a good thing to determine | 16:09 |
| *** salv-orlando has quit IRC | 16:09 | |
| mnaser | afaik it was failing on centos on the weekend too and xenial | 16:09 |
| mnaser | so it was not os specific | 16:09 |
| mnaser | what i always noticed was it was almost always packet though :x | 16:09 |
| *** heyongli has quit IRC | 16:11 | |
| *** heyongli has joined #openstack-infra | 16:11 | |
| clarkb | I approved the removal of the mirror and as ianw notes we'll need to manually clean up the configs and afs | 16:11 |
| clarkb | mnaser: it would be nice if the osa specific proxy defs would be written in OSA compat form and for everyone else | 16:12 |
| clarkb | but we can figure that out in a followup | 16:12 |
| mnaser | clarkb: yes, that is something that i would like us to do | 16:12 |
| mnaser | the problem right now is that the gates are so unstable that it's so hard to get stuff in so once that is stable enough that is the next step | 16:13 |
| mnaser | literally as i speak, a job failed in gate because of mariadb, gr | 16:15 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add release note for static driver changes https://review.openstack.org/577858 | 16:16 |
| *** tesseract has quit IRC | 16:18 | |
| *** heyongli has quit IRC | 16:21 | |
| *** Bhujay has joined #openstack-infra | 16:21 | |
| *** heyongli has joined #openstack-infra | 16:22 | |
| mnaser | hmm | 16:22 |
| mnaser | any infra-root can help me troubleshoot this? http://downloads.mariadb.com/MariaDB/mariadb-10.1.33/yum/centos7-amd64/ works, http://mirror.ca-ymq-1.vexxhost.openstack.org:8080/mariadb/MariaDB/mariadb-10.1.33/yum/centos7-amd64/ gives a 404 | 16:22 |
| clarkb | mnaser: ya let me check the proxy log | 16:23 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix missing dequeue on synchronized pull request variant two https://review.openstack.org/577355 | 16:23 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix missing dequeue on synchronized pull request https://review.openstack.org/577355 | 16:24 |
| fungi | mtreinish: reading up more on the "SQUAT failed to open index file" messages from cyrus/imap in mail.log, they're apparently just debug info indicating that the imapd performed an on-demand search due to lack of a search index. it shouldn't be causing the lpmqtt silence nor failure to clear the mbox | 16:24 |
| *** derekh has quit IRC | 16:24 | |
| *** ykarel|away has quit IRC | 16:24 | |
| *** shardy has quit IRC | 16:25 | |
| corvus | fungi: i can confirm those are harmless | 16:26 |
| fungi | thanks corvus! | 16:26 |
| *** jpich has quit IRC | 16:29 | |
| clarkb | mnaser: ProxyPassReverse "/mariadb" "https://downloads.mariadb.org/" needs a / at the end of "/mariadb/" | 16:29 |
| clarkb | mnaser: if you look the url gets rewritten to drop that / then things break | 16:29 |
| Shrews | infra-root: We good to go on a nodepool launcher restart? | 16:30 |
| clarkb | Shrews: I think so | 16:30 |
| fungi | wfm | 16:30 |
| mordred | ++ | 16:30 |
| Shrews | ok. will do them 1 at a time | 16:30 |
| clarkb | tobiash: approved (seems like you addresed corvus' comment) | 16:31 |
| tobiash | \o/ | 16:31 |
| mtreinish | fungi: hmm, then I wonder why it's failing | 16:31 |
| *** heyongli has quit IRC | 16:32 | |
| *** heyongli has joined #openstack-infra | 16:32 | |
| mtreinish | nothing changed in dec. from what I can tell | 16:32 |
| fungi | yeah, lpmqtt hadn't had any new commits for a year | 16:32 |
| *** jesslampe has joined #openstack-infra | 16:33 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Add missing / to mariadb proxy cache config https://review.openstack.org/577865 | 16:33 |
| fungi | mtreinish: symptomology suggests it's not finding messages it expects in the inbox, which would both account for it not emitting events and not removing those messages from the mbox | 16:33 |
| clarkb | mnaser: ^ I think that fixes it | 16:33 |
| corvus | fungi: i misread that and now i think 'smtpology' sounds like a great name for a program. | 16:34 |
| fungi | ooh, yes! | 16:34 |
| mtreinish | fungi: right, and when I looked through the code and the example message you provided the processing should have processed the message body correctly | 16:34 |
| fungi | mtreinish: it's occasionally (maybe once or twice a day?) logging that it found a messages which failed initial filtering, but then i don't find those message ids in the mbox either which makes it sound like lpmqtt is actually deleting the messages it doesn't have any use for | 16:35 |
| mtreinish | the only place the failure would have been is in the imap usage and getting the message body to the processing function | 16:35 |
| mtreinish | which was something I couldn't really test manually | 16:35 |
| openstackgerrit | Merged openstack-infra/system-config master: Drop ubuntu-mariadb mirror https://review.openstack.org/577328 | 16:35 |
| mtreinish | fungi: yeah lpmqtt will delete the messages after it's processed it and builds the dict to put on the queue to send over mqtt | 16:35 |
| mtreinish | that's a flag we turned on in the config | 16:36 |
| mtreinish | by default it just marks the message as read, but doesn't delete anything. But for space reasons we turned on delete processed messages | 16:36 |
| fungi | well it seems to delete the messages it says it's ignoring, but not other messages it should be processing | 16:37 |
| mtreinish | fungi: https://github.com/openstack-infra/lpmqtt/blob/master/lpmqtt/lp.py#L106 it looks like it should only delete if the output dict has a value (which means the processing returned something) | 16:38 |
| pabelanger | an interesting side affect from removing zuul-cloner from 3pci jobs, is now means the local CI will need to stand up more zuul-mergers to keep up with gerrit traffic. A single zuul-merger / zuul-executor doesn't seem to been keeping up right now | 16:39 |
| pabelanger | working to see if I can get some numbers and maybe report back | 16:39 |
| clarkb | pabelanger: isn't that more to do with zuulv3 vs v2? zuul-cloner under v3 is just a shim to put v3 merger merged refs in the right places | 16:40 |
| pabelanger | yah, zuulv2 vs zuulv3. | 16:40 |
| fungi | this is removing zuul-cloner from a zuulv2 environment i guess | 16:41 |
| fungi | so you're putting additional git pull load on the v2 mergers as a result? | 16:41 |
| clarkb | I think the cloner is orthogonal | 16:42 |
| *** heyongli has quit IRC | 16:42 | |
| pabelanger | fungi: in this case, we have zuulv3 running now, testing 3pci before we move rest of jobs over to it, but until we scale up zuul-mergers, we cannot flip the switch. Up until now it has been single zuul-merger / zuul-executor running | 16:42 |
| clarkb | v3 merges more stuff to evaluate potential configs | 16:42 |
| *** heyongli has joined #openstack-infra | 16:42 | |
| pabelanger | yah | 16:42 |
| openstackgerrit | Merged openstack-infra/nodepool master: Add release note for static driver changes https://review.openstack.org/577858 | 16:43 |
| Shrews | #status log Nodepool launchers restarted with latest code | 16:43 |
| openstackstatus | Shrews: finished logging | 16:43 |
| pabelanger | in our case, we aren't actually using any configs from upstream projects, this is just to merge patchset to proper sha1 | 16:43 |
| fungi | ohhh, so you mean moving from v2 to v3 you have to add required-projects entries instead of just relying on zuul-cloner, and so the mergers are performing work for more projects per job | 16:43 |
| *** jpena is now known as jpena|off | 16:43 | |
| Shrews | infra-root: launchers restarted. not seeing any immediate issues | 16:43 |
| fungi | thanks Shrews! | 16:43 |
| pabelanger | fungi: yes, we're now testing (legacy-dsvm-base-multinode) jobs and has overloaded our single merger / executor, trying to keep up with gerrit stream | 16:44 |
| pabelanger | our zuul is evaluating the patch to even see if we should run a job | 16:45 |
| pabelanger | https://review.rdoproject.org/zuul3/status.html | 16:45 |
| pabelanger | today, we have no nova jobs defined, but require-projects openstack/nova for dsvm | 16:45 |
| clarkb | ya so the bug as tristanC pointed out is that we do the merges even if include: [] is set | 16:45 |
| clarkb | at least for the config evaluation, may still need to do the cat job or whatever it is that updates the bsae repo state | 16:46 |
| pabelanger | clarkb: ah, okay. I missed that | 16:46 |
| openstackgerrit | Fatema Khalid Sherif proposed openstack-infra/storyboard-webclient master: Expose ordering criteria in search UI https://review.openstack.org/577870 | 16:46 |
| fungi | i wonder if we also don't yet have an option to say that we want to be able to incorporate repos from this connection into jobs but we want to ignore events for those repos? | 16:46 |
| clarkb | fungi: we have that option, but we ignore it and do unnecessary merges aiui | 16:46 |
| clarkb | it is a bug | 16:46 |
| clarkb | tristanC: was debugging it | 16:47 |
| fungi | oh, that is the include: [] merges bug you mean | 16:47 |
| fungi | okay, now i get it | 16:47 |
| pabelanger | yes, I need to follow up with tristanC to see if he has started work on a patch | 16:47 |
| fungi | i saw the discussion about the bug, just didn't realize that include: [] is also supposed to cause zuul to completely ignore events for the project | 16:47 |
| pabelanger | same, I new it would not load config for them | 16:48 |
| pabelanger | knew* | 16:48 |
| Shrews | corvus: pabelanger: fyi, there are 2 remaining ready+locked nodes that seem to be locked by zuul (ids 0000209647 and 0000209651 for requests 100-0004542709 and 100-0004542710, respectively). just fyi | 16:48 |
| mnaser | clarkb: ah dang, i dun goofed. thanks for noticing that. | 16:48 |
| *** heyongli has quit IRC | 16:52 | |
| *** heyongli has joined #openstack-infra | 16:52 | |
| *** gyee has joined #openstack-infra | 16:55 | |
| mnaser | http://logs.openstack.org/38/577838/1/check/openstack-ansible-functional-distro_install-opensuse-423/1fe2cd5/job-output.txt.gz#_2018-06-25_14_54_59_474847 | 16:55 |
| mnaser | another packethost job that timed out, opensuse too | 16:55 |
| *** amoralej is now known as amoralej|off | 16:56 | |
| *** udesale has quit IRC | 16:56 | |
| pabelanger | mnaser: maybe check to see what network limits are for packethost? the mirror is busy: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64951&rra_id=all | 16:57 |
| pabelanger | wonder if network is getting throttled | 16:57 |
| *** mriedem has quit IRC | 16:57 | |
| *** mriedem has joined #openstack-infra | 16:58 | |
| *** pcichy has quit IRC | 16:59 | |
| *** psachin` has quit IRC | 16:59 | |
| *** auristor has quit IRC | 17:01 | |
| *** heyongli has quit IRC | 17:02 | |
| *** heyongli has joined #openstack-infra | 17:03 | |
| *** salv-orlando has joined #openstack-infra | 17:05 | |
| *** jamesmcarthur has quit IRC | 17:06 | |
| Shrews | am i reading this graph wrong (http://grafana.openstack.org/dashboard/db/zuul-status?panelId=18&fullscreen), or are those Used and Deleting lines much higher on the graph than they should be? They seem to be obscurring the In Use line | 17:08 |
| clarkb | Shrews: it is additive (probably wrong term) so each line sits atop the one below it based on its value | 17:10 |
| Shrews | clarkb: oh, that explains it | 17:10 |
| clarkb | when deleting and used are small they sit very close to in use | 17:10 |
| clarkb | and ya can obscure it | 17:10 |
| *** salv-orlando has quit IRC | 17:10 | |
| Shrews | pebkac error confirmed :) | 17:11 |
| *** heyongli has quit IRC | 17:13 | |
| *** heyongli has joined #openstack-infra | 17:13 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/infra-specs master: Update config management for the Infra Control Plane https://review.openstack.org/565550 | 17:14 |
| mordred | infra-root: that should be ready for review now ^^ | 17:15 |
| clarkb | infra-root I've put some notes on the hiera cleanup I think I need to do or sort ou at the bottom of https://etherpad.openstack.org/p/201806-cert-refresh please leave notes if you have information on any of those | 17:16 |
| clarkb | mordred: nice | 17:16 |
| *** elbragstad is now known as lbragstad | 17:16 | |
| *** lpetrut has quit IRC | 17:19 | |
| *** lpetrut has joined #openstack-infra | 17:20 | |
| pabelanger | config-core: is there any reason not to move http://git.openstack.org/cgit/openstack-infra/project-config/tree/roles/use-cached-repos into openstack-zuul-jobs for 3pci? I don't believe it needs to be run in trusted context | 17:20 |
| *** yamahata has quit IRC | 17:22 | |
| clarkb | pabelanger: I agree it doesn't need to run in a trusted context, everything happens on the remote nodes | 17:22 |
| clarkb | and there are no secrets | 17:23 |
| *** heyongli has quit IRC | 17:23 | |
| mordred | pabelanger: I'd argue if we're going to move it somewhere we should move it to zuul-jobs - while the /opt/git cache is specific to our nodes, it's reasonable to imagine other people wanting to copy that system and use that role in their base jobs | 17:23 |
| *** heyongli has joined #openstack-infra | 17:23 | |
| mordred | and nothing about what it's doing is fundamentally tied to openstack | 17:23 |
| pabelanger | clarkb: mordred: zuul-jobs also works if we want to do that | 17:23 |
| mordred | corvus: ^^ thoughts? | 17:24 |
| pabelanger | right now, we (rdo) want to use it in the context of 3pci jobs for openstack, but it does apply to large git repos too, if they are included on nodes | 17:24 |
| *** e0ne has joined #openstack-infra | 17:26 | |
| *** iyamahat has quit IRC | 17:26 | |
| mordred | yah. and writing it up as an interface people can use should they wish to - but without requiring it - seems potentially empowering | 17:26 |
| mnaser | any infra-root mind +2W my goof here? https://review.openstack.org/#/c/577865/ | 17:26 |
| *** iranzo has quit IRC | 17:27 | |
| dhellmann | I have what might be an involved zuul question related to the python3 migration. What is the best forum for asking it (here or which mailing list)? | 17:27 |
| *** lpetrut has quit IRC | 17:28 | |
| clarkb | dhellmann: zuul channel may be best if its very zuul specific (or zuul's mailing list) | 17:28 |
| mordred | yeah - but if it's about an openstack job transition - I'd say here or the openstack-infra mailing list - kinda depends I guess | 17:29 |
| mordred | (it's a good question) | 17:29 |
| dhellmann | clarkb : it has to do with the issues you brought up about branch specifiers. I'm trying to work out if we should leave any jobs that are configured to only run on master in project-config rather than moving them | 17:29 |
| dhellmann | so it's about zuul, but it's really about our use of zuul | 17:29 |
| *** lpetrut has joined #openstack-infra | 17:30 | |
| dhellmann | I guess it's "involved" because I wanted to understand better clarkb's comment about mixing job settings in branches and having branch specifiers was "complicated" (I think that's the word he used) | 17:30 |
| clarkb | re complicated, the docs try to go over it | 17:30 |
| * clarkb finds a link | 17:31 | |
| *** jesslampe has quit IRC | 17:31 | |
| dhellmann | for example, translation-jobs is a project-template that has 1 job in 2 queues both times it is tied to the master branch | 17:31 |
| *** jesslampe has joined #openstack-infra | 17:31 | |
| *** lpetrut has quit IRC | 17:31 | |
| *** jesslampe has quit IRC | 17:31 | |
| dhellmann | so, should we just leave that in project-config? or should we copy it over to the repo and then as the branch ages to a stable/$foo that job will just not be run | 17:32 |
| *** lpetrut has joined #openstack-infra | 17:32 | |
| *** dhajare has quit IRC | 17:32 | |
| dhellmann | there's another job having to do with periodic jobs on stable branches, and for that one I'll probably make a new job template that doesn't include the branch specifiers and then we can move that one over | 17:32 |
| dhellmann | oh, sorry, 2 jobs in 1 queue | 17:33 |
| clarkb | dhellmann: https://zuul-ci.org/docs/zuul/user/config.html#attr-job.branches "Otherwise, all of the job variants which match that branch are used when freezing the job." | 17:33 |
| *** myoung is now known as myoung|biab | 17:33 | |
| clarkb | that sentence is the one that makes it complicated. Means that your stable config will be applied to the master branch if they both match | 17:33 |
| AJaeger | dhellmann: translation-jobs is special, we need to maintain that for every release and update - the translators don't handle all branches. So, in the past we said those stay in project-config. | 17:33 |
| *** heyongli has quit IRC | 17:33 | |
| AJaeger | dhellmann: but best talk with eumel8 on what to do there. | 17:33 |
| *** heyongli has joined #openstack-infra | 17:33 | |
| *** pjrusak has joined #openstack-infra | 17:33 | |
| dhellmann | AJaeger : have you read through https://review.openstack.org/#/c/575933/ yet? | 17:34 |
| dhellmann | so it sounds like anything that should only run on master needs to stay in project-config and anything that should run on stable branches can and should move | 17:34 |
| dhellmann | because we don't want stable branches adding jobs to master | 17:34 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Don't add implied branch matchers to project-pipeline variants https://review.openstack.org/577881 | 17:34 |
| clarkb | dhellmann: or if you drop the branches: specifier entirely and rely on the implicit applies to my current branch rule everything should just work | 17:35 |
| AJaeger | dhellmann: I read about that goal but did not understood these yet - let me read | 17:35 |
| *** jamesmcarthur has joined #openstack-infra | 17:35 | |
| dhellmann | clarkb : yeah, we can do that for stable branches, but not for things that should only run on master, right? | 17:35 |
| *** eernst has joined #openstack-infra | 17:35 | |
| clarkb | dhellmann: should work for all branches the same if you drop the explicit branches: rules | 17:36 |
| dhellmann | AJaeger : I rolled in some work to move most of the job settings out of project-config and into project trees to make them easier to manage (something the infra team asked for) | 17:36 |
| dhellmann | clarkb : ok, but if I don't want a job to run on a stable branch, like those translation jobs, then I don't see how moving it in-tree is the right thing to do? | 17:36 |
| *** pcichy has joined #openstack-infra | 17:36 | |
| dhellmann | I either have to move it with the branch specifier intact, or not move it | 17:36 |
| AJaeger | dhellmann: I'm fine moving those jobs over in general. The problem is that translations stop earlier than those are retired and not all branches are translated by default. So, going over 80 repositories for translations does not scale ;( | 17:37 |
| dhellmann | I'm looking at http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/project-templates.yaml#n613 | 17:37 |
| AJaeger | (for each release) | 17:37 |
| clarkb | ya I think those may be an exception where you want them in a branchless repo like project-config | 17:37 |
| dhellmann | AJaeger : ok, I would be happy to leave them in project-config, I'm trying to establish whether we have an exception here | 17:38 |
| dhellmann | and it sounds like it | 17:38 |
| dhellmann | so I'll update the docs to indicate that | 17:38 |
| clarkb | because yes, you make stable branch and now it either applies to stable and master or you have a branches: master rule on stable which will apply the stable config to the master branch which is weird | 17:38 |
| AJaeger | dhellmann: talk with eumel8 once he's around as i18n PTL - if he agrees, I'm not asking for an exception ;) | 17:38 |
| dhellmann | I just want to get the directions right, I don't care where things live :-) | 17:39 |
| dhellmann | there are going to be enough of these that I want to have clear detailed instructions so I don't have to do them all myself | 17:39 |
| AJaeger | dhellmann: and with the translation jobs, we have branch specifiers, so should be easy to do this in project-config | 17:39 |
| *** iyamahat has joined #openstack-infra | 17:39 | |
| AJaeger | dhellmann: I agree with KISS | 17:39 |
| corvus | mordred, pabelanger, clarkb: yeah, we also discussed maybe having a super version of that role which uses cached repos if they are there, or rsync if it's not | 17:39 |
| dhellmann | I see there are a couple of project templates that have specified other branches | 17:40 |
| dhellmann | it's probably easier to say "leave the translation settings in project-config" than worry about that | 17:40 |
| *** lpetrut has quit IRC | 17:40 | |
| *** pblaho has quit IRC | 17:40 | |
| *** lpetrut has joined #openstack-infra | 17:40 | |
| *** e0ne has quit IRC | 17:40 | |
| clarkb | as an alternative, we could have the translation jobs run on stable and master then just noop on stable | 17:41 |
| clarkb | but that will eat up test nodes | 17:41 |
| pabelanger | mordred: clarkb: corvus: yes, I'm happy to see about writing that. We are having that exact issue now. | 17:41 |
| *** yamahata has joined #openstack-infra | 17:42 | |
| corvus | pabelanger: maybe it already does that? what does happen today for us if we don't have a repo cached? | 17:42 |
| *** eernst has quit IRC | 17:42 | |
| AJaeger | corvus: are you happy with ianw's nodepool consolidation change? Please review https://review.openstack.org/#/c/577045/ - mnaser wanted your final blessing | 17:42 |
| corvus | dhellmann, clarkb, AJaeger: i agree it sounds like p-c is the best place for the translation jobs now | 17:43 |
| *** trown is now known as trown|lunch | 17:43 | |
| *** heyongli has quit IRC | 17:43 | |
| *** heyongli has joined #openstack-infra | 17:44 | |
| *** jamesmcarthur has quit IRC | 17:44 | |
| pabelanger | corvus: use-cached-repos will clone from canonical_hostname (https) if project is not found in /opt/git. So gerrit on our case, then mirror-workspace-git-repos would use git mirror from executor | 17:44 |
| dhellmann | clarkb , AJaeger , mordred , corvus : new draft at https://review.openstack.org/575933 | 17:44 |
| pabelanger | would need to test, but cloning from gerrit in our case, could increase the load on it, since rdo doesn't have git farm | 17:46 |
| clarkb | dhellmann: the diff against the previous patchset looks good to me | 17:46 |
| AJaeger | dhellmann: change looks fine to me , thanks | 17:46 |
| dhellmann | clarkb, AJaeger: thanks | 17:47 |
| corvus | AJaeger, mnaser, ianw: grafana change 577045 lgtm; did not approve because it's not 100% clear to me that's safe for grafana01 and i'm not up on the current status on grafana02. | 17:48 |
| clarkb | infra-root I'm going to start my hiera cleanup with cacti to make sure I understand the interfaction between fqdn and group. The cacti group is set in site.pp and groups.txt so should be fine and the fqdn file has no delta against the group file in hiera according todiff. This is a good first service since its for our use and less "user" facing | 17:49 |
| fungi | mtreinish: corvus: oho! i think cyrus-imapd is serving messages out of /var/spool/cyrus/mail/l/user/lpmqtt for some reason, but exim is delivering them to /var/spool/mail/lpmqtt instead (i find the non-lp messages it's complaining about inside the former tree). further circumstantial evidence, /etc/exim4/exim4.conf was last modified 2017-12-19 which is the same day we started piling up undeleted bug | 17:49 |
| fungi | e-mails in /var/spool/mail/lpmqtt and seems to also be when https://review.openstack.org/527498 was approved (the openstack_project::firehose class does supply custom "cyrus" exim router and transport though it's not immediately clear why it's broken) | 17:49 |
| AJaeger | corvus: let's ask ianw to self-approve 577045 then when he thinks it's fine - thanks | 17:50 |
| clarkb | ah the other thing to check is what files end up on disk in the target node(s) | 17:50 |
| clarkb | in this case fqdn/ is empty on our cacti.o.o host as its actually cacti02.openstack.org | 17:50 |
| clarkb | infra-root taking the hiera lock for this | 17:51 |
| fungi | mtreinish: corvus: i wonder if the messages end up matching the localuser router and so never hit the cyrus router now that 527498 changed the configuration order | 17:53 |
| *** e0ne has joined #openstack-infra | 17:53 | |
| *** heyongli has quit IRC | 17:54 | |
| fungi | that would 100% explain everything i'm seeing | 17:54 |
| *** heyongli has joined #openstack-infra | 17:54 | |
| fungi | since lpmqtt _is_ the name of a local shell account in addition to having a cyrus account | 17:54 |
| *** felipemonteiro has joined #openstack-infra | 17:54 | |
| *** jesslampe has joined #openstack-infra | 17:55 | |
| *** harlowja has joined #openstack-infra | 17:55 | |
| *** felipemonteiro_ has joined #openstack-infra | 17:56 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add NODEPOOL_MARIADB_PROXY and drop NODEPOOL_MARIADB_MIRROR https://review.openstack.org/577677 | 17:58 |
| *** janki has quit IRC | 17:59 | |
| *** felipemonteiro has quit IRC | 18:00 | |
| mtreinish | fungi: oh, that sounds like it would explain everything | 18:00 |
| corvus | fungi, mtreinish: i agree. maybe we should add a flag to disable the localuser router and we can add it back in at the end if we want it? | 18:01 |
| *** jamesmcarthur has joined #openstack-infra | 18:03 | |
| fungi | corvus: oh, like basically vendor in a copy of the localuser router in the array in puppet? | 18:03 |
| fungi | yeah, that could do the trick | 18:03 |
| *** heyongli has quit IRC | 18:04 | |
| corvus | fungi: yup. "disable the built-in local delivery for this host on which i'm running an actual mailstore" doesn't sound like a crazy flag to me :) | 18:04 |
| *** heyongli has joined #openstack-infra | 18:04 | |
| corvus | and following that up by re-implementing the local delivery as a backup option for accounts not in the mailstore also sounds not-crazy | 18:05 |
| fungi | awesome, i'll give that a shot. thanks! | 18:06 |
| *** felipemonteiro_ has quit IRC | 18:06 | |
| *** felipemonteiro_ has joined #openstack-infra | 18:06 | |
| *** salv-orlando has joined #openstack-infra | 18:06 | |
| *** salv-orlando has quit IRC | 18:11 | |
| *** heyongli has quit IRC | 18:14 | |
| *** heyongli has joined #openstack-infra | 18:14 | |
| *** jamesmcarthur has quit IRC | 18:14 | |
| *** jamesmcarthur has joined #openstack-infra | 18:15 | |
| *** jamesmcarthur has quit IRC | 18:15 | |
| *** jamesmcarthur has joined #openstack-infra | 18:15 | |
| *** diablo_rojo has joined #openstack-infra | 18:17 | |
| vkmc | andreaf, hey o/ I'm some tests in tempest, I was wondering if you could help me clarifying this... I'm not sure in which cases it's convenient to use the resource_setup or the setUp methods in tempest | 18:17 |
| *** jiapei has quit IRC | 18:17 | |
| vkmc | andreaf, tosky shared me this guide https://docs.openstack.org/tempest/latest/write_tests.html, and it's not very clear (or maybe I'm missing something) | 18:18 |
| *** jamesmcarthur has quit IRC | 18:18 | |
| vkmc | s/I'm some tests/I'm working on some tests/g | 18:18 |
| *** Hal has joined #openstack-infra | 18:18 | |
| *** Hal is now known as Guest56949 | 18:19 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/puppet-exim master: Make localuser router optional https://review.openstack.org/577893 | 18:20 |
| mtreinish | vkmc: heh, I wrote those docs. I tried to put the reasons to use each setup functions in the docstring in the code example under: https://docs.openstack.org/tempest/latest/write_tests.html#setupclass | 18:20 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Check "cyrus" Exim router before "localuser" https://review.openstack.org/577895 | 18:20 |
| mtreinish | if you don't know where to put it resource setup is normally fine | 18:20 |
| fungi | corvus: mtreinish: ^ | 18:20 |
| vkmc | mtreinish, o/ | 18:20 |
| mtreinish | vkmc: but it also depends on what you're trying to create and how things are being accessed by the test methods | 18:21 |
| vkmc | mtreinish, so... in my case I have a shared resource and in the gate we are running tests with concurrency == 8 | 18:21 |
| vkmc | mtreinish, so creating the resource as a class method will lead to race errors | 18:22 |
| vkmc | s/as a/in a/g | 18:22 |
| *** jamesmcarthur has joined #openstack-infra | 18:22 | |
| vkmc | mtreinish, that's the reason I was going to use setUp | 18:22 |
| mtreinish | vkmc: by default tempest concurrency is at the class level. So methods in the same test class will run serially | 18:22 |
| mtreinish | vkmc: but if you want to use setUp that's fine. That runs after setupclass (or any of the methods in that doc) | 18:24 |
| vkmc | mtreinish, all right | 18:24 |
| *** heyongli has quit IRC | 18:24 | |
| vkmc | in which cases using setUp for environment setup is advised instead of resources_setup? | 18:24 |
| *** heyongli has joined #openstack-infra | 18:25 | |
| *** dhajare has joined #openstack-infra | 18:26 | |
| mtreinish | it normally isn't used in tempest because most resource creation done at the test method level is just done in that test method (and cleaned up with an addCleanup). If it's a common resource between methods in the class we just do it at the class level (normally in resource_setup) and rely on running things in parallel as classes | 18:27 |
| mtreinish | this is normally because when tempest is creating resources it's going out to the service and creating real things which can have a lot of overhead | 18:27 |
| mtreinish | so if you can (or need to) share a resource do it at the class level | 18:28 |
| mtreinish | and if the test needs it's own thing make it explicitly created in the test | 18:28 |
| vkmc | all right | 18:28 |
| vkmc | mtreinish++ | 18:29 |
| vkmc | thanks | 18:29 |
| mtreinish | sure, np. Let me know if you've got any other questions | 18:30 |
| vkmc | will do | 18:31 |
| *** jamesmcarthur has quit IRC | 18:31 | |
| *** Swami has joined #openstack-infra | 18:31 | |
| mtreinish | also if you've got any suggestions on that doc let me know. I meant to circle back around and add more to it, but never really did | 18:31 |
| mtreinish | fungi: that lgtm, but I'm not an exim expert | 18:32 |
| *** heyongli has quit IRC | 18:35 | |
| *** heyongli has joined #openstack-infra | 18:35 | |
| fungi | me neither! | 18:35 |
| *** Guest56949 has quit IRC | 18:38 | |
| *** sshnaidm is now known as sshnaidm|afk | 18:41 | |
| *** jcoufal has quit IRC | 18:43 | |
| *** heyongli has quit IRC | 18:45 | |
| *** heyongli has joined #openstack-infra | 18:45 | |
| clarkb | infra-root any reason to keep infracloud hieradata around? | 18:45 |
| clarkb | I can't think of any and it will be in the git history | 18:45 |
| clarkb | also Jenkins hierdata | 18:46 |
| lbragstad | o/ hey all | 18:47 |
| lbragstad | i have a really dumb mailing list question | 18:47 |
| vkmc | mtreinish, will do | 18:47 |
| lbragstad | does our mailman deployment support In-Reply-To? i just subscribed to a mailing list but i want to reply to a message that's already in discussion | 18:48 |
| *** Swami_ has joined #openstack-infra | 18:48 | |
| *** jamesmcarthur has joined #openstack-infra | 18:49 | |
| clarkb | lbragstad: I want to say that if you preserve the message thread id (or whatever thta header is) then mailman figure it out | 18:49 |
| clarkb | fungi: ^ is that right | 18:49 |
| *** jesslampe has quit IRC | 18:50 | |
| lbragstad | so something like this would work? https://webapps.stackexchange.com/questions/23197/reply-to-mailman-archived-message | 18:50 |
| clarkb | ya that looks right | 18:52 |
| *** felipemonteiro_ has quit IRC | 18:52 | |
| *** felipemonteiro_ has joined #openstack-infra | 18:53 | |
| *** jcoufal has joined #openstack-infra | 18:53 | |
| *** jcoufal has quit IRC | 18:54 | |
| fungi | clarkb: i would delete the infracloud hiera data, yes | 18:54 |
| *** heyongli has quit IRC | 18:55 | |
| fungi | i assume we've already cleaned up the manually-tracked ansible inventory and associated cronjobs | 18:55 |
| *** heyongli has joined #openstack-infra | 18:55 | |
| clarkb | I don't think so, last time I did a global ansible command run a bunch of ic hosts showed up | 18:56 |
| clarkb | (they all failed bceause ssh didn't work) | 18:56 |
| clarkb | I think we removed that file from our puppet but not from disk on the puppet master | 18:56 |
| clarkb | ya /etc/ansible/hosts/infracloud still exists | 18:56 |
| clarkb | I can remove that file too and see what happens | 18:57 |
| *** jamesmcarthur has quit IRC | 18:57 | |
| fungi | lbragstad: yep, i've done exactly as described there in the past either replying to messages which went to the list before i subscribed or realizing after i deleted one that i wanted to reply. worth noting i'm playing around with mailman3 which i gather provides a means of obtaining the correct message-id from the ml archive without having to pull down a month of compressed raw messages | 18:58 |
| fungi | so... that'll be nice if so | 18:58 |
| *** jamesmcarthur has joined #openstack-infra | 18:59 | |
| clarkb | ok after lunch I will remove all of the fqdn/*ic.openstack.org files and group/infracloud.yaml as well was /etc/ansible/hosts/infracloud. Then also anything related to Jenkins in hiera | 19:00 |
| clarkb | I'm hoping that simplifying and cleaning this up will make it easier to udnerstand what is we are managing (and make it easier to implement mordred's spec) | 19:00 |
| mnaser | another opensuse packethost timeout.. http://zuul.openstack.org/stream.html?uuid=bf744d832a2b4bc0a7db2851eb1c7ff6&logfile=console.log | 19:00 |
| mnaser | can anyone ssh into that machine and see what's up? or maybe if we put a hold now | 19:01 |
| mnaser | just to see what it's hanging on | 19:01 |
| clarkb | mnaser: what is a job name to hold? | 19:01 |
| clarkb | and project I guess | 19:01 |
| mnaser | openstack/openstack-ansible-tests, openstack-ansible-functional-opensuse-423 | 19:01 |
| mnaser | i wonder if the process will be killed upon timeout or if it'll stay there stuck | 19:02 |
| fungi | lbragstad: oh, looks like you can also get the appropriate message id out of the html version of the archive if you view page source on a message and then look for "In-Reply-To" (though the message id will be url %-encoded which is sort of a pain) | 19:02 |
| clarkb | mnaser: I've put a hold in place | 19:03 |
| clarkb | going to eat lunch now will check if the hold caught anything after | 19:04 |
| *** jamesmcarthur has quit IRC | 19:04 | |
| *** jamesmcarthur has joined #openstack-infra | 19:04 | |
| fungi | lbragstad: for simple message ids it's not too bad... In-Reply-To=%3C20180126173020.24xbtzbtuutf7awu%40yuggoth.org%3E becomes In-Reply-To: <20180126173020.24xbtzbtuutf7awu@yuggoth.org> which isn't too terrible | 19:04 |
| *** heyongli has quit IRC | 19:05 | |
| fungi | could probably write a half-dozen-line python script to retrieve, extract and decode the right message-id from a pipermail archive page | 19:05 |
| *** heyongli has joined #openstack-infra | 19:06 | |
| *** salv-orlando has joined #openstack-infra | 19:07 | |
| *** salv-orlando has quit IRC | 19:11 | |
| *** rockyg has joined #openstack-infra | 19:13 | |
| openstackgerrit | Merged openstack-infra/storyboard master: use required enums to validate 'type' args https://review.openstack.org/545170 | 19:14 |
| openstackgerrit | Merged openstack-infra/storyboard master: mark worklist filter_criteria as a required field https://review.openstack.org/545405 | 19:15 |
| openstackgerrit | Merged openstack-infra/storyboard master: mark FilterCriterion title as a mandatory field https://review.openstack.org/545406 | 19:16 |
| *** heyongli has quit IRC | 19:16 | |
| *** heyongli has joined #openstack-infra | 19:16 | |
| *** iyamahat_ has joined #openstack-infra | 19:19 | |
| *** rockyg has quit IRC | 19:22 | |
| *** iyamahat has quit IRC | 19:22 | |
| lbragstad | fungi: oh - so for http://lists.openstack.org/pipermail/edge-computing/2018-June/000235.html | 19:26 |
| *** heyongli has quit IRC | 19:26 | |
| *** heyongli has joined #openstack-infra | 19:26 | |
| lbragstad | %3C678C1A99-2302-438D-8D5D-91ED44B10FF9%40windriver.com%3E would be what i'd put in the In-Reply-To header? | 19:26 |
| lbragstad | that's a neat trick, i was looking through the HTML version for the id, but didn't find it | 19:27 |
| *** pcaruana has joined #openstack-infra | 19:27 | |
| *** slaweq has joined #openstack-infra | 19:29 | |
| mnaser | another packethost timeout, this time on centos-7 http://logs.openstack.org/38/577838/1/check/openstack-ansible-functional-centos-7/089d056/ :( | 19:31 |
| fungi | lbragstad: well, after url %-decoding anyway, so <678C1A99-2302-438D-8D5D-91ED44B10FF9@windriver.com> | 19:35 |
| *** heyongli has quit IRC | 19:36 | |
| *** heyongli has joined #openstack-infra | 19:36 | |
| *** dhajare has quit IRC | 19:39 | |
| *** trown|lunch is now known as trown | 19:42 | |
| *** yamamoto has quit IRC | 19:44 | |
| *** yamamoto has joined #openstack-infra | 19:46 | |
| *** jesslampe has joined #openstack-infra | 19:46 | |
| *** heyongli has quit IRC | 19:46 | |
| *** heyongli has joined #openstack-infra | 19:47 | |
| *** auristor has joined #openstack-infra | 19:48 | |
| *** yamamoto has quit IRC | 19:49 | |
| AJaeger | could anybody review a change to infra-manual, please? https://review.openstack.org/#/c/574858/ is needed so that we can merge https://review.openstack.org/#/c/574860/ | 19:49 |
| *** yamamoto has joined #openstack-infra | 19:49 | |
| *** yamamoto has quit IRC | 19:49 | |
| *** heyongli has quit IRC | 19:57 | |
| *** rockyg has joined #openstack-infra | 19:57 | |
| *** heyongli has joined #openstack-infra | 19:57 | |
| *** yolanda has quit IRC | 20:01 | |
| *** yolanda has joined #openstack-infra | 20:01 | |
| clarkb | mnaser my suse hold hasn't caught anything yet | 20:02 |
| mnaser | https://review.openstack.org/#/c/577884/3 it timed out but maybe because the job was already running.. | 20:02 |
| mnaser | another job timed out on centos-7 at packethost | 20:02 |
| clarkb | mnaser: http://logs.openstack.org/38/577838/1/check/openstack-ansible-functional-centos-7/089d056/job-output.txt.gz#_2018-06-25_17_21_18_823220 is an odd one | 20:03 |
| clarkb | I wonder if we are losing network connectivity from the executor to the test node | 20:03 |
| *** florianf has quit IRC | 20:03 | |
| mnaser | i can provide many other jobs if that helps | 20:03 |
| mnaser | the repeating pattern has been packet | 20:03 |
| clarkb | mnaser: does osa try to use nested virt? I wonder if its crashing the VMs | 20:04 |
| mnaser | clarkb: all these failures are way before we hit tempest | 20:04 |
| clarkb | (and the logging we get is lagged behind due to buffering) | 20:04 |
| mnaser | there is a lot that happens from that state to tempest running | 20:05 |
| mnaser | that would be a lot of buffering :p | 20:05 |
| clarkb | ok | 20:05 |
| mnaser | http://logs.openstack.org/38/577838/1/check/openstack-ansible-functional-distro_install-ubuntu-xenial/0b57320/ is another job that failed at packet, same thing | 20:05 |
| mnaser | it's almost always related to pip | 20:05 |
| *** heyongli has quit IRC | 20:07 | |
| *** heyongli has joined #openstack-infra | 20:07 | |
| *** salv-orlando has joined #openstack-infra | 20:08 | |
| clarkb | we have other times but they don't appear to be so pip specific | 20:08 |
| clarkb | mnaser: do you use our pypi mirror? | 20:08 |
| mnaser | clarkb: there is a lot of things that werent using mirrors, i was working on pip mirrors, AFAIK they do, but with a held node i can confirm that | 20:09 |
| clarkb | seems like the majority of the timeouts in packethost are osa specific | 20:10 |
| clarkb | (which could be scheduling chance or indicate a relationship between the two) | 20:11 |
| mnaser | maybe we're not using the pip mirrors | 20:11 |
| mnaser | it's honestly very likely | 20:11 |
| clarkb | fungi: http://logs.openstack.org/58/565358/14/check/openstack-tox-py35/aa30b12/job-output.txt is a good chance that one is related to fewer cpus | 20:12 |
| mnaser | fwiw i think we should run jobs with a hardcoded nprocs of 8 | 20:12 |
| mnaser | because most of the time a lot of the things like tempest just sleep/wait for something | 20:12 |
| clarkb | mnaser: thtas up to the cloud flavor though | 20:13 |
| clarkb | not something we can hardcode (we have ability to tune it in the case of this cloud though) | 20:13 |
| *** salv-orlando has quit IRC | 20:13 | |
| clarkb | on osic we ran 4vcpu and it was faster than any other cloud | 20:13 |
| fungi | also varying the available cpu count has a tendency to expose threading races | 20:14 |
| fungi | so you don't accidentally build stuff that just happens to work with 8vcpus by accident | 20:15 |
| *** felipemonteiro_ has quit IRC | 20:15 | |
| mnaser | that's a good point fungi | 20:15 |
| *** felipemonteiro_ has joined #openstack-infra | 20:15 | |
| mnaser | if we can put a hold on openstack-ansible-functional-centos-7, openstack-ansible-functional-opensuse-423 or openstack-ansible-functional-ubuntu-xenial it would behelpful figuring this out | 20:15 |
| *** jesslampe has quit IRC | 20:15 | |
| clarkb | mnaser: I have put a hold on opensuse job | 20:16 |
| clarkb | just waiting for it to catch a failure now | 20:16 |
| clarkb | https://git.openstack.org/cgit/openstack/openstack-ansible-pip_install/tree/tasks/install_online.yml is the lsat thing logged in that one above | 20:16 |
| mnaser | they all seem to be pip related when they et stuck | 20:16 |
| clarkb | it says it has 5 retries left though and we don't see any of them logging | 20:17 |
| clarkb | and then sits for an hour and 14 minutes | 20:17 |
| mnaser | i'm guessing something happens where the process gets stuck | 20:17 |
| *** heyongli has quit IRC | 20:17 | |
| *** myoung|biab is now known as myoung | 20:17 | |
| *** heyongli has joined #openstack-infra | 20:17 | |
| clarkb | https://git.openstack.org/cgit/openstack/openstack-ansible-pip_install/tree/defaults/main.yml#n21 is the url it should be getting | 20:18 |
| mnaser | do we mirror that somewhere | 20:18 |
| clarkb | we bake it into our images | 20:18 |
| clarkb | /home/zuul/files/get-pip.py or something | 20:19 |
| clarkb | /opt/cache/files/get-pip.py is the path | 20:19 |
| clarkb | says project-config/nodepool/elements/cache-devstack/source-repository-pip | 20:20 |
| *** kgiusti has left #openstack-infra | 20:20 | |
| clarkb | on a random centos7 node in packethost I can wget https://bootstrap.pypa.io/get-pip.py just fine | 20:21 |
| *** lpetrut has quit IRC | 20:21 | |
| mnaser | clarkb: is it possible to get access to a vm that's running? i want to verify if the mirrors are being setup properly | 20:22 |
| mnaser | i'm okay with racing with the job to gather what i need | 20:23 |
| clarkb | mnaser: messages log has some interesting stuff in it | 20:23 |
| clarkb | mnaser: http://logs.openstack.org/38/577838/1/check/openstack-ansible-functional-centos-7/089d056/logs/host/messages.txt.gz I think it likely this is our problem | 20:23 |
| mnaser | oh? link? also yay now mariadb is not responding that made a job fail | 20:24 |
| * mnaser flips table | 20:24 | |
| mordred | clarkb, mnaser: "Failed to activate service 'org.freedesktop.login1': timed out" ... because of course what you want is a desktop login :) | 20:25 |
| mnaser | mordred: new meaning for an aio deployment, you get a desktop too with firefox and horizon as homepage | 20:25 |
| clarkb | http://logs.openstack.org/38/577838/1/check/openstack-ansible-functional-centos-7/089d056/logs/host/messages.txt.gz#_Jun_25_17_21_08 I think networking may be crashing? | 20:25 |
| clarkb | and it basically goes quiet until the job times out | 20:26 |
| mnaser | the timing lines up.. | 20:26 |
| mordred | clarkb: yup | 20:27 |
| *** katkapilatova has quit IRC | 20:27 | |
| *** heyongli has quit IRC | 20:27 | |
| *** heyongli has joined #openstack-infra | 20:28 | |
| clarkb | internet seems to think it is a guest vm kernel issue not so much the hypervisor but I wouldn't rule that out either at this point | 20:28 |
| clarkb | especially since suse and centos seem to be affected | 20:28 |
| mnaser | could be offloading | 20:28 |
| clarkb | ya could be bad hardware | 20:28 |
| clarkb | since its a checksum failure | 20:28 |
| mnaser | but we're not leaving the hypervisor | 20:29 |
| mnaser | clarkb: you don't have access to the physical infra right? | 20:30 |
| clarkb | the get-pip.py url is off hypervisor | 20:30 |
| mnaser | that's true | 20:30 |
| clarkb | mnaser: no, we have admin account on the cloud but no physical hypervisor access | 20:30 |
| * mnaser is wondering if all those bad vms are hitting the same bad hv | 20:30 | |
| clarkb | ya I wouldn't be surprised | 20:30 |
| clarkb | mnaser: lets get a list of jobs thta failed and we can work backward thorugh nodepool logs to get a list of instance uuids? | 20:30 |
| clarkb | then maybe john studarus can check if they all ended up on the same compute node | 20:31 |
| mnaser | the next time i see one stuck, we can `openstack server show` and find out what compute node its on | 20:31 |
| clarkb | mnaser: assuming we can figure out the admin creds ya, not sure how ianw set that up (but also ianw should be awake soon) | 20:31 |
| lbragstad | fungi: thanks for the tips, that worked! | 20:32 |
| mnaser | in the meantime, ips for failed jobs is good? | 20:32 |
| mnaser | or ips+timeline or build_id or? | 20:32 |
| clarkb | mnaser: http://logs.openstack.org/38/577838/1/check/openstack-ansible-functional-centos-7/089d056/zuul-info/host-info.centos-7.yaml ansible_nodename fact is what we can use to get the uuid from nodepool | 20:33 |
| clarkb | that is probably the best info at this point, maybe just collect a list of links of these host info files | 20:33 |
| clarkb | mordred: there isn't anything special about the hieradata/production/fqdn/foo.yaml files that just have {} in them right | 20:35 |
| clarkb | I think those must of been stub files for instances that didn't otherwise have hiera data, but I can delete them if they have group data | 20:35 |
| clarkb | fungi: fyi that hostinfo file claims 8vcpu | 20:36 |
| clarkb | crazy idea, nova should put a hypervisor instance id of some sort in the metadata of the intsance | 20:36 |
| clarkb | I guess that could be abused by users | 20:36 |
| clarkb | infra-root I am removing jenkins and infracloud hieradata now (will still be in git log) | 20:37 |
| *** heyongli has quit IRC | 20:38 | |
| *** jamesmcarthur has quit IRC | 20:38 | |
| *** heyongli has joined #openstack-infra | 20:38 | |
| mordred | clarkb: ++ | 20:38 |
| mnaser | clarkb: https://etherpad.openstack.org/p/packet-timeout-vms | 20:39 |
| mnaser | also i THINK nova does give a hypervisor id in the metadata | 20:39 |
| mnaser | let me check | 20:39 |
| mordred | clarkb: hostId is in the server record | 20:40 |
| mnaser | ^ that is visible via the API | 20:40 |
| mnaser | in this case I see a VM on our cloud with hostId '00c6db73f904bf75a27173783aa56cdac7df9bbe7e36ad44eabb6ece' | 20:40 |
| clarkb | mnaser: is it visiable in config drive? | 20:40 |
| mnaser | checking | 20:40 |
| clarkb | if so we should hvae glean log that on boot | 20:40 |
| clarkb | mordred: ^ | 20:40 |
| mordred | clarkb: oh - I don't know if it goes into config-drive | 20:41 |
| mnaser | im checking if it idoes | 20:41 |
| clarkb | infra-root my one concern with removing the jenkins stuff is our ssh private key for instance access but we seem to define the jenkins ssh proviate key in nodepool.yaml and zuul-executor.yaml | 20:41 |
| clarkb | my reading of the puppet is that we don't use the jenkins file content anymore so this should be a noop | 20:42 |
| mnaser | clarkb, mordred: nope, it's not there | 20:42 |
| mordred | kk | 20:43 |
| *** pcaruana has quit IRC | 20:43 | |
| clarkb | #status log Jenkins and Infracloud data removed from hieradata | 20:45 |
| openstackstatus | clarkb: finished logging | 20:45 |
| *** felipemonteiro__ has joined #openstack-infra | 20:45 | |
| clarkb | mnaser: thanks, grepping logs now for cloud instance uuids | 20:46 |
| *** heyongli has quit IRC | 20:48 | |
| *** heyongli has joined #openstack-infra | 20:48 | |
| *** felipemonteiro_ has quit IRC | 20:48 | |
| *** yamamoto has joined #openstack-infra | 20:50 | |
| clarkb | all have uuids post ed now | 20:52 |
| *** camunoz has quit IRC | 20:55 | |
| clarkb | I've confirmed that Ic760cc55f8e17fa7f39f2dd0433f5560aa8e2d65 removed /etc/ansible/hosts/infracloud from system-config so will remove the file from that dir on the puppetmaster | 20:55 |
| *** e0ne has quit IRC | 20:57 | |
| *** slaweq has quit IRC | 20:57 | |
| *** e0ne has joined #openstack-infra | 20:57 | |
| clarkb | I'm going to write an email to John now | 20:58 |
| fungi | thanks clarkb! | 20:58 |
| *** heyongli has quit IRC | 20:58 | |
| *** yamamoto has quit IRC | 20:58 | |
| *** heyongli has joined #openstack-infra | 20:58 | |
| *** e0ne has quit IRC | 20:59 | |
| *** e0ne has joined #openstack-infra | 20:59 | |
| *** e0ne has quit IRC | 21:00 | |
| *** trown is now known as trown|outtypewww | 21:00 | |
| *** dpawlik_ has joined #openstack-infra | 21:01 | |
| *** EmilienM has joined #openstack-infra | 21:03 | |
| *** kukacz_ has quit IRC | 21:03 | |
| *** felipemonteiro__ has quit IRC | 21:03 | |
| *** tobiash has quit IRC | 21:03 | |
| *** felipemonteiro__ has joined #openstack-infra | 21:03 | |
| *** niska` has joined #openstack-infra | 21:03 | |
| *** dpawlik has quit IRC | 21:03 | |
| *** stephenfin has quit IRC | 21:03 | |
| *** EmilienM_PTO has quit IRC | 21:04 | |
| *** harlowja has quit IRC | 21:04 | |
| *** e0ne has joined #openstack-infra | 21:04 | |
| *** EmilienM has quit IRC | 21:04 | |
| *** EmilienM has joined #openstack-infra | 21:04 | |
| *** gaelL_ has joined #openstack-infra | 21:05 | |
| *** felipemonteiro_ has joined #openstack-infra | 21:05 | |
| *** kukacz_ has joined #openstack-infra | 21:05 | |
| *** tobiash has joined #openstack-infra | 21:05 | |
| *** stephenfin has joined #openstack-infra | 21:05 | |
| clarkb | mnaser: https://access.redhat.com/solutions/654283 that may also hint at the problem? | 21:05 |
| *** niska has quit IRC | 21:06 | |
| *** gaelL has quit IRC | 21:06 | |
| clarkb | except that is vmware specific it appears? | 21:06 |
| *** e0ne has quit IRC | 21:07 | |
| *** e0ne has joined #openstack-infra | 21:07 | |
| *** heyongli has quit IRC | 21:08 | |
| *** salv-orlando has joined #openstack-infra | 21:08 | |
| *** felipemonteiro__ has quit IRC | 21:08 | |
| *** heyongli has joined #openstack-infra | 21:09 | |
| clarkb | generic-receive-offload is on and large-receive-offload is off on a random instance I check | 21:09 |
| *** pjrusak has quit IRC | 21:10 | |
| clarkb | mnaser: theory time, OSA uses lxc which sets up a linux bridge | 21:13 |
| clarkb | linux bridge can't be combined with gro or lro | 21:13 |
| *** salv-orlando has quit IRC | 21:13 | |
| clarkb | if this cloud is enabling this on the main interfaces then this could explain why osa is seeing it but everyone else isn't | 21:13 |
| clarkb | in general openstack testing does use bridges and ovs but it doesn't span them to the main host instance interface | 21:13 |
| clarkb | instead it does tunnels over them using virtual l2 devices | 21:14 |
| mnaser | i dont think we do either but im not sure how our ci works | 21:14 |
| clarkb | mnaser: I don't think you do directly but lxc does | 21:14 |
| mnaser | vms talk to the internet via nat on eth0 in the container | 21:14 |
| mnaser | and then we have br-mgmt which bridges all the vms together | 21:14 |
| clarkb | default lxc behavior is to bridge the lxc container bridge to your main interface iirc | 21:14 |
| mnaser | we don't want that so if that's a thing we need to stop | 21:15 |
| *** ldnunes has quit IRC | 21:15 | |
| mnaser | hey | 21:15 |
| mnaser | i have a stuck vm right now | 21:15 |
| mnaser | clarkb: http://zuul.openstack.org/stream.html?uuid=29ffaa56ae8c47ecaf9babe412062125&logfile=console.log | 21:15 |
| fungi | not often someone's happy about that ;) | 21:16 |
| mnaser | lol :p | 21:16 |
| mnaser | there shouldn't a bridged interface to the main one afaik though. should be nat is how it reaches things | 21:16 |
| ianw | clarkb: sorry, what admin settings are we talking about? | 21:17 |
| clarkb | ianw: being able to see what hypervisor an instance is running on | 21:18 |
| clarkb | in particular aa25e2a6-f298-4079-af4c-978f3c00c08f on packethost right now | 21:18 |
| *** heyongli has quit IRC | 21:19 | |
| *** heyongli has joined #openstack-infra | 21:19 | |
| clarkb | mnaser: ya ens3 isn't on the two bridges on that host | 21:19 |
| ianw | clarkb: oh, well the openstackci user has admin role in it's project now, so it has as much access as it can get? | 21:19 |
| mnaser | clarkb: as expected .. strace show anything? | 21:19 |
| mnaser | i guess the syslogs probably contain the same issue | 21:20 |
| clarkb | ianw: ah gotcha just reuse that user | 21:20 |
| clarkb | mnaser: they do | 21:20 |
| clarkb | the syslogs that is | 21:20 |
| clarkb | mnaser: strace shows pip blocking on read() | 21:21 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard master: add an index for the story_storytags table https://review.openstack.org/577931 | 21:21 |
| *** e0ne has quit IRC | 21:21 | |
| mnaser | i guess we're running into that same issue | 21:21 |
| clarkb | which lines up with networking crashed now everything is out for lunch | 21:21 |
| mnaser | do the etherfaces have gro/etc enabled | 21:22 |
| ianw | 20 | compute-amd64-09 | QEMU | 21:22 |
| ianw | they're all qemu | 21:22 |
| mnaser | also interesting that the interface name is ens3.. are they using virtio? :x | 21:22 |
| clarkb | mnaser: thats debuntu biosdevname behavior | 21:22 |
| mnaser | ah okay | 21:22 |
| clarkb | on centos it is eth0 | 21:23 |
| clarkb | ianw: like not kvm + qemu just qemu? | 21:23 |
| *** jesslampe has joined #openstack-infra | 21:23 | |
| clarkb | ianw: that would surprise me because I think these instances have vmx enabled in them implying it is kvm proper | 21:24 |
| ianw | hmm, that is just the output of "hypervisor list" | 21:25 |
| ianw | but yeah, it just says QEMU | 21:27 |
| *** heyongli has quit IRC | 21:29 | |
| *** heyongli has joined #openstack-infra | 21:29 | |
| mnaser | clarkb, mordred: https://review.openstack.org/#/c/577933/ :> | 21:30 |
| mnaser | http://lists.openstack.org/pipermail/openstack-dev/2018-June/131804.html hopefully it lands this release | 21:30 |
| clarkb | mnaser: fungi ianw I sent email to john with you included | 21:30 |
| clarkb | as a side note gro is enabled on our other clouds too | 21:31 |
| *** agopi has quit IRC | 21:31 | |
| mnaser | do we know what release of openstack is running/what type of networking | 21:31 |
| clarkb | still blocking on that read() so ya Ithink this is the issue, kernel breaks on that read, never returns to user space and it just sits there | 21:31 |
| clarkb | mnaser: I don't recall but fungi may have gotten those details | 21:32 |
| mnaser | we could probably find out with curl http://169.254.169.254/openstack | 21:32 |
| mnaser | the dates there would corrolate to what is the newest api version there | 21:32 |
| clarkb | 2016-10-06 | 21:33 |
| clarkb | is the most recent date | 21:33 |
| clarkb | no hostid in the meta_data.json from that though | 21:34 |
| mnaser | clarkb: because the change above is the one to add it :P | 21:34 |
| mnaser | NEWTON_TWO = '2016-10-06' | 21:34 |
| mnaser | so if '2017-02-22' that means its a newton cloud | 21:34 |
| clarkb | compute-amd64-08 is the hypervisor according to my admin srever show | 21:35 |
| openstackgerrit | Kendall Nelson proposed openstack-infra/storyboard master: Add to & Reorganize StoryBoard GUI Docs https://review.openstack.org/577934 | 21:36 |
| *** nicolasbock has quit IRC | 21:37 | |
| openstackgerrit | Kendall Nelson proposed openstack-infra/storyboard master: Add to & Reorganize StoryBoard GUI Docs https://review.openstack.org/577934 | 21:38 |
| *** heyongli has quit IRC | 21:39 | |
| *** heyongli has joined #openstack-infra | 21:39 | |
| *** jesslampe has quit IRC | 21:40 | |
| *** kjackal has quit IRC | 21:42 | |
| *** felipemonteiro_ has quit IRC | 21:42 | |
| clarkb | fungi: can we remove fqdn/release.slave.openstack.org.yaml from hiera? I think you did a bunch of the porting work for what used to run on that server? | 21:43 |
| *** rockyg has quit IRC | 21:44 | |
| *** rcernin has joined #openstack-infra | 21:45 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add missing / to mariadb proxy cache config https://review.openstack.org/577865 | 21:48 |
| *** hashar has quit IRC | 21:48 | |
| *** heyongli has quit IRC | 21:49 | |
| *** heyongli has joined #openstack-infra | 21:50 | |
| fungi | clarkb: definitely. that host no longer exists | 21:52 |
| fungi | clarkb: mnaser: i was not provided any software version or network implementation info, no | 21:54 |
| *** boden has quit IRC | 21:54 | |
| *** yamamoto has joined #openstack-infra | 21:54 | |
| *** edmondsw has quit IRC | 21:58 | |
| *** jesslampe has joined #openstack-infra | 21:59 | |
| *** yamamoto has quit IRC | 21:59 | |
| *** heyongli has quit IRC | 22:00 | |
| *** heyongli has joined #openstack-infra | 22:00 | |
| *** jesslampe has quit IRC | 22:05 | |
| ianw | clarkb / fungi : while i think of it, a small stack to stop keystone warnings every time we look at packethost too -> https://review.openstack.org/577331 | 22:06 |
| *** salv-orlando has joined #openstack-infra | 22:09 | |
| *** ihar has quit IRC | 22:10 | |
| *** heyongli has quit IRC | 22:10 | |
| *** heyongli has joined #openstack-infra | 22:10 | |
| *** bobh has quit IRC | 22:11 | |
| *** salv-orlando has quit IRC | 22:14 | |
| *** harlowja has joined #openstack-infra | 22:14 | |
| *** mriedem has quit IRC | 22:18 | |
| *** heyongli has quit IRC | 22:20 | |
| *** heyongli has joined #openstack-infra | 22:20 | |
| *** yolanda_ has joined #openstack-infra | 22:21 | |
| clarkb | I'm going to leave stackalytics hiera because it still has stuff in system-config/manifests/site.pp | 22:21 |
| clarkb | pabelanger: is fqdn/zuulv3-dev.openstack.org.yaml and fqdn/zuulv3.openstack.org.yaml safe to delete from hiera? | 22:22 |
| *** yolanda has quit IRC | 22:22 | |
| fungi | should be. neither of those fqdns exist now | 22:23 |
| clarkb | puppetdb.openstack.org is another one I'll cleanup as we don't run that service anymore. Also it still hsa dns record(s) I will clean up | 22:27 |
| clarkb | actually we don't still have ansible puppet configured to submit those records do we? | 22:27 |
| clarkb | if so I worry changing dns might start causing everyting to "fail". Need to investigate | 22:27 |
| *** yolanda__ has joined #openstack-infra | 22:28 | |
| *** heyongli has quit IRC | 22:30 | |
| *** heyongli has joined #openstack-infra | 22:31 | |
| *** yolanda_ has quit IRC | 22:31 | |
| clarkb | we skip that task properly | 22:32 |
| clarkb | corvus: pabelanger I am going to remove hiera data for fqdn/zuulv3.openstack.org.yaml and fqdn/zuulv3-dev.openstack.org as these hosts don't seem to exist anymore | 22:37 |
| corvus | ++ | 22:39 |
| *** heyongli has quit IRC | 22:41 | |
| *** heyongli has joined #openstack-infra | 22:41 | |
| clarkb | #status log puppetdb.openstack.org A and AAAA dns records removed | 22:48 |
| openstackstatus | clarkb: finished logging | 22:48 |
| *** heyongli has quit IRC | 22:51 | |
| *** heyongli has joined #openstack-infra | 22:51 | |
| openstackgerrit | Merged openstack-infra/storyboard master: add an index for the story_storytags table https://review.openstack.org/577931 | 22:54 |
| *** yamamoto has joined #openstack-infra | 22:56 | |
| *** edmondsw has joined #openstack-infra | 22:57 | |
| *** agopi has joined #openstack-infra | 22:58 | |
| *** Hal has joined #openstack-infra | 23:00 | |
| *** Hal is now known as Guest63248 | 23:00 | |
| *** heyongli has quit IRC | 23:01 | |
| clarkb | mnaser: still something weird with the proxying of mariadb | 23:01 |
| *** heyongli has joined #openstack-infra | 23:01 | |
| *** yamamoto has quit IRC | 23:02 | |
| clarkb | I think it may be that we proxy at :8080/mariadb but downloads.mariadb.org roots their hosting at /mariadb as well | 23:02 |
| *** caphrim007_ has joined #openstack-infra | 23:03 | |
| *** caphrim007 has quit IRC | 23:06 | |
| clarkb | mnaser: reading logs I see a 301 | 23:08 |
| mnaser | clarkb: yeah maybe the URL is confusing it? Could you do a small change to have it say mdb and see what it does | 23:09 |
| *** caphrim007_ has quit IRC | 23:09 | |
| mnaser | I’m not sure I understand the root cause | 23:09 |
| *** salv-orlando has joined #openstack-infra | 23:10 | |
| *** heyongli has quit IRC | 23:11 | |
| *** caphrim007 has joined #openstack-infra | 23:11 | |
| *** heyongli has joined #openstack-infra | 23:12 | |
| clarkb | mnaser: using browser debugging it almost looks like it is redirecty to itself and same path which maybe confuses the proxy? | 23:12 |
| clarkb | oh wait that is yum/ vs yum it redirects yum to yum/ | 23:12 |
| mnaser | I noticed trailing slashes are important | 23:13 |
| *** caphrim007 has quit IRC | 23:14 | |
| *** salv-orlando has quit IRC | 23:15 | |
| *** hongbin has quit IRC | 23:17 | |
| *** pahuang has quit IRC | 23:19 | |
| *** heyongli has quit IRC | 23:22 | |
| *** heyongli has joined #openstack-infra | 23:22 | |
| *** pahuang has joined #openstack-infra | 23:23 | |
| clarkb | aha | 23:23 |
| clarkb | .com not .org | 23:23 |
| fungi | if anybody knows of any boards i storyboard.openstack.org which were previously very slow to load, please try them again and see if they're any snappier | 23:24 |
| fungi | er, boards in | 23:24 |
| clarkb | mnaser: ya fiddling with it in my browser with network dumps open and modifying configs on the server I have a config that works. Will push in a moment | 23:25 |
| ianw | oh i guess we need to absent the mirror link via puppet for mariadb, lest we go and manually delete it all | 23:26 |
| clarkb | oh that might be part of the problem | 23:28 |
| clarkb | I got it to work by changing the prefix to MariaDB and plumbing that straight to https://downloads.mariadb.com/MariaDB | 23:28 |
| clarkb | the com and the different prefix both seem necessary as switching to old prefix isn't working for me yet | 23:29 |
| clarkb | oh I get it | 23:30 |
| *** tpsilva has quit IRC | 23:30 | |
| clarkb | they are doing relative redirects and those don't get rewritten on the pass back | 23:31 |
| clarkb | so we need to share the same MariaDB prefix | 23:31 |
| clarkb | (so that the redirects work | 23:31 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Remove mariadb link from mirrors https://review.openstack.org/577943 | 23:31 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Revert "Remove mariadb link from mirrors" https://review.openstack.org/577944 | 23:31 |
| *** heyongli has quit IRC | 23:32 | |
| *** heyongli has joined #openstack-infra | 23:32 | |
| *** jamesmcarthur has joined #openstack-infra | 23:33 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Fix MariaDB proxy backend details https://review.openstack.org/577945 | 23:33 |
| clarkb | mnaser: ianw ^ I think that fixes it | 23:34 |
| *** tosky has quit IRC | 23:34 | |
| *** heyongli has quit IRC | 23:42 | |
| *** heyongli has joined #openstack-infra | 23:42 | |
| ianw | clarkb: i stacked https://review.openstack.org/#/c/577331/ ontop of a basic cloud.yaml syntax parser, which also requires updating the linters to python3, because i wanted to use python3 tempfile stuff in the checker | 23:44 |
| *** auristor has quit IRC | 23:48 | |
| *** gyee has quit IRC | 23:51 | |
| *** heyongli has quit IRC | 23:52 | |
| *** heyongli has joined #openstack-infra | 23:53 | |
| *** rh-jelabarre has quit IRC | 23:53 | |
| *** jamesmcarthur has quit IRC | 23:55 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add debian-security mirror https://review.openstack.org/577316 | 23:56 |
| *** Guest63248 has quit IRC | 23:56 | |
| *** rpioso is now known as rpioso|afk | 23:57 | |
| *** yamamoto has joined #openstack-infra | 23:57 | |
| *** auristor has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!