| *** caphrim007 has quit IRC | 00:03 | |
| *** bobh has quit IRC | 00:03 | |
| *** caphrim007 has joined #openstack-infra | 00:04 | |
| *** caphrim007 has quit IRC | 00:05 | |
| *** heyongli has quit IRC | 00:06 | |
| *** heyongli has joined #openstack-infra | 00:06 | |
| *** caphrim007 has joined #openstack-infra | 00:06 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Fix paused handler exception handling https://review.openstack.org/575515 | 00:08 |
|---|---|---|
| *** caphrim007 has quit IRC | 00:11 | |
| *** heyongli has quit IRC | 00:16 | |
| *** heyongli has joined #openstack-infra | 00:17 | |
| *** yamamoto has joined #openstack-infra | 00:23 | |
| *** heyongli has quit IRC | 00:26 | |
| *** heyongli has joined #openstack-infra | 00:27 | |
| *** yamamoto has quit IRC | 00:27 | |
| *** rlandy is now known as rlandy|afk | 00:29 | |
| openstackgerrit | qingszhao proposed openstack-infra/grafyaml master: change default python 3 env in tox to 3.5 https://review.openstack.org/575574 | 00:33 |
| *** heyongli has quit IRC | 00:37 | |
| *** heyongli has joined #openstack-infra | 00:37 | |
| *** hongbin has joined #openstack-infra | 00:42 | |
| *** dingyichen has joined #openstack-infra | 00:42 | |
| *** germs has joined #openstack-infra | 00:45 | |
| *** germs has quit IRC | 00:45 | |
| *** germs has joined #openstack-infra | 00:45 | |
| *** heyongli has quit IRC | 00:47 | |
| *** heyongli has joined #openstack-infra | 00:47 | |
| *** jesslampe has quit IRC | 00:48 | |
| *** germs has quit IRC | 00:50 | |
| *** r-daneel has joined #openstack-infra | 00:52 | |
| *** hongbin has quit IRC | 00:54 | |
| *** heyongli has quit IRC | 00:57 | |
| *** heyongli has joined #openstack-infra | 00:57 | |
| *** sthussey has joined #openstack-infra | 01:00 | |
| *** eernst has quit IRC | 01:06 | |
| *** heyongli has quit IRC | 01:07 | |
| *** heyongli has joined #openstack-infra | 01:08 | |
| *** gyee has quit IRC | 01:09 | |
| *** eernst has joined #openstack-infra | 01:09 | |
| *** bobh has joined #openstack-infra | 01:10 | |
| *** eernst has quit IRC | 01:10 | |
| *** eernst_ has joined #openstack-infra | 01:11 | |
| *** lifeless_ has quit IRC | 01:13 | |
| *** lifeless has joined #openstack-infra | 01:13 | |
| *** eernst_ has quit IRC | 01:15 | |
| *** heyongli has quit IRC | 01:18 | |
| *** heyongli has joined #openstack-infra | 01:18 | |
| *** xinliang3 has quit IRC | 01:18 | |
| *** caphrim007 has joined #openstack-infra | 01:20 | |
| *** lifeless_ has joined #openstack-infra | 01:23 | |
| *** yamamoto has joined #openstack-infra | 01:24 | |
| *** lifeless has quit IRC | 01:25 | |
| *** heyongli has quit IRC | 01:28 | |
| *** heyongli has joined #openstack-infra | 01:28 | |
| *** yamamoto has quit IRC | 01:29 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: angular6 fix suggestion https://review.openstack.org/573494 | 01:30 |
| *** rlandy|afk is now known as rlandy | 01:36 | |
| *** heyongli has quit IRC | 01:38 | |
| *** heyongli has joined #openstack-infra | 01:38 | |
| *** caphrim007 has quit IRC | 01:39 | |
| *** lifeless has joined #openstack-infra | 01:40 | |
| *** lifeless_ has quit IRC | 01:41 | |
| *** hongbin has joined #openstack-infra | 01:47 | |
| *** markvoelker has joined #openstack-infra | 01:47 | |
| *** heyongli has quit IRC | 01:48 | |
| *** heyongli has joined #openstack-infra | 01:49 | |
| *** lifeless has quit IRC | 01:54 | |
| *** heyongli has quit IRC | 01:59 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add default network to packethost https://review.openstack.org/575547 | 01:59 |
| *** heyongli has joined #openstack-infra | 01:59 | |
| *** lifeless has joined #openstack-infra | 02:00 | |
| *** kjackal has joined #openstack-infra | 02:05 | |
| *** mriedem has quit IRC | 02:07 | |
| *** heyongli has quit IRC | 02:09 | |
| *** heyongli has joined #openstack-infra | 02:09 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add support for Ansible extra-vars flag https://review.openstack.org/546474 | 02:10 |
| *** diablo_rojo has quit IRC | 02:15 | |
| *** heyongli has quit IRC | 02:19 | |
| *** heyongli has joined #openstack-infra | 02:19 | |
| *** markvoelker has quit IRC | 02:21 | |
| *** yamamoto has joined #openstack-infra | 02:26 | |
| *** heyongli has quit IRC | 02:29 | |
| *** heyongli has joined #openstack-infra | 02:30 | |
| *** yamamoto has quit IRC | 02:31 | |
| *** hemna_ has quit IRC | 02:31 | |
| *** boris_42_ has quit IRC | 02:38 | |
| *** heyongli has quit IRC | 02:40 | |
| *** heyongli has joined #openstack-infra | 02:40 | |
| *** bobh has quit IRC | 02:44 | |
| *** germs has joined #openstack-infra | 02:46 | |
| *** germs has quit IRC | 02:46 | |
| *** germs has joined #openstack-infra | 02:46 | |
| *** heyongli has quit IRC | 02:50 | |
| *** diablo_rojo has joined #openstack-infra | 02:50 | |
| *** agopi has quit IRC | 02:50 | |
| *** germs has quit IRC | 02:50 | |
| *** germs has joined #openstack-infra | 02:50 | |
| *** germs has quit IRC | 02:50 | |
| *** germs has joined #openstack-infra | 02:50 | |
| *** heyongli has joined #openstack-infra | 02:50 | |
| *** agopi has joined #openstack-infra | 02:50 | |
| ianw | oh dear, looks like we don't validate the yaml :/ | 02:57 |
| *** germs has quit IRC | 03:00 | |
| *** heyongli has quit IRC | 03:00 | |
| *** heyongli has joined #openstack-infra | 03:00 | |
| *** annp has joined #openstack-infra | 03:03 | |
| *** bobh has joined #openstack-infra | 03:03 | |
| *** agopi has quit IRC | 03:07 | |
| *** bobh has quit IRC | 03:08 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add missing __init__.py files https://review.openstack.org/575591 | 03:09 |
| *** heyongli has quit IRC | 03:10 | |
| *** heyongli has joined #openstack-infra | 03:11 | |
| *** ramishra has joined #openstack-infra | 03:13 | |
| *** bobh has joined #openstack-infra | 03:15 | |
| *** d0ugal_ has joined #openstack-infra | 03:17 | |
| *** d0ugal has quit IRC | 03:17 | |
| *** markvoelker has joined #openstack-infra | 03:18 | |
| *** heyongli has quit IRC | 03:21 | |
| *** heyongli has joined #openstack-infra | 03:21 | |
| *** yamamoto has joined #openstack-infra | 03:27 | |
| *** rlandy has quit IRC | 03:27 | |
| *** psachin has joined #openstack-infra | 03:27 | |
| *** bobh has quit IRC | 03:29 | |
| *** david-lyle has joined #openstack-infra | 03:30 | |
| *** sthussey has quit IRC | 03:30 | |
| *** heyongli has quit IRC | 03:31 | |
| *** heyongli has joined #openstack-infra | 03:31 | |
| *** dklyle has quit IRC | 03:32 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add support for Ansible extra-vars flag https://review.openstack.org/546474 | 03:32 |
| *** yamamoto has quit IRC | 03:33 | |
| *** eernst has joined #openstack-infra | 03:33 | |
| *** bobh has joined #openstack-infra | 03:34 | |
| *** bobh has quit IRC | 03:38 | |
| *** heyongli has quit IRC | 03:41 | |
| *** heyongli has joined #openstack-infra | 03:41 | |
| *** bobh has joined #openstack-infra | 03:42 | |
| *** agopi has joined #openstack-infra | 03:42 | |
| *** lpetrut has joined #openstack-infra | 03:46 | |
| *** bobh has quit IRC | 03:47 | |
| *** markvoelker has quit IRC | 03:51 | |
| *** heyongli has quit IRC | 03:51 | |
| *** heyongli has joined #openstack-infra | 03:52 | |
| *** bobh has joined #openstack-infra | 03:56 | |
| *** yamahata has joined #openstack-infra | 03:56 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Fix typo in packethost configuration https://review.openstack.org/575605 | 04:00 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: tools/sorted_modules.py : make python3 safe https://review.openstack.org/575606 | 04:00 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add a clouds.yaml checker https://review.openstack.org/575607 | 04:00 |
| *** bobh has quit IRC | 04:00 | |
| *** heyongli has quit IRC | 04:02 | |
| *** heyongli has joined #openstack-infra | 04:02 | |
| *** bobh has joined #openstack-infra | 04:03 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add a clouds.yaml checker https://review.openstack.org/575607 | 04:05 |
| *** ykarel_ has joined #openstack-infra | 04:08 | |
| *** haleyb has quit IRC | 04:08 | |
| *** bobh has quit IRC | 04:08 | |
| *** haleyb has joined #openstack-infra | 04:09 | |
| *** heyongli has quit IRC | 04:12 | |
| *** heyongli has joined #openstack-infra | 04:12 | |
| *** bobh has joined #openstack-infra | 04:15 | |
| *** dhajare has joined #openstack-infra | 04:20 | |
| *** gouthamr has quit IRC | 04:22 | |
| *** stevebaker has quit IRC | 04:22 | |
| *** heyongli has quit IRC | 04:22 | |
| *** lpetrut has quit IRC | 04:22 | |
| *** heyongli has joined #openstack-infra | 04:23 | |
| *** dmellado has quit IRC | 04:23 | |
| *** bobh has quit IRC | 04:23 | |
| *** bobh has joined #openstack-infra | 04:24 | |
| *** bobh has quit IRC | 04:28 | |
| *** yamamoto has joined #openstack-infra | 04:29 | |
| *** hongbin has quit IRC | 04:30 | |
| *** heyongli has quit IRC | 04:32 | |
| *** ociuhandu_ has joined #openstack-infra | 04:32 | |
| *** heyongli has joined #openstack-infra | 04:33 | |
| *** yamamoto has quit IRC | 04:33 | |
| *** yamamoto has joined #openstack-infra | 04:33 | |
| *** ociuhandu has quit IRC | 04:34 | |
| *** germs has joined #openstack-infra | 04:37 | |
| *** germs has quit IRC | 04:37 | |
| *** germs has joined #openstack-infra | 04:37 | |
| *** bobh has joined #openstack-infra | 04:37 | |
| *** germs has quit IRC | 04:41 | |
| *** bobh has quit IRC | 04:42 | |
| *** heyongli has quit IRC | 04:43 | |
| *** heyongli has joined #openstack-infra | 04:43 | |
| *** bobh has joined #openstack-infra | 04:45 | |
| *** markvoelker has joined #openstack-infra | 04:48 | |
| *** bobh has quit IRC | 04:50 | |
| *** heyongli has quit IRC | 04:53 | |
| *** heyongli has joined #openstack-infra | 04:53 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove failed_when when creating /tmp/console-None.log https://review.openstack.org/574672 | 04:53 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Don't run setfiles on /boot/efi https://review.openstack.org/575315 | 04:54 |
| *** bobh has joined #openstack-infra | 04:54 | |
| *** links has joined #openstack-infra | 04:55 | |
| *** eernst has quit IRC | 04:58 | |
| *** caphrim007 has joined #openstack-infra | 04:59 | |
| *** bobh has quit IRC | 04:59 | |
| *** slaweq has joined #openstack-infra | 05:00 | |
| *** caphrim007 has quit IRC | 05:03 | |
| *** heyongli has quit IRC | 05:03 | |
| *** heyongli has joined #openstack-infra | 05:03 | |
| *** bobh has joined #openstack-infra | 05:06 | |
| tristanC | ianw: perhaps it's codesearch that causes spikes on git servers? | 05:07 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Rename action-general to actiongeneral https://review.openstack.org/575617 | 05:09 |
| *** bobh has quit IRC | 05:11 | |
| ianw | tristanC: there is a spike about every 10 minutes, but codesearch doesn't update that frequently ... it's once a day iirc? | 05:12 |
| *** heyongli has quit IRC | 05:13 | |
| *** janki has joined #openstack-infra | 05:13 | |
| *** nicolasbock has joined #openstack-infra | 05:14 | |
| *** heyongli has joined #openstack-infra | 05:14 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add supercedent pipeline manager https://review.openstack.org/571932 | 05:15 |
| *** caphrim007 has joined #openstack-infra | 05:17 | |
| *** nicolasbock has quit IRC | 05:18 | |
| *** pcaruana has quit IRC | 05:18 | |
| *** markvoelker has quit IRC | 05:21 | |
| *** heyongli has quit IRC | 05:24 | |
| *** heyongli has joined #openstack-infra | 05:24 | |
| *** bobh has joined #openstack-infra | 05:25 | |
| *** vivsoni has quit IRC | 05:26 | |
| tristanC | ianw: it depends on the ms-between-poll setting | 05:27 |
| *** vivsoni has joined #openstack-infra | 05:27 | |
| *** bobh has quit IRC | 05:30 | |
| *** heyongli has quit IRC | 05:34 | |
| *** dhajare has quit IRC | 05:34 | |
| ianw | tristanC: yeah it's once a day -> https://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/codesearch.pp#n30 | 05:34 |
| *** heyongli has joined #openstack-infra | 05:34 | |
| *** bobh has joined #openstack-infra | 05:34 | |
| ianw | ohhh, but that's the complete reindex, i see, maybe it does poll known projects | 05:35 |
| tristanC | ianw: the default poll is every 30sec | 05:39 |
| *** bobh has quit IRC | 05:39 | |
| openstackgerrit | Merged openstack-infra/system-config master: Fix typo in packethost configuration https://review.openstack.org/575605 | 05:41 |
| *** heyongli has quit IRC | 05:44 | |
| *** bobh has joined #openstack-infra | 05:44 | |
| *** heyongli has joined #openstack-infra | 05:44 | |
| *** salv-orlando has joined #openstack-infra | 05:47 | |
| *** bobh has quit IRC | 05:49 | |
| *** ykarel__ has joined #openstack-infra | 05:50 | |
| *** aeng has quit IRC | 05:50 | |
| *** salv-orlando has quit IRC | 05:52 | |
| *** lathiat_ has joined #openstack-infra | 05:52 | |
| *** lathiat_ has quit IRC | 05:52 | |
| *** ykarel_ has quit IRC | 05:53 | |
| *** lathiat has quit IRC | 05:53 | |
| *** lathiat has joined #openstack-infra | 05:54 | |
| *** heyongli has quit IRC | 05:54 | |
| *** heyongli has joined #openstack-infra | 05:55 | |
| *** lathiat has quit IRC | 05:57 | |
| *** bobh has joined #openstack-infra | 05:57 | |
| *** lathiat has joined #openstack-infra | 06:00 | |
| *** lathiat has quit IRC | 06:00 | |
| *** pgadiya has joined #openstack-infra | 06:02 | |
| *** pgadiya has quit IRC | 06:02 | |
| *** bobh has quit IRC | 06:02 | |
| *** lathiat has joined #openstack-infra | 06:04 | |
| *** bobh has joined #openstack-infra | 06:04 | |
| *** heyongli has quit IRC | 06:05 | |
| *** heyongli has joined #openstack-infra | 06:05 | |
| *** dhajare has joined #openstack-infra | 06:08 | |
| *** bobh has quit IRC | 06:09 | |
| *** iyamahat has joined #openstack-infra | 06:10 | |
| *** dmsimard has quit IRC | 06:13 | |
| *** heyongli has quit IRC | 06:15 | |
| *** heyongli has joined #openstack-infra | 06:15 | |
| *** iyamahat has quit IRC | 06:17 | |
| *** bobh has joined #openstack-infra | 06:18 | |
| *** lpetrut has joined #openstack-infra | 06:22 | |
| *** agopi has quit IRC | 06:24 | |
| *** heyongli has quit IRC | 06:25 | |
| *** heyongli has joined #openstack-infra | 06:25 | |
| *** dmsimard has joined #openstack-infra | 06:27 | |
| *** bobh has quit IRC | 06:33 | |
| *** heyongli has quit IRC | 06:35 | |
| *** heyongli has joined #openstack-infra | 06:36 | |
| *** bobh has joined #openstack-infra | 06:39 | |
| *** alexchadin has joined #openstack-infra | 06:41 | |
| *** vivsoni has quit IRC | 06:42 | |
| *** Bhujay has joined #openstack-infra | 06:43 | |
| flaper87 | ianw: hey, around? | 06:44 |
| *** bobh has quit IRC | 06:44 | |
| *** pcaruana has joined #openstack-infra | 06:44 | |
| flaper87 | ianw: fungi set an autohold for a job yday and I was wondering wether you had a way to give me access to those nodes | 06:45 |
| *** heyongli has quit IRC | 06:46 | |
| *** bobh has joined #openstack-infra | 06:46 | |
| *** heyongli has joined #openstack-infra | 06:46 | |
| *** anteaya has quit IRC | 06:47 | |
| *** vivsoni has joined #openstack-infra | 06:48 | |
| *** slaweq has quit IRC | 06:50 | |
| *** bobh has quit IRC | 06:50 | |
| *** ykarel_ has joined #openstack-infra | 06:52 | |
| *** ykarel__ has quit IRC | 06:55 | |
| *** zul has quit IRC | 06:56 | |
| *** zul has joined #openstack-infra | 06:56 | |
| *** heyongli has quit IRC | 06:56 | |
| *** heyongli has joined #openstack-infra | 06:56 | |
| *** bobh has joined #openstack-infra | 06:57 | |
| *** slaweq has joined #openstack-infra | 07:01 | |
| *** bobh has quit IRC | 07:02 | |
| ianw | flaper87: umm, yep, if they've held ... | 07:03 |
| flaper87 | ianw: https://github.com/flaper87.keys that's my key | 07:04 |
| flaper87 | oh wait, do you mean they weren't hold? | 07:04 |
| ianw | there's three nodes on hold, do you know the ip address or something of the one you're interested in? | 07:04 |
| flaper87 | 2s, getting you the job name | 07:04 |
| *** dmellado has joined #openstack-infra | 07:04 | |
| flaper87 | ianw: tripleo-ci-centos-7-scenario009-multinode-oooq review 574233 | 07:05 |
| flaper87 | ianw: there should be 2 nodes | 07:05 |
| *** heyongli has quit IRC | 07:06 | |
| ianw | ok 23.253.207.46 and 23.253.207.225 | 07:06 |
| *** heyongli has joined #openstack-infra | 07:07 | |
| *** ykarel_ is now known as ykarel | 07:07 | |
| *** rcernin has quit IRC | 07:08 | |
| *** germs has joined #openstack-infra | 07:08 | |
| *** germs has quit IRC | 07:08 | |
| *** germs has joined #openstack-infra | 07:08 | |
| *** bobh has joined #openstack-infra | 07:08 | |
| ianw | flaper87: ok ,try root@ ^ addresses | 07:09 |
| *** zhangfei has joined #openstack-infra | 07:10 | |
| flaper87 | ianw: brilliant, I'm in | 07:10 |
| flaper87 | thanks | 07:10 |
| flaper87 | will let you know when I'm done with it | 07:10 |
| *** e0ne has joined #openstack-infra | 07:11 | |
| *** bobh has quit IRC | 07:12 | |
| *** ramishra has quit IRC | 07:13 | |
| *** germs has quit IRC | 07:13 | |
| *** heyongli has quit IRC | 07:16 | |
| *** tesseract has joined #openstack-infra | 07:17 | |
| *** dpawlik has joined #openstack-infra | 07:17 | |
| *** heyongli has joined #openstack-infra | 07:17 | |
| *** bobh has joined #openstack-infra | 07:18 | |
| *** markvoelker has joined #openstack-infra | 07:18 | |
| *** ramishra has joined #openstack-infra | 07:20 | |
| *** bobh has quit IRC | 07:22 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: package-afs-centos: fix package name https://review.openstack.org/575665 | 07:25 |
| *** heyongli has quit IRC | 07:27 | |
| *** heyongli has joined #openstack-infra | 07:27 | |
| *** tosky has joined #openstack-infra | 07:28 | |
| *** bobh has joined #openstack-infra | 07:30 | |
| *** alexchadin has quit IRC | 07:32 | |
| *** amoralej|off is now known as amoralej | 07:32 | |
| *** florianf has joined #openstack-infra | 07:33 | |
| *** bobh has quit IRC | 07:35 | |
| *** heyongli has quit IRC | 07:37 | |
| *** heyongli has joined #openstack-infra | 07:37 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Add iscsi-boot element https://review.openstack.org/511494 | 07:38 |
| *** gfidente has joined #openstack-infra | 07:40 | |
| *** dmellado has quit IRC | 07:42 | |
| *** dmellado has joined #openstack-infra | 07:42 | |
| *** alexchadin has joined #openstack-infra | 07:43 | |
| *** diablo_rojo has quit IRC | 07:44 | |
| *** Bhujay has quit IRC | 07:46 | |
| *** jpena|off is now known as jpena | 07:47 | |
| *** heyongli has quit IRC | 07:47 | |
| *** heyongli has joined #openstack-infra | 07:47 | |
| *** bobh has joined #openstack-infra | 07:49 | |
| *** ramishra has quit IRC | 07:49 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: job: add ansible-tags and ansible-skip-tags attribute https://review.openstack.org/575672 | 07:50 |
| *** markvoelker has quit IRC | 07:50 | |
| *** hashar has joined #openstack-infra | 07:51 | |
| *** ramishra has joined #openstack-infra | 07:51 | |
| *** ccamacho has joined #openstack-infra | 07:51 | |
| *** bobh has quit IRC | 07:54 | |
| *** jbadiapa_ is now known as jbadiapa | 07:54 | |
| *** heyongli has quit IRC | 07:57 | |
| *** heyongli has joined #openstack-infra | 07:58 | |
| *** verdurin has quit IRC | 07:59 | |
| *** verdurin has joined #openstack-infra | 07:59 | |
| flaper87 | ianw: hey, I'm done with those nodes and ready to re-run the job. Can we keep the autohold active? | 08:01 |
| flaper87 | would like to investigate the nodes one more time | 08:01 |
| *** Bhujay has joined #openstack-infra | 08:02 | |
| *** Bhujay has quit IRC | 08:02 | |
| *** d0ugal_ has quit IRC | 08:04 | |
| *** d0ugal has joined #openstack-infra | 08:04 | |
| *** d0ugal has quit IRC | 08:04 | |
| *** d0ugal has joined #openstack-infra | 08:04 | |
| *** jpich has joined #openstack-infra | 08:06 | |
| *** heyongli has quit IRC | 08:08 | |
| *** heyongli has joined #openstack-infra | 08:08 | |
| *** bobh has joined #openstack-infra | 08:08 | |
| *** bobh has quit IRC | 08:13 | |
| *** Bhujay has joined #openstack-infra | 08:14 | |
| *** dougsz has joined #openstack-infra | 08:14 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: zk: use kazoo retry facilities https://review.openstack.org/535537 | 08:18 |
| *** bobh has joined #openstack-infra | 08:18 | |
| *** heyongli has quit IRC | 08:18 | |
| *** heyongli has joined #openstack-infra | 08:18 | |
| *** threestrands has quit IRC | 08:20 | |
| *** bobh has quit IRC | 08:23 | |
| *** janki has quit IRC | 08:25 | |
| *** alexchadin has quit IRC | 08:27 | |
| *** heyongli has quit IRC | 08:28 | |
| *** heyongli has joined #openstack-infra | 08:29 | |
| *** bobh has joined #openstack-infra | 08:29 | |
| *** bauzas is now known as PapaOurs | 08:30 | |
| *** alexchadin has joined #openstack-infra | 08:30 | |
| *** bobh has quit IRC | 08:34 | |
| *** derekh has joined #openstack-infra | 08:36 | |
| *** alexchadin has quit IRC | 08:36 | |
| *** bobh has joined #openstack-infra | 08:37 | |
| *** heyongli has quit IRC | 08:38 | |
| *** heyongli has joined #openstack-infra | 08:39 | |
| *** bobh has quit IRC | 08:41 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Don't run setfiles on /boot/efi https://review.openstack.org/575315 | 08:42 |
| *** yamamoto has quit IRC | 08:45 | |
| *** bobh has joined #openstack-infra | 08:47 | |
| *** ykarel is now known as ykarel|lunch | 08:47 | |
| *** markvoelker has joined #openstack-infra | 08:48 | |
| *** heyongli has quit IRC | 08:49 | |
| *** heyongli has joined #openstack-infra | 08:49 | |
| *** bobh has quit IRC | 08:51 | |
| *** sileht has joined #openstack-infra | 08:53 | |
| *** panda|off is now known as panda | 08:54 | |
| *** alexchadin has joined #openstack-infra | 08:56 | |
| *** lifeless has quit IRC | 08:57 | |
| *** heyongli has quit IRC | 08:59 | |
| *** heyongli has joined #openstack-infra | 08:59 | |
| *** bobh has joined #openstack-infra | 09:02 | |
| *** gouthamr has joined #openstack-infra | 09:02 | |
| *** electrofelix has joined #openstack-infra | 09:03 | |
| *** yamamoto has joined #openstack-infra | 09:03 | |
| *** bobh has quit IRC | 09:06 | |
| *** bobh has joined #openstack-infra | 09:07 | |
| *** germs has joined #openstack-infra | 09:09 | |
| *** germs has quit IRC | 09:09 | |
| *** germs has joined #openstack-infra | 09:09 | |
| *** heyongli has quit IRC | 09:09 | |
| *** heyongli has joined #openstack-infra | 09:09 | |
| *** lifeless has joined #openstack-infra | 09:10 | |
| *** bobh has quit IRC | 09:11 | |
| *** jtomasek_ has joined #openstack-infra | 09:12 | |
| *** jtomasek has quit IRC | 09:12 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Fix acceptance test setup script for puppet 4 https://review.openstack.org/575698 | 09:12 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-cgit master: Update Gemfile for Zuulv3 https://review.openstack.org/571987 | 09:13 |
| *** germs has quit IRC | 09:14 | |
| *** janki has joined #openstack-infra | 09:16 | |
| *** heyongli has quit IRC | 09:19 | |
| *** heyongli has joined #openstack-infra | 09:20 | |
| *** markvoelker has quit IRC | 09:22 | |
| *** ykarel|lunch is now known as ykarel | 09:25 | |
| *** stevebaker has joined #openstack-infra | 09:26 | |
| *** heyongli has quit IRC | 09:30 | |
| *** heyongli has joined #openstack-infra | 09:30 | |
| *** bobh has joined #openstack-infra | 09:30 | |
| *** zoli is now known as zoli|lunch | 09:35 | |
| *** bobh has quit IRC | 09:35 | |
| openstackgerrit | Nguyen Van Trung proposed openstack/diskimage-builder master: Add iscsi-boot element for CentOS images https://review.openstack.org/542708 | 09:37 |
| *** psachin has quit IRC | 09:38 | |
| *** lifeless has quit IRC | 09:38 | |
| *** bobh has joined #openstack-infra | 09:38 | |
| *** heyongli has quit IRC | 09:40 | |
| *** alexchadin has quit IRC | 09:40 | |
| *** heyongli has joined #openstack-infra | 09:40 | |
| openstackgerrit | Nguyen Van Trung proposed openstack/diskimage-builder master: Add iscsi-boot element for CentOS images https://review.openstack.org/542708 | 09:40 |
| *** egonzalez has joined #openstack-infra | 09:41 | |
| *** bobh has quit IRC | 09:42 | |
| egonzalez | Hi, is there a way to create zuul secrets and read only in merge jobs instead of all? | 09:43 |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Add anchors to link to specific parts of index https://review.openstack.org/575445 | 09:45 |
| *** lifeless has joined #openstack-infra | 09:45 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Add tenant yaml validation option to scheduler https://review.openstack.org/574265 | 09:47 |
| *** bobh has joined #openstack-infra | 09:49 | |
| *** heyongli has quit IRC | 09:50 | |
| *** heyongli has joined #openstack-infra | 09:50 | |
| *** bobh has quit IRC | 09:53 | |
| *** bobh has joined #openstack-infra | 09:58 | |
| *** heyongli has quit IRC | 10:00 | |
| *** heyongli has joined #openstack-infra | 10:01 | |
| *** bobh has quit IRC | 10:02 | |
| *** rcernin has joined #openstack-infra | 10:06 | |
| *** dtantsur|afk is now known as dtantsur | 10:06 | |
| *** bobh has joined #openstack-infra | 10:08 | |
| *** heyongli has quit IRC | 10:11 | |
| *** heyongli has joined #openstack-infra | 10:11 | |
| *** bobh has quit IRC | 10:12 | |
| *** stakeda has quit IRC | 10:17 | |
| *** markvoelker has joined #openstack-infra | 10:18 | |
| *** bobh has joined #openstack-infra | 10:21 | |
| *** heyongli has quit IRC | 10:21 | |
| *** heyongli has joined #openstack-infra | 10:21 | |
| *** lifeless_ has joined #openstack-infra | 10:22 | |
| *** lifeless has quit IRC | 10:23 | |
| *** bobh has quit IRC | 10:25 | |
| *** alexchadin has joined #openstack-infra | 10:25 | |
| *** heyongli has quit IRC | 10:31 | |
| *** roman_g has joined #openstack-infra | 10:31 | |
| *** heyongli has joined #openstack-infra | 10:31 | |
| *** boden has joined #openstack-infra | 10:33 | |
| *** lifeless_ has quit IRC | 10:39 | |
| *** lifeless has joined #openstack-infra | 10:41 | |
| *** heyongli has quit IRC | 10:41 | |
| *** heyongli has joined #openstack-infra | 10:42 | |
| *** dtantsur is now known as dtantsur|afk | 10:45 | |
| *** bobh has joined #openstack-infra | 10:47 | |
| *** pbourke has joined #openstack-infra | 10:50 | |
| *** zoli|lunch is now known as zoli | 10:50 | |
| *** gfidente has quit IRC | 10:50 | |
| *** markvoelker has quit IRC | 10:51 | |
| *** jesslampe has joined #openstack-infra | 10:51 | |
| *** heyongli has quit IRC | 10:52 | |
| *** heyongli has joined #openstack-infra | 10:52 | |
| *** bobh has quit IRC | 10:52 | |
| *** lifeless_ has joined #openstack-infra | 10:55 | |
| *** bobh has joined #openstack-infra | 10:56 | |
| *** lifeless has quit IRC | 10:56 | |
| *** jesslampe has quit IRC | 10:56 | |
| *** lifeless has joined #openstack-infra | 11:00 | |
| *** bobh has quit IRC | 11:00 | |
| *** lifeless_ has quit IRC | 11:01 | |
| *** heyongli has quit IRC | 11:02 | |
| *** heyongli has joined #openstack-infra | 11:02 | |
| *** yolanda has quit IRC | 11:03 | |
| *** lifeless has quit IRC | 11:05 | |
| *** yolanda has joined #openstack-infra | 11:06 | |
| *** gfidente has joined #openstack-infra | 11:08 | |
| *** roman_g has quit IRC | 11:09 | |
| *** germs has joined #openstack-infra | 11:10 | |
| *** germs has quit IRC | 11:10 | |
| *** germs has joined #openstack-infra | 11:10 | |
| *** roman_g has joined #openstack-infra | 11:10 | |
| *** lifeless has joined #openstack-infra | 11:10 | |
| *** jpena is now known as jpena|lunch | 11:12 | |
| *** heyongli has quit IRC | 11:12 | |
| *** heyongli has joined #openstack-infra | 11:13 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Fix paused handler exception handling https://review.openstack.org/575515 | 11:13 |
| *** yamamoto has quit IRC | 11:13 | |
| *** germs has quit IRC | 11:14 | |
| *** lifeless_ has joined #openstack-infra | 11:15 | |
| *** lifeless has quit IRC | 11:15 | |
| *** annp has quit IRC | 11:16 | |
| *** bobh has joined #openstack-infra | 11:17 | |
| *** bobh has quit IRC | 11:22 | |
| *** heyongli has quit IRC | 11:22 | |
| *** heyongli has joined #openstack-infra | 11:23 | |
| *** bobh has joined #openstack-infra | 11:28 | |
| *** xarses has quit IRC | 11:29 | |
| *** xarses has joined #openstack-infra | 11:29 | |
| *** ldnunes has joined #openstack-infra | 11:32 | |
| *** heyongli has quit IRC | 11:33 | |
| *** bobh has quit IRC | 11:33 | |
| *** heyongli has joined #openstack-infra | 11:33 | |
| *** vivsoni has quit IRC | 11:34 | |
| mnaser | hi infra-root, yesterday, frickler mentioned that he was going to put a hold on vms that fail for http://zuul.openstack.org/builds.html?job_name=openstack-ansible-deploy-ceph-ubuntu-xenial .. i see that one failed not long ago so i'd liek to dive in | 11:35 |
| *** alexchadin has quit IRC | 11:35 | |
| *** jpena|lunch is now known as jpena|off | 11:38 | |
| *** rh-jelabarre has joined #openstack-infra | 11:41 | |
| *** heyongli has quit IRC | 11:43 | |
| *** rh-jelabarre has quit IRC | 11:43 | |
| *** heyongli has joined #openstack-infra | 11:44 | |
| *** rh-jelabarre has joined #openstack-infra | 11:44 | |
| *** rh-jelabarre has quit IRC | 11:46 | |
| *** rh-jelabarre has joined #openstack-infra | 11:46 | |
| *** rh-jelabarre has quit IRC | 11:47 | |
| *** markvoelker has joined #openstack-infra | 11:48 | |
| *** rh-jelabarre has joined #openstack-infra | 11:48 | |
| *** rh-jelabarre has quit IRC | 11:48 | |
| *** yamamoto has joined #openstack-infra | 11:49 | |
| ianw | mnaser: do you know the ip of the one that failed? | 11:50 |
| *** bobh has joined #openstack-infra | 11:51 | |
| *** alexchadin has joined #openstack-infra | 11:51 | |
| ianw | mnaser: i can't see any on hold for the latest failure which seems to be http://logs.openstack.org/76/571776/1/gate/openstack-ansible-deploy-ceph-ubuntu-xenial/ba76d9e/zuul-info/host-info.ubuntu-xenial.yaml | 11:52 |
| *** heyongli has quit IRC | 11:53 | |
| *** heyongli has joined #openstack-infra | 11:54 | |
| *** bobh has quit IRC | 11:56 | |
| ianw | mnaser: i couldn't see any current holds for it, but i put one in for you | 11:56 |
| ianw | | openstack | git.openstack.org/openstack/openstack-ansible | openstack-ansible-deploy-ceph-ubuntu-xenial | .* | 1 | mnaser request | | 11:56 |
| mnaser | ianw: ah, thank you. Will it hold the next failure? | 11:56 |
| *** bobh has joined #openstack-infra | 11:59 | |
| *** edmondsw has joined #openstack-infra | 12:01 | |
| *** markvoelker has quit IRC | 12:03 | |
| *** markvoelker has joined #openstack-infra | 12:03 | |
| *** heyongli has quit IRC | 12:03 | |
| *** bobh has quit IRC | 12:04 | |
| *** heyongli has joined #openstack-infra | 12:04 | |
| ianw | mnaser: it should do :) there have been a number of zuul restarts etc in the last little bit, so possibly the last one went missing | 12:04 |
| *** jpena|off is now known as jpena | 12:05 | |
| ianw | it won't be long till the A team is here who i'm sure can help :) | 12:05 |
| *** e0ne has quit IRC | 12:08 | |
| *** e0ne has joined #openstack-infra | 12:08 | |
| *** trown|outtypewww is now known as trown | 12:09 | |
| *** bobh has joined #openstack-infra | 12:09 | |
| *** lyarwood is now known as lyaaaaaarwood | 12:10 | |
| *** heyongli has quit IRC | 12:14 | |
| *** bobh has quit IRC | 12:14 | |
| *** heyongli has joined #openstack-infra | 12:14 | |
| *** rlandy has joined #openstack-infra | 12:19 | |
| *** bobh has joined #openstack-infra | 12:19 | |
| *** rh-jelabarre has joined #openstack-infra | 12:20 | |
| mnaser | ianw: ah okay cool, thanks :) | 12:23 |
| *** bobh has quit IRC | 12:24 | |
| *** heyongli has quit IRC | 12:24 | |
| *** heyongli has joined #openstack-infra | 12:24 | |
| *** e0ne has quit IRC | 12:26 | |
| mordred | ianw: https://review.openstack.org/#/c/575580 looks great - thanks! Shrews ... if you get a sec, wanna look at it? | 12:30 |
| *** bobh has joined #openstack-infra | 12:31 | |
| *** stephenfin is now known as finucannot | 12:32 | |
| *** kgiusti has joined #openstack-infra | 12:34 | |
| *** heyongli has quit IRC | 12:34 | |
| *** heyongli has joined #openstack-infra | 12:34 | |
| *** ykarel has quit IRC | 12:39 | |
| *** ykarel has joined #openstack-infra | 12:39 | |
| *** efried is now known as fried_rice | 12:40 | |
| *** bobh has quit IRC | 12:41 | |
| *** heyongli has quit IRC | 12:44 | |
| *** heyongli has joined #openstack-infra | 12:45 | |
| *** rlandy_ has joined #openstack-infra | 12:46 | |
| *** bobh has joined #openstack-infra | 12:46 | |
| *** hemna_ has joined #openstack-infra | 12:46 | |
| *** lifeless_ has quit IRC | 12:46 | |
| *** dmellado_ has joined #openstack-infra | 12:46 | |
| *** yamamoto has quit IRC | 12:46 | |
| *** gouthamr has quit IRC | 12:46 | |
| *** stevebaker has quit IRC | 12:47 | |
| fungi | flaper87: looks like the old nodes aren't deleted yet, i'll do that now and set the autohold up again | 12:48 |
| flaper87 | fungi: thanks | 12:48 |
| *** rlandy_ has quit IRC | 12:48 | |
| *** rlandy has quit IRC | 12:48 | |
| *** rlandy_ has joined #openstack-infra | 12:48 | |
| *** bobh has quit IRC | 12:48 | |
| flaper87 | fungi: I have re-run the recheck already. Should I restart it? | 12:48 |
| *** dmellado has quit IRC | 12:49 | |
| *** rlandy_ is now known as rlandy | 12:49 | |
| fungi | i think it should be fine | 12:49 |
| *** myoung|off is now known as myoung | 12:49 | |
| fungi | autohold is in place now | 12:49 |
| *** Nil_ has quit IRC | 12:49 | |
| flaper87 | perfect, thanks | 12:49 |
| Shrews | mordred: i'm wondering if that new try-except should be just around the connect call | 12:51 |
| Shrews | ianw: ^^ | 12:51 |
| *** bobh has joined #openstack-infra | 12:53 | |
| *** egonzalez has quit IRC | 12:54 | |
| *** heyongli has quit IRC | 12:55 | |
| Shrews | maybe that doesn't matter | 12:55 |
| *** heyongli has joined #openstack-infra | 12:55 | |
| Shrews | meh | 12:56 |
| *** bobh has quit IRC | 12:57 | |
| *** bobh has joined #openstack-infra | 12:57 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-cgit master: Update Gemfile for Zuulv3 https://review.openstack.org/571987 | 12:57 |
| *** yamamoto has joined #openstack-infra | 13:00 | |
| *** Nil_ has joined #openstack-infra | 13:02 | |
| *** rcernin has quit IRC | 13:02 | |
| *** yamamoto has quit IRC | 13:04 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Fix paused handler exception handling https://review.openstack.org/575515 | 13:05 |
| *** heyongli has quit IRC | 13:05 | |
| *** heyongli has joined #openstack-infra | 13:05 | |
| *** germs has joined #openstack-infra | 13:11 | |
| *** germs has quit IRC | 13:11 | |
| *** germs has joined #openstack-infra | 13:11 | |
| *** dave-mccowan has joined #openstack-infra | 13:11 | |
| mnaser | hi infra-root, 575715 failed which is openstack-ansible-deploy-ceph-ubuntu-xenial | 13:12 |
| mnaser | i should have a vm ready for me, i think? :) | 13:12 |
| Shrews | mnaser: looks like it | 13:14 |
| *** iyamahat has joined #openstack-infra | 13:15 | |
| *** yamamoto has joined #openstack-infra | 13:15 | |
| *** heyongli has quit IRC | 13:15 | |
| *** krtaylor has joined #openstack-infra | 13:15 | |
| *** heyongli has joined #openstack-infra | 13:15 | |
| *** germs has quit IRC | 13:15 | |
| mnaser | Shrews: https://github.com/mnaser.keys ? :) | 13:16 |
| *** dave-mccowan has quit IRC | 13:16 | |
| Shrews | two of them actually | 13:16 |
| mnaser | ideally the 575715 is the failure that i'm trying to handle | 13:16 |
| *** dave-mcc_ has joined #openstack-infra | 13:16 | |
| *** rh-jelabarre has quit IRC | 13:18 | |
| Shrews | mnaser: sure, just a sec | 13:18 |
| *** dhajare has quit IRC | 13:19 | |
| Shrews | mordred: can you help mnaser? my ssh key appears to be missing | 13:19 |
| *** jesslampe has joined #openstack-infra | 13:20 | |
| *** yamamoto has quit IRC | 13:20 | |
| *** r-daneel has quit IRC | 13:20 | |
| mordred | mnaser: ssh root@146.20.110.218 should work for you | 13:20 |
| *** sthussey has joined #openstack-infra | 13:20 | |
| mnaser | mordred: awesome, this will be *super* useful | 13:20 |
| mnaser | i'll inform infra when im done | 13:20 |
| *** amoralej is now known as amoralej|lunch | 13:22 | |
| *** eharney has joined #openstack-infra | 13:23 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/system-config master: Update ssh key for Shrews https://review.openstack.org/575731 | 13:24 |
| *** rh-jelabarre has joined #openstack-infra | 13:25 | |
| *** heyongli has quit IRC | 13:25 | |
| *** heyongli has joined #openstack-infra | 13:26 | |
| *** eernst has joined #openstack-infra | 13:26 | |
| *** links has quit IRC | 13:27 | |
| *** jesslampe has quit IRC | 13:29 | |
| Shrews | fungi: can you push 575731 ^ through for me? | 13:31 |
| *** ykarel is now known as ykarel|away | 13:31 | |
| fungi | looking | 13:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove inactive rooters from nodepool nodes https://review.openstack.org/575733 | 13:33 |
| mordred | fungi, Shrews: noticed that ^^ while looking at the Shrews key | 13:34 |
| fungi | mordred: i think i already had a change for that which has been on hold because pabelanger said it needed some careful transition to avoid breaking nodepool? | 13:34 |
| fungi | looking | 13:34 |
| fungi | https://review.openstack.org/489711 | 13:35 |
| Shrews | unclear on how that would break nodepool | 13:35 |
| mordred | fungi: oh - well I certainly don't want to break nodepool | 13:35 |
| mordred | ah - the ansile module thing | 13:35 |
| mordred | I thought we maybe fixed that? | 13:35 |
| *** heyongli has quit IRC | 13:36 | |
| fungi | if so i can rebase my change... it catches a lot of additional instances of those users in random places in the repo besides just that file | 13:36 |
| mordred | oh - actually, I think the issue is that _openstack_ doens't support editing keys | 13:36 |
| *** heyongli has joined #openstack-infra | 13:36 | |
| fungi | yeah we have to switch to a new keys object or something? | 13:36 |
| *** ociuhandu_ has quit IRC | 13:37 | |
| *** ociuhandu has joined #openstack-infra | 13:37 | |
| *** dave-mcc_ has quit IRC | 13:39 | |
| *** pblaho has quit IRC | 13:39 | |
| mordred | yeah - either a delete/re-create or we need to make a new one and then update nodepool to use it | 13:40 |
| mordred | the second is likely the better choice, come to think of it | 13:40 |
| *** ykarel|away has quit IRC | 13:40 | |
| mordred | since we can stage that as a three-patch expand/contract sequence | 13:40 |
| fungi | i can take playbooks/clouds_layouts.yml out of 489711 if you want to turn 575733 into that staged rollout | 13:41 |
| mordred | fungi: sounds good to me | 13:41 |
| fungi | cool, doing that now | 13:41 |
| dhellmann | gmann and I could really use some help from a zuulv3 expert in this ML thread: http://lists.openstack.org/pipermail/openstack-dev/2018-June/131533.html | 13:42 |
| *** bobh has quit IRC | 13:43 | |
| mordred | fungi: actually - we need to squish the Shrews update into that too, come to think of it | 13:43 |
| Shrews | ugh | 13:44 |
| mordred | sok - I'm on it | 13:44 |
| *** yamamoto has joined #openstack-infra | 13:45 | |
| *** mriedem has joined #openstack-infra | 13:45 | |
| dhellmann | corvus or clarkb : maybe you can help answer the question we're having in that thread? | 13:45 |
| *** heyongli has quit IRC | 13:46 | |
| *** heyongli has joined #openstack-infra | 13:46 | |
| *** dansmith is now known as superdan | 13:47 | |
| *** lbragstad is now known as elbragstad | 13:48 | |
| *** yamamoto has quit IRC | 13:49 | |
| *** r-daneel has joined #openstack-infra | 13:50 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add new keypair for updated infra-root members https://review.openstack.org/575731 | 13:51 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Clean up accounts for former root sysadmins https://review.openstack.org/489711 | 13:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Use infra-root-keys-2018-06-15 in nodepool https://review.openstack.org/575744 | 13:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove old infra-root-keys keypair https://review.openstack.org/575745 | 13:56 |
| mordred | fungi: ^^ I think that should do it | 13:56 |
| Shrews | +2 from me | 13:56 |
| *** heyongli has quit IRC | 13:56 | |
| *** heyongli has joined #openstack-infra | 13:56 | |
| *** yamamoto has joined #openstack-infra | 14:00 | |
| *** eernst has quit IRC | 14:01 | |
| dhellmann | do we have any examples of creating virtualenvs within playbooks/roles? | 14:03 |
| dhellmann | I'm having trouble creating one under python 3 and then getting anything installed, because ansible can't find the pip command inside the venv | 14:03 |
| dhellmann | http://logs.openstack.org/16/574916/5/check/test-release-openstack-python3/1368b14/ara-report/result/ff8239a5-5091-4824-a42d-c84d6029200a/ | 14:03 |
| dhellmann | the job definition is in https://review.openstack.org/574842 | 14:04 |
| *** yamamoto has quit IRC | 14:04 | |
| *** caphrim007 has quit IRC | 14:04 | |
| *** heyongli has quit IRC | 14:06 | |
| *** r-daneel_ has joined #openstack-infra | 14:07 | |
| *** caphrim007_ has joined #openstack-infra | 14:07 | |
| *** heyongli has joined #openstack-infra | 14:07 | |
| *** yamahata has quit IRC | 14:07 | |
| *** r-daneel has quit IRC | 14:08 | |
| *** r-daneel_ is now known as r-daneel | 14:08 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Fix paused handler exception handling https://review.openstack.org/575515 | 14:11 |
| *** caphrim007_ has quit IRC | 14:11 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: package-afs-centos: use synchronize flags for delete https://review.openstack.org/575223 | 14:12 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: package-afs-centos: fix package name https://review.openstack.org/575665 | 14:12 |
| *** caphrim007 has joined #openstack-infra | 14:13 | |
| *** yamamoto has joined #openstack-infra | 14:15 | |
| *** heyongli has quit IRC | 14:17 | |
| *** dave-mccowan has joined #openstack-infra | 14:17 | |
| *** heyongli has joined #openstack-infra | 14:17 | |
| *** caphrim007 has quit IRC | 14:17 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: convert check-python-release to a script so we can see what it does https://review.openstack.org/575750 | 14:19 |
| *** yamamoto has quit IRC | 14:20 | |
| *** eharney has quit IRC | 14:20 | |
| *** florianf has quit IRC | 14:21 | |
| *** florianf has joined #openstack-infra | 14:22 | |
| fungi | dhellmann: i'm not aware of any examples off the top of my head, but i'd probably use `virtualenv -p python3 foo` or `python3 -m venv foo` and then `foo/bin/pip install whatever` and `foo/bin/whatever blah` | 14:22 |
| *** amoralej|lunch is now known as amoralej | 14:23 | |
| *** iyamahat has quit IRC | 14:23 | |
| dhellmann | ok, there's an ansible module for pip, and that seems to take args to use virtualenv, but then the results don't work | 14:23 |
| dhellmann | this may be a python3 issue with ansible | 14:23 |
| *** germs has joined #openstack-infra | 14:23 | |
| *** germs has quit IRC | 14:23 | |
| *** germs has joined #openstack-infra | 14:23 | |
| fungi | oh, you know, we might have a role for virtualenv stuff | 14:23 |
| *** bdodd_ is now known as bdodd | 14:24 | |
| dhellmann | I got fed up with the lack of debug output anyway, so I am working on converting it to a shell script | 14:25 |
| mordred | dhellmann: fwiw, the ensure-sphinx role in zuul-jobs creates a virtualenv and installs things in to it with pip | 14:26 |
| dhellmann | mordred : does it use python3? | 14:26 |
| mordred | dhellmann: it supports doing so, yes (it's a flag) | 14:26 |
| dhellmann | ok, I'll take a look at that | 14:26 |
| fungi | yeah, the bindep and fetch-zuul-cloner roles do too, but i don't see examples of using python3 | 14:26 |
| fungi | they seem to use the ansible "pip" module as best i can tell | 14:27 |
| mordred | yah - as does ensure-sphinx | 14:27 |
| *** heyongli has quit IRC | 14:27 | |
| *** heyongli has joined #openstack-infra | 14:27 | |
| dhellmann | ah, you know, I think the chdir bit was in the wrong place in mine | 14:27 |
| mordred | I'm starting to think though that we should maybe define a standard variable name that can be set that ensure-sphinx and bindep and whatever else will honor | 14:27 |
| dhellmann | wrong indent level | 14:27 |
| fungi | so i guess if we plumb through whatever parameter the pip module docs say sets the python interpreter version/path that should work? | 14:27 |
| mordred | so if someone wants to do python3 they can set _one_ job variable and the relevant roles will all do the right thing | 14:28 |
| fungi | that would be swell | 14:28 |
| clarkb | mordred: I almost want to invert it | 14:28 |
| dhellmann | that would be nice | 14:28 |
| mordred | clarkb: what do you mean by invert it? (I probably agree) | 14:28 |
| clarkb | mordred: python3 everything unless you set the common var to python2 | 14:29 |
| mordred | clarkb: oh - yeah- I'd be good with that | 14:29 |
| mordred | maybe we start with rolling out a common variable name similar to zuul_work_dir - maybe zuul_python_version? | 14:29 |
| clarkb | it will shake out places we arent python3 ready and jump us ahead otherwise | 14:29 |
| mordred | and then when we're comfortable we can swap the default value | 14:30 |
| *** yamamoto has joined #openstack-infra | 14:30 | |
| mordred | or zuul_python I guess - since the value it takes is "python2" or "python3" | 14:30 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove glance legacy job https://review.openstack.org/551019 | 14:32 |
| *** yamamoto has quit IRC | 14:34 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add missing __init__.py files https://review.openstack.org/575591 | 14:35 |
| *** rpittau has quit IRC | 14:36 | |
| *** heyongli has quit IRC | 14:37 | |
| *** heyongli has joined #openstack-infra | 14:38 | |
| corvus | dhellmann: if i understand the thread you were asking about earlier -- the primary question is "does tempest-full-py3 add the project for the change under test (zuul.project) to LIBS_FROM_GIT?". you suggest that it does not do that, and i believe you are correct. an implied second question is "should it?" i believe the old jobs did not automatically add anything to LIBS_FROM_GIT, including the | 14:40 |
| corvus | project of the change under test (ZUUL_PROJECT). (cc: clarkb) | 14:40 |
| *** hongbin has joined #openstack-infra | 14:41 | |
| dhellmann | corvus : yes, you have understood my main question correctly. I think we need the second job, tempest-full-py3-src to add the library-under-test to the job | 14:41 |
| *** ramishra has quit IRC | 14:41 | |
| *** rpioso|afk is now known as rpioso | 14:42 | |
| corvus | dhellmann, clarkb: did the libraries always have two jobs, and one of them was a no-op? | 14:44 |
| dhellmann | the libraries only use the lib-forward-testing template, which only has the -src job, afaict | 14:45 |
| *** yamamoto has joined #openstack-infra | 14:45 | |
| dhellmann | they did not use the integrated-gate template with the "standard" devstack gate job | 14:45 |
| clarkb | corvus: ya I think they only had the not noop -src jobs | 14:45 |
| corvus | ok that makes sense | 14:45 |
| dhellmann | I guess there's some source of confusion about whatever causes zuul to add the required-projects list of things to LIBS_FROM_GIT | 14:46 |
| dhellmann | gmann seems to think that applies to the repo-under-test as well as the explicit extra required-projects | 14:46 |
| corvus | https://docs.openstack.org/devstack/latest/zuul_roles.html#rolevar-write-devstack-local-conf.devstack_localrc | 14:47 |
| *** heyongli has quit IRC | 14:47 | |
| *** heyongli has joined #openstack-infra | 14:48 | |
| dhellmann | in https://review.openstack.org/#/c/575164/2/.zuul.d/project.yaml I am setting LIBS_FROM_GIT based on the project-under-test to avoid having to list itself in required-projects and make it so we can have a project-template | 14:48 |
| corvus | dhellmann: a consequence of the implementation in 575164 is that will disable the automatic setting of LIBS_FROM_GIT. that's probably fine for the cases we're immediately considering, but might this job be used by other projects which need more than one item in LIBS_FROM_GIT? | 14:48 |
| dhellmann | that seems to work (I see the lib being installed from source) | 14:48 |
| dhellmann | I'm not sure what circumstances might lead to needing that | 14:49 |
| *** yamamoto has quit IRC | 14:49 | |
| dhellmann | do you think it's better to just use tempeset-full-py3 with required-projects explicitly set to a list including "self" instead of trying to enable a project-template? | 14:50 |
| *** myoung is now known as myoung|bbl | 14:54 | |
| *** mriedem has quit IRC | 14:54 | |
| *** mriedem has joined #openstack-infra | 14:54 | |
| clarkb | That may help simplify the job cinfig but be surprising to users? | 14:55 |
| clarkb | unfortunately I think the split between libs and apps and how they are tested already confuses people | 14:56 |
| dhellmann | I think it makes the job config more likely to be wrong | 14:57 |
| *** heyongli has quit IRC | 14:58 | |
| *** heyongli has joined #openstack-infra | 14:58 | |
| dhellmann | if the base job was "smart" enough to include the current project automatically but still allow required-projects that would simplify lots of cross-repo test scenarios | 14:58 |
| dhellmann | I'm just not sure how to do that | 14:58 |
| corvus | dhellmann: i was thinking about situations like this, where multiple libraries need to be in libs-from-git (i'm going to call it LFG from here): https://review.openstack.org/548331 | 14:58 |
| corvus | dhellmann: i'm also trying to figure out if that change accomplishes about the same goal as what you're working on? | 14:59 |
| corvus | dhellmann: (but you're doing it with py3) | 14:59 |
| corvus | basically, is the "forward testing" template the thing that defines "actually test this lib" ? | 14:59 |
| *** eernst has joined #openstack-infra | 14:59 | |
| *** yamamoto has joined #openstack-infra | 15:00 | |
| dhellmann | yes, that's what the current template does and I'm trying to make a python3 version of the same | 15:00 |
| dhellmann | those messaging jobs have some other customizations to them, so adding the required-projects list isn't a big burden | 15:01 |
| dhellmann | what I want is something simple I can just add to all of the libraries to avoid having to create separate custom jobs | 15:01 |
| corvus | dhellmann: so i think if we went with your template as-is, we would have to manually set LFG in the messaging jobs | 15:01 |
| *** pcaruana has quit IRC | 15:01 | |
| *** iyamahat has joined #openstack-infra | 15:02 | |
| *** bobh has joined #openstack-infra | 15:03 | |
| dhellmann | where is the code that takes required-projects and sets LIBS_FROM_GIT? | 15:03 |
| corvus | exploring the idea of automatically setting lfg to include the project-under-test -- that could be accomplished fairly easily with a modification to the devstack ansible module which generates it. we could add an extra job variable to turn that behavior on and off. but i wonder, is that something that we could enable everywhere? | 15:03 |
| *** r-daneel has quit IRC | 15:03 | |
| *** alexchadin has quit IRC | 15:04 | |
| *** dpawlik has quit IRC | 15:04 | |
| *** r-daneel has joined #openstack-infra | 15:04 | |
| dhellmann | it seems like that would be the least surprising thing to do. It would make it so that adding the job to a project automatically tested that project. | 15:04 |
| clarkb | ++ | 15:04 |
| corvus | dhellmann: http://git.openstack.org/cgit/openstack-dev/devstack/tree/roles/write-devstack-local-conf/library/devstack_local_conf.py#n243 | 15:04 |
| *** yamamoto has quit IRC | 15:04 | |
| dhellmann | I think LIBS_FROM_GIT ignores anything that isn't a library, so it should be safe to just always add the current project to it | 15:05 |
| corvus | the first part is "don't do the automatic thing if the user manually set LFG". the second part adds all required projects to LFG | 15:05 |
| corvus | dhellmann: it does now; i had to change that a bit | 15:05 |
| *** zhangfei has quit IRC | 15:05 | |
| dhellmann | oh, that's surprising, but I guess good that it works that way now | 15:05 |
| * dhellmann has to join the release team meeting, already in progress | 15:06 | |
| *** e0ne has joined #openstack-infra | 15:06 | |
| panda | hi, we are planning the transition to zullv3, and missing some information. For example, is we switch from legacy dsvm parent to devstack parent, do we need to use also the new nodesets, or we can use old images taht still provide /etc/nodepool/subnodes ? | 15:06 |
| corvus | dhellmann, clarkb: i'll write a ml response with the summary so far | 15:06 |
| dhellmann | corvus : thanks | 15:06 |
| *** dpawlik has joined #openstack-infra | 15:07 | |
| clarkb | panda: you would switch to the non legacy nodesets and rely on ansible inventory info for mulyinode rather than the primary vs subnode delegation system | 15:07 |
| clarkb | panda: you also have the opportunity to define you own nodesets with groupings that make sense for your job(s) | 15:08 |
| *** heyongli has quit IRC | 15:08 | |
| *** heyongli has joined #openstack-infra | 15:08 | |
| panda | clarkb: ok, but if want to make it moving with baby steps, will it be possible to use the new parent with the legacy images ? we have soem tooling that still relies on the subnodes interface and we have to unsderstand if we need to do everything in one passage or not | 15:09 |
| panda | pass* | 15:09 |
| fungi | there are no "legacy images" | 15:10 |
| fungi | all jobs draw from the same sets of images | 15:11 |
| clarkb | ya just legacy organization of nodesets | 15:11 |
| clarkb | I'm not sure where we right the old nodepool data for compatibility but we do do so for legacy jobs and as long as you have that you can continue to use the nodes that eay | 15:11 |
| fungi | mostly has to do with what the nodes are named (which is usually only a relevant distinction for multinode jobs, i think?) | 15:11 |
| clarkb | *write | 15:12 |
| *** dpawlik has quit IRC | 15:12 | |
| fungi | oh, and what files get written into /etc/nodepool on the nodes, right | 15:12 |
| panda | ok, so what really creates /etc/nodepool/subnodes ? | 15:12 |
| fungi | i think there's just an ansible role which is generating those early in the legacy jobs right now? | 15:12 |
| clarkb | fungi panda yes I think so, just unsure of location | 15:13 |
| panda | so maybe we can try to include this in our jobs while still using the new parents | 15:13 |
| *** eharney has joined #openstack-infra | 15:14 | |
| *** kgiusti has left #openstack-infra | 15:14 | |
| mordred | playbooks/legacy/pre.yaml in openstack-zuul-jobs is what writes out the old /etc/nodepool files | 15:15 |
| mordred | so as long as the job uses that, either directly or via the legacy-base - the old /etc/nodepool stuff will be written | 15:16 |
| panda | mordred: ah, thanks, and the information in /etc/nodepool will still be valid at this point ? | 15:16 |
| mordred | it should be :) | 15:17 |
| panda | mordred: ok fungi clarkb thanks. :) will try | 15:18 |
| *** heyongli has quit IRC | 15:18 | |
| Shrews | new jobs using legacy playbooks? i thought we were recommending against that | 15:18 |
| *** heyongli has joined #openstack-infra | 15:18 | |
| *** gouthamr has joined #openstack-infra | 15:19 | |
| mordred | we are - I thnk this is as a stepping stone towards new jobs without having to do a full all-at-once transition | 15:20 |
| panda | yes | 15:20 |
| *** kgiusti has joined #openstack-infra | 15:20 | |
| panda | otherwise we'll need to refactor all the parts taht use subnodes in the same step | 15:20 |
| openstackgerrit | Merged openstack-infra/system-config master: Add new keypair for updated infra-root members https://review.openstack.org/575731 | 15:22 |
| *** iyamahat has quit IRC | 15:22 | |
| *** heyongli has quit IRC | 15:28 | |
| *** heyongli has joined #openstack-infra | 15:29 | |
| *** eharney has quit IRC | 15:29 | |
| *** yamamoto has joined #openstack-infra | 15:30 | |
| *** lpetrut has quit IRC | 15:30 | |
| *** yamamoto has quit IRC | 15:34 | |
| *** gouthamr has quit IRC | 15:38 | |
| *** kiennt26 has joined #openstack-infra | 15:38 | |
| *** gouthamr has joined #openstack-infra | 15:38 | |
| *** heyongli has quit IRC | 15:39 | |
| *** bobh_ has joined #openstack-infra | 15:39 | |
| *** heyongli has joined #openstack-infra | 15:39 | |
| clarkb | corvus: fungi mordred mwhahaha weshay|ruck following up with the proxy cache logging stuff from yesterday. The only stuff I see in the port 8081 is 307 responses from dockerhub. We don't cache these which is expected. I think this may imply that dockerhub is no longer redirecting to the location that we are caching (otherwise we would see requests for that location in the server log as well) | 15:40 |
| clarkb | my attempt to see where the redirect is to is foiled by the fact that apparently you have to authenticate to dockerhub to get a 307? and everything is tls so a tcpdump won't help other than to possibly identfy the new remote ip address if tcpdumping the client side | 15:41 |
| fungi | an interesting observation | 15:41 |
| fungi | can you wget and turn on verbose logging? | 15:42 |
| clarkb | fungi: that is what I tried and docker says go away | 15:42 |
| fungi | huh | 15:42 |
| clarkb | (except with curl) | 15:42 |
| clarkb | {"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"tripleomaster/centos-binary-rabbitmq","Action":"pull"}]}]} | 15:42 |
| fungi | you have to authenticate downloads? | 15:42 |
| clarkb | against https://registry-1.docker.io/v2/tripleomaster/centos-binary-rabbitmq/blobs/sha256:68812f14498bef94084841e41908c3ae82333dfa00834162d2f8e0ee813a6d0b | 15:42 |
| clarkb | fungi: I don't recall needing to do this as a docker user so it must be some client auth not user auth? | 15:43 |
| *** bobh has quit IRC | 15:43 | |
| clarkb | dockers inability to play nice with http caching is more than a little frustrating | 15:43 |
| clarkb | we also have to tell apache its ok to cache their sha256 addressed blobs because they tell apache not to in headers iirc | 15:43 |
| pabelanger | keep in mind, there are 2 types of docker caching now, depending on the version of docker so may need 8002 | 15:44 |
| fungi | huh, yeah that url tells my browser it's not authenticated too | 15:44 |
| pabelanger | https://review.openstack.org/542370/ | 15:44 |
| *** annp has joined #openstack-infra | 15:44 | |
| clarkb | pabelanger: ya tripleo is all on the old one, but both expect a common redirect dest | 15:45 |
| clarkb | and that redirect dest appears to be wrong | 15:45 |
| clarkb | pabelanger: how did you debug the redirection before? | 15:45 |
| *** yamamoto has joined #openstack-infra | 15:45 | |
| *** dpawlik has joined #openstack-infra | 15:45 | |
| pabelanger | clarkb: just ran tcpdump on the mirror and looked at headers | 15:45 |
| pabelanger | then testing locally with different docker clients | 15:45 |
| clarkb | pabelanger: oh right the front tend is not https only the backend | 15:45 |
| pabelanger | so maybe https://dseasb33srnrn.cloudfront.net is no longer the redirect | 15:46 |
| clarkb | pabelanger: that is my hunch, running tcpdump now and waiting for traffic to go by | 15:47 |
| pabelanger | k, I might have some time this afternoon to help if needed | 15:47 |
| *** armaan has joined #openstack-infra | 15:48 | |
| *** heyongli has quit IRC | 15:49 | |
| *** heyongli has joined #openstack-infra | 15:49 | |
| *** dpawlik has quit IRC | 15:50 | |
| *** yamamoto has quit IRC | 15:50 | |
| clarkb | of course now that I have a tcpdump going there is no traffic :) | 15:51 |
| *** iyamahat has joined #openstack-infra | 15:51 | |
| *** gyee has joined #openstack-infra | 15:52 | |
| clarkb | I'm going to be afk in about an hour to go berry picking with the kids. Farm's first day open to public is today and hoping that if I go early I'll beat crowds and get my pick pf raspberries :) | 15:52 |
| fungi | or your filter is mistyped ;) | 15:52 |
| clarkb | fungi: possibly | 15:53 |
| clarkb | `tcpdump -i any -A port 8081` is my command | 15:53 |
| *** iyamahat has quit IRC | 15:53 | |
| fungi | i'd add -l and -n | 15:54 |
| fungi | and tcp | 15:54 |
| clarkb | udp is blocked so should be fine to not specify | 15:55 |
| clarkb | I've added -l and -n | 15:55 |
| fungi | sudo tcpdump -Alni eth0 tcp port 8081 | 15:55 |
| fungi | something along those lines | 15:55 |
| fungi | without -l you get chunked buffering | 15:55 |
| fungi | -l is useful when you're looking for more instant gratification ;) | 15:56 |
| mnaser | oooooo | 15:56 |
| clarkb | ya I tend to be impatient :) | 15:56 |
| mnaser | TIL thanks fungi | 15:56 |
| *** eernst has quit IRC | 15:57 | |
| mnaser | i like to tack on e because it adds gre/vlan tags and stuff | 15:57 |
| clarkb | oh you know what I wonder if the data is geo local because cloudfront | 15:57 |
| mnaser | which is quite useful when you're troubleshooting with that | 15:57 |
| clarkb | thta could explain it and would make this far more complicated | 15:57 |
| clarkb | if we cofirm this is the problem in limestone we should check all of the other mirrors too | 15:57 |
| clarkb | some may actually work | 15:57 |
| fungi | mnaser: yeah, if i care about layer 2 i add -e | 15:57 |
| clarkb | ya if I a mdebugging openstack -e tend sto be valuable | 15:58 |
| fungi | and one or more -v | 15:58 |
| fungi | also on openbsd the pflog virtual interface uses the layer 2 fields to encode ruleset info | 15:58 |
| fungi | i love that they implemented their firewall logging to use pcap | 15:59 |
| *** heyongli has quit IRC | 15:59 | |
| fungi | you can tcpdump on the pflog virtual interface to see live action, or record to file and parse/filter it with tcpdump -r later | 15:59 |
| *** heyongli has joined #openstack-infra | 16:00 | |
| clarkb | mnaser: http://logs.openstack.org/74/74/f854d00290721e199a92c82d843e50e0e4e6481b/third-party-check/kata-runsh/aa6fb9b/job-output.txt.gz#_2018-06-15_04_22_21_564643 is an unfortunate failure, Looks like they merged uncompileable code. Any idea if that is still an issue? | 16:00 |
| *** yamamoto has joined #openstack-infra | 16:00 | |
| clarkb | its in external code but is vendored so seems like that should be fixable kata end | 16:00 |
| *** caphrim007 has joined #openstack-infra | 16:02 | |
| *** kiennt26 has quit IRC | 16:02 | |
| *** dmellado_ has quit IRC | 16:02 | |
| *** yamamoto has quit IRC | 16:04 | |
| *** ykarel|away has joined #openstack-infra | 16:05 | |
| *** caphrim007 has quit IRC | 16:06 | |
| *** caphrim007 has joined #openstack-infra | 16:06 | |
| *** caphrim007 has quit IRC | 16:06 | |
| *** dave-mccowan has quit IRC | 16:06 | |
| *** caphrim007 has joined #openstack-infra | 16:06 | |
| *** heyongli has quit IRC | 16:09 | |
| *** heyongli has joined #openstack-infra | 16:10 | |
| EmilienM | can someone confirm we can't use regex on http://zuul.openstack.org/builds.html right? | 16:13 |
| EmilienM | e.g. for jobs | 16:13 |
| clarkb | EmilienM: I don't recall support for that in the api | 16:14 |
| EmilienM | kk no prob | 16:14 |
| *** yamamoto has joined #openstack-infra | 16:15 | |
| fungi | sounds like a reasonable feature request though | 16:16 |
| clarkb | I'm beginning to think I may need to induce my own docker pull | 16:20 |
| *** heyongli has quit IRC | 16:20 | |
| *** yamamoto has quit IRC | 16:20 | |
| *** heyongli has joined #openstack-infra | 16:20 | |
| *** r-mibu has joined #openstack-infra | 16:20 | |
| *** Bhujay has quit IRC | 16:20 | |
| *** armaan has quit IRC | 16:23 | |
| clarkb | I'm not going to be able to induce a pull before berry picking. I'll restart tcpdump in screen and hopefully the scrollback buffer is big enough to catch headers | 16:25 |
| clarkb | (should be if only 307s are going by) | 16:25 |
| clarkb | nevermind got one, Location: https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/09/099d2cd5ed5371429d83438a8b3f8cea33111621e798b200f49db6be1a8ae718/data?verify=1529082803-vqu%2FIAS4te3u1hFEGTVoZg%2B2V8g%3D | 16:26 |
| *** jpena is now known as jpena|off | 16:27 | |
| clarkb | I think that confirms my theory. | 16:27 |
| pabelanger | haven't seen that redirect before | 16:28 |
| clarkb | pabelanger: must be a new change on their side | 16:28 |
| *** jpena|off is now known as jpena | 16:29 | |
| clarkb | I actually like it as it allows them to repoint that at whatever and we won't break | 16:29 |
| *** jpena is now known as jpena|away | 16:29 | |
| clarkb | but also they changed providers I guess | 16:29 |
| pabelanger | so, we could create a new CacheEnable disk for that, or just append for existing cloudfront | 16:30 |
| *** panda is now known as panda|off | 16:30 | |
| *** yamamoto has joined #openstack-infra | 16:30 | |
| *** heyongli has quit IRC | 16:30 | |
| clarkb | I'll add a new entry in case the old one is still used in some cases | 16:30 |
| clarkb | patch in a few | 16:30 |
| *** heyongli has joined #openstack-infra | 16:30 | |
| *** fried_rice is now known as fried_rolls | 16:32 | |
| *** r-daneel has quit IRC | 16:33 | |
| *** r-daneel has joined #openstack-infra | 16:33 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Add new dockerhub cloudflare backend https://review.openstack.org/575787 | 16:35 |
| *** yamamoto has quit IRC | 16:35 | |
| clarkb | something like that, its untested and I need to afk nowish. But I think this is likely to be the cause | 16:35 |
| clarkb | corvus: fungi mordred mwhahaha weshay|ruck ^ fyi | 16:35 |
| mwhahaha | EmilienM: -^ | 16:36 |
| pabelanger | clarkb: looks right, +2 | 16:37 |
| *** jpich has quit IRC | 16:39 | |
| *** heyongli has quit IRC | 16:40 | |
| *** heyongli has joined #openstack-infra | 16:40 | |
| *** derekh has quit IRC | 16:41 | |
| *** ykarel|away has quit IRC | 16:43 | |
| boden | hi, does anyone have visibility into how 3rd party CI systems are handling “tox siblings” and “depends-on” patches? For example the project’s job layout contains required-projects (install projects from master), and the patch also has a depends-on from one of those (install project from master + a depends-on patch) | 16:43 |
| *** yamamoto has joined #openstack-infra | 16:45 | |
| *** hamzy_ has joined #openstack-infra | 16:48 | |
| *** yamamoto has quit IRC | 16:49 | |
| fungi | boden: tox-siblings is very much a zuul v3 role, so if third-party ci systems are running zuul 2.x or something besides zuul entirely (jenkins gerrit-trigger plugin? home-grown ci?) then they're likely not doing anything like it at all | 16:50 |
| *** gouthamr has quit IRC | 16:50 | |
| boden | fungi ack thx, that’s what I suspected | 16:50 |
| fungi | as for depends-on, if they're running zuul _and_ including the set of projects with the depends-on then they're probably respecting it | 16:50 |
| *** gouthamr has joined #openstack-infra | 16:50 | |
| *** heyongli has quit IRC | 16:50 | |
| *** heyongli has joined #openstack-infra | 16:51 | |
| fungi | but a lot of third-party ci operators limit the repositories they're including in their zuul configuration so depends-on probably doesn't work if it's reaching outside the set they're configured for | 16:51 |
| *** hamzy has quit IRC | 16:51 | |
| *** lyaaaaaarwood has quit IRC | 16:51 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/infra-specs master: WIP Update config management for the Infra Control Plane https://review.openstack.org/565550 | 16:51 |
| mordred | that is not close to being done - but I thought I'd push up some wip | 16:51 |
| fungi | boden: and again, for zuul 2.x, only if the job framework has support for using something like zuul-cloner or devstack-gate to pull in the source related to the depends-on | 16:52 |
| fungi | so not for, say, typical tox jobs | 16:52 |
| *** dougsz has quit IRC | 16:53 | |
| boden | fungi thanks… to date I’m not up to speed with the internal of the 3rd party CI, so I’ll have to dig a bit more into what we have in place | 16:53 |
| *** yamamoto has joined #openstack-infra | 16:55 | |
| *** yamamoto has quit IRC | 16:55 | |
| *** zoli is now known as zoli|gone | 16:57 | |
| *** zoli|gone is now known as zoli | 16:57 | |
| *** gfidente has quit IRC | 16:59 | |
| *** heyongli has quit IRC | 17:01 | |
| *** heyongli has joined #openstack-infra | 17:01 | |
| *** tesseract has quit IRC | 17:01 | |
| EmilienM | clarkb: thanks for your patch | 17:05 |
| *** eharney has joined #openstack-infra | 17:07 | |
| *** e0ne has quit IRC | 17:08 | |
| *** heyongli has quit IRC | 17:11 | |
| *** heyongli has joined #openstack-infra | 17:11 | |
| *** amoralej is now known as amoralej|off | 17:14 | |
| *** bobh has joined #openstack-infra | 17:17 | |
| EmilienM | I'm purging tripleo gate/check so we give space for clarkb's patch to land ASAP | 17:20 |
| EmilienM | mwhahaha: ^ FYI | 17:20 |
| *** bobh_ has quit IRC | 17:20 | |
| *** heyongli has quit IRC | 17:21 | |
| *** heyongli has joined #openstack-infra | 17:22 | |
| fungi | well, we can bypass check anyway if it's a hurry | 17:24 |
| *** germs has quit IRC | 17:24 | |
| *** germs has joined #openstack-infra | 17:24 | |
| *** germs has quit IRC | 17:24 | |
| *** germs has joined #openstack-infra | 17:24 | |
| *** ccamacho has quit IRC | 17:24 | |
| *** germs has quit IRC | 17:24 | |
| *** anteaya has joined #openstack-infra | 17:26 | |
| *** gouthamr_ has joined #openstack-infra | 17:27 | |
| *** gouthamr has quit IRC | 17:27 | |
| *** gouthamr_ is now known as gouthamr | 17:27 | |
| *** ociuhandu has quit IRC | 17:29 | |
| EmilienM | fungi: please if you can | 17:29 |
| corvus | clarkb, dhellmann: https://review.openstack.org/575801 Add the project under test to LIBS_FROM_GIT | 17:29 |
| dhellmann | corvus : oh, nice, I was going to try for that after lunch but you beat me to it | 17:30 |
| corvus | clarkb, dhellmann: i went ahead and wrote that patch in case we decide that's the next step. there's a lot more detail and hand-wavey questions in the ml reply | 17:30 |
| fungi | EmilienM: i've pushed it into the gate pipeline now. it should get node assignments in short order | 17:30 |
| EmilienM | fungi: and https://review.openstack.org/#/c/575790/ https://review.openstack.org/#/c/575799/ https://review.openstack.org/#/c/575798/ https://review.openstack.org/#/c/575796/ https://review.openstack.org/#/c/575795/ | 17:30 |
| corvus | dhellmann: oh, sorry to deprive you of the joy -- but maybe i left a bug in there! :) | 17:30 |
| EmilienM | fungi: I've purged all other tripleo jobs in the check/gate | 17:31 |
| EmilienM | thanks | 17:31 |
| dhellmann | corvus : I'm not sure I would have come up with all of the details in this solution, so I'm happy to use the review as a learning tool :-) | 17:31 |
| *** heyongli has quit IRC | 17:31 | |
| corvus | dhellmann: if, however, that does work, you can depends-on it and try it out | 17:31 |
| *** heyongli has joined #openstack-infra | 17:32 | |
| dhellmann | oh, good, I'll put together a patch to try that | 17:32 |
| *** dave-mccowan has joined #openstack-infra | 17:33 | |
| dhellmann | what is the difference between the existing self.projects and the new self.project? | 17:33 |
| fungi | EmilienM: some of those aren't approved yet. i can't enqueue them into the gate until they have workflow +1 | 17:33 |
| EmilienM | fungi: I think i've approved them all | 17:34 |
| fungi | EmilienM: 575799 was missing approval, i'll check again | 17:34 |
| *** pbourke has quit IRC | 17:34 | |
| EmilienM | oh indeed | 17:34 |
| EmilienM | sorry for that, fixed | 17:34 |
| *** janki has quit IRC | 17:34 | |
| electrofelix | anyone interested in reviewing a change for git-review to support parsing a change from the full url - https://review.openstack.org/424823 | 17:36 |
| corvus | dhellmann: self.projects == zuul.projects (the list of all projects in the dependency chain + any required-projects). self.project == zuul.project (the project of the current change under test). you can see the values in the inventory: http://logs.openstack.org/64/575164/2/check/tempest-full-py3/9aa50ad/zuul-info/inventory.yaml | 17:36 |
| dhellmann | corvus : testing with https://review.openstack.org/#/c/575802/ | 17:36 |
| dhellmann | ah, ok | 17:37 |
| corvus | dhellmann: and to tie that all up, the "required:" boolean in zuul.projects is false if the project is only there because it's in the dependency chain, and true if it's in required-projects. | 17:37 |
| corvus | that's what the auto-lfg bit checks for | 17:38 |
| dhellmann | where does the dependency chain come from? required-projects on inherited jobs? | 17:38 |
| dhellmann | or depends-on? | 17:38 |
| corvus | dhellmann: depends-on, git dependencies, and changes ahead in the gate pipeline | 17:39 |
| corvus | (they all appear exactly the same to a zuul job) | 17:39 |
| dhellmann | ah, right, the pipeline | 17:39 |
| *** heyongli has quit IRC | 17:42 | |
| *** heyongli has joined #openstack-infra | 17:42 | |
| *** diablo_rojo has joined #openstack-infra | 17:45 | |
| *** pcichy has quit IRC | 17:46 | |
| *** jpena|away is now known as jpena | 17:47 | |
| EmilienM | fungi: thank you again | 17:48 |
| *** hemna_ has quit IRC | 17:49 | |
| *** heyongli has quit IRC | 17:52 | |
| *** heyongli has joined #openstack-infra | 17:52 | |
| *** boris_42_ has joined #openstack-infra | 17:53 | |
| fungi | yw | 17:56 |
| *** e0ne has joined #openstack-infra | 18:00 | |
| *** chandankumar is now known as chkumar|pto | 18:01 | |
| *** stevebaker has joined #openstack-infra | 18:01 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add new dockerhub cloudflare backend https://review.openstack.org/575787 | 18:02 |
| *** heyongli has quit IRC | 18:02 | |
| *** heyongli has joined #openstack-infra | 18:02 | |
| *** hemna_ has joined #openstack-infra | 18:02 | |
| *** electrofelix has quit IRC | 18:05 | |
| *** rlpple has joined #openstack-infra | 18:08 | |
| *** bhavik1 has joined #openstack-infra | 18:11 | |
| *** agopi has joined #openstack-infra | 18:11 | |
| *** jesslampe has joined #openstack-infra | 18:12 | |
| EmilienM | sorry to ask again but where are the logs to follow when puppet has run on the infra nodes? | 18:12 |
| *** heyongli has quit IRC | 18:12 | |
| EmilienM | I used to have http://puppet-dashboard.openstack.org/ | 18:12 |
| EmilienM | but not working | 18:13 |
| *** heyongli has joined #openstack-infra | 18:13 | |
| fungi | EmilienM: nobody ever built a replacement after that software was abandoned | 18:13 |
| *** lseki has joined #openstack-infra | 18:13 | |
| *** jpena is now known as jpena|off | 18:13 | |
| EmilienM | no problem | 18:13 |
| EmilienM | just wondering where I can follow if puppet was run | 18:13 |
| fungi | we emit ansible callback events to the firehose now which could probably be watched to spot when it runs | 18:14 |
| fungi | but that's real-time and not a historical log | 18:14 |
| EmilienM | ok no problem | 18:15 |
| fungi | though i bet we could build a pretty simple dashboard which logs a filtered subset of that stream now | 18:16 |
| *** bhavik1 has quit IRC | 18:18 | |
| *** lseki has left #openstack-infra | 18:19 | |
| *** lseki has joined #openstack-infra | 18:21 | |
| *** heyongli has quit IRC | 18:23 | |
| *** heyongli has joined #openstack-infra | 18:23 | |
| lseki | hi, I noticed that many openstack-tox-pylint jobs are reporting TIMED_OUT since a while | 18:23 |
| corvus | infra-root: i'm traveling to china for openinfradays next week, so i may not be around as much | 18:25 |
| *** florianf has quit IRC | 18:26 | |
| anteaya | corvus, safe travels | 18:26 |
| lseki | is that normal? I'm trying to figure out if it's the same issue I'm facing on my downstrem CI... | 18:27 |
| corvus | lseki: you should still get a log file which you can use to examine the cause of the timeout | 18:28 |
| fungi | corvus: thanks for the heads up! going to be at openinfradays cn? | 18:29 |
| fungi | (i'm not, but glad we have people in the community who will be) | 18:30 |
| *** rlandy_ has joined #openstack-infra | 18:31 | |
| corvus | yep, delivering a talk on zuul | 18:31 |
| *** rlandy_ has quit IRC | 18:31 | |
| *** r-daneel_ has joined #openstack-infra | 18:31 | |
| fungi | right on! | 18:31 |
| *** rlandy_ has joined #openstack-infra | 18:31 | |
| fungi | and thanks for going | 18:31 |
| *** r-daneel has quit IRC | 18:32 | |
| *** r-daneel_ is now known as r-daneel | 18:32 | |
| *** rlandy has quit IRC | 18:33 | |
| *** heyongli has quit IRC | 18:33 | |
| *** heyongli has joined #openstack-infra | 18:33 | |
| lseki | corvus: well, the logs are not very informative... it only says RUN END RESULT_TIMED_OUT after 25min of silence | 18:35 |
| lseki | like this one http://logs.openstack.org/41/565241/11/check/openstack-tox-pylint/864c19c/job-output.txt.gz#_2018-06-13_12_15_45_347296 | 18:35 |
| fungi | lseki: you might look in the job-output.json to see if there was any useful additional detail omitted from the log | 18:35 |
| *** rlandy_ is now known as rlandy | 18:36 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: fix the check-python-release role https://review.openstack.org/575750 | 18:37 |
| dhellmann | mordred, fungi: ^^ should fix the broken python3 packaging test role. I tested it with a local change in the release-test repo if you want to see the output: https://review.openstack.org/#/c/575751/ | 18:38 |
| EmilienM | fungi: did puppet run alraedy? | 18:39 |
| EmilienM | fungi: I'm waiting for it to do 5 rechecks and see if that helped | 18:39 |
| fungi | EmilienM: i'm operating a very hot wok at the moment, but once food is done i'll check | 18:39 |
| EmilienM | I also told tripleo team to not move anything to gate until situations improves | 18:39 |
| EmilienM | send me some plz :D | 18:40 |
| clarkb | if anyine is windering ripe berries are awesome | 18:43 |
| *** Goneri has joined #openstack-infra | 18:43 | |
| *** heyongli has quit IRC | 18:43 | |
| *** heyongli has joined #openstack-infra | 18:44 | |
| anteaya | clarkb, I was wondering, thank you | 18:45 |
| clarkb | they even had early season blackberries | 18:47 |
| anteaya | ooh that is early for blackberries | 18:49 |
| anteaya | in my mind blackberries mean August to me | 18:49 |
| fungi | EmilienM: checking now. i'll try to send you some beef fried rice but worried it'll clog the router | 18:50 |
| clarkb | anteaya: these were "silvan" blackberries and apparebtly they ripen early | 18:50 |
| anteaya | awesome | 18:50 |
| anteaya | seems early even for them: https://www.oregon-berries.com/variety/silvan-blackberry/ | 18:51 |
| anteaya | must be all your sun and rain | 18:52 |
| fungi | EmilienM: Jun 15 18:26:50 mirror02 puppet-user[17423]: (/Stage[main]/Openstack_project::Mirror/Httpd::Vhost[mirror02.dfw.rax.openstack.org]/File[50-mirror02.dfw.rax.openstack.org.conf]/content) content changed '{md5}09f54270d53cf4fc1cd50f2a72bac079' to '{md5}d720c33b25814f45ba0b2eb2b82b3669' | 18:52 |
| fungi | so the mirrors should have updated around 18:30 utc | 18:52 |
| fungi | i'd go ahead and test | 18:52 |
| *** heyongli has quit IRC | 18:53 | |
| *** heyongli has joined #openstack-infra | 18:54 | |
| *** rlpple has left #openstack-infra | 18:54 | |
| *** kaiokmo has quit IRC | 18:55 | |
| fungi | clarkb: growing up in the wilderness, all the blackberries on the mountainside were ripening this time of year. i guess i just always assumed all blackberries do | 18:56 |
| fungi | didn't realize there were multiple kinds | 18:58 |
| clarkb | fungi: ya most blackberries here are ripe in august | 18:58 |
| fungi | we'd get dewberries first, then blackberries, then strawberries and huckleberries and mulberries around the same timeframe i think | 18:58 |
| fungi | oh, and grapes in late summer too (fox grapes, scuppernongs and muscadines) | 18:59 |
| clarkb | EmilienM: fungi "cache miss: attempting entity save" | 19:00 |
| clarkb | we are hitting the docker backend now at least | 19:00 |
| clarkb | but as expected the cache needs to be populated | 19:01 |
| clarkb | we'll probably want to follow up in a few hours or on monday to see if the cache is hitting more frequently | 19:01 |
| *** heyongli has quit IRC | 19:04 | |
| *** heyongli has joined #openstack-infra | 19:04 | |
| anteaya | I'd love to have June blackberries | 19:08 |
| *** heyongli has quit IRC | 19:14 | |
| *** heyongli has joined #openstack-infra | 19:14 | |
| *** rlandy has quit IRC | 19:15 | |
| clarkb | 607:ff68:100:54:f816:3eff:fe17:8ae9 - - [15/Jun/2018:18:58:20 +0000] "GET /cloudflare/registry-v2/docker/registry/v2/blobs/sha256/7d/7dc0dca2b1516961d6b3200564049db0a6e0410b370bb2189e2efae0d368616f/data?verify=1529092100-t9ye%2F3CUo6K15%2BPXIO35vuJAiI0%3D HTTP/1.1" 200 74693368 cache hit | 19:17 |
| clarkb | "http://mirror.regionone.limestone.openstack.org:8081/registry-1.docker/v2/tripleopike/centos-binary-aodh-api/blobs/sha256:7dc0dca2b1516961d6b3200564049db0a6e0410b370bb2189e2efae0d368616f" "docker/1.13.1 go/go1.9.4 kernel/3.10.0-862.3.2.el7.x86_64 os/linux arch/amd64 UpstreamClient(docker-sdk-python/2.4.2)" | 19:17 |
| clarkb | EmilienM: fungi ^ seems to be working | 19:17 |
| *** felipemonteiro has joined #openstack-infra | 19:18 | |
| clarkb | not a small layer either | 19:18 |
| *** fried_rolls is now known as fried_rice | 19:18 | |
| *** rlandy has joined #openstack-infra | 19:18 | |
| fungi | 74mb? | 19:19 |
| fungi | (roughly) | 19:19 |
| clarkb | fungi: ya | 19:19 |
| clarkb | so far that is the only cache hit | 19:22 |
| clarkb | (I think that may mostly be due to needing to populate the cache) | 19:23 |
| EmilienM | fungi: ack | 19:24 |
| clarkb | and now 50 cache hits | 19:24 |
| clarkb | so ya it seems to be working \o/ | 19:24 |
| fungi | excellent | 19:24 |
| clarkb | 75! | 19:24 |
| *** heyongli has quit IRC | 19:24 | |
| *** heyongli has joined #openstack-infra | 19:24 | |
| clarkb | EmilienM: it will be useful to know if this helps job runtimes/reliability, but I think the caching proxies are doing the right thing now | 19:25 |
| fungi | would also be good to know if we start overrunning our cache space | 19:25 |
| fungi | yay it's fixed! (let's hope it's not so popular that it can't keep up?) | 19:26 |
| *** vtapia has quit IRC | 19:26 | |
| clarkb | fungi: ya that is a minor concern but we have 2x available on top of what we clean back to | 19:27 |
| clarkb | 50GB we clean to and ~100GB total | 19:27 |
| clarkb | fungi: also we reorged the fs structure to be easier on stat() | 19:27 |
| fungi | here's hoping it's sane | 19:28 |
| clarkb | though the current htcacheclean has been running for 27 minutes :/ | 19:28 |
| fungi | better than the 12(?) hours it used to take | 19:28 |
| EmilienM | clarkb: ok | 19:29 |
| clarkb | fungi: I WIP'd https://review.openstack.org/#/c/575520/1 since that data may not hold up now that we are dockering again | 19:29 |
| clarkb | fungi: looking at timestamps of existing objects in the cache I think this may have broken at the end of May so hasn't been long | 19:30 |
| *** felipemonteiro has quit IRC | 19:30 | |
| *** Goneri has quit IRC | 19:31 | |
| *** lifeless has joined #openstack-infra | 19:34 | |
| *** heyongli has quit IRC | 19:34 | |
| *** heyongli has joined #openstack-infra | 19:35 | |
| *** david-lyle is now known as dklyle | 19:37 | |
| *** rlandy_ has joined #openstack-infra | 19:39 | |
| *** Goneri has joined #openstack-infra | 19:39 | |
| *** rlandy has quit IRC | 19:41 | |
| *** pcichy has joined #openstack-infra | 19:43 | |
| *** heyongli has quit IRC | 19:45 | |
| *** heyongli has joined #openstack-infra | 19:45 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add openstack-service-broker project https://review.openstack.org/573459 | 19:48 |
| *** r-daneel has quit IRC | 19:52 | |
| AJaeger | mordred: https://review.openstack.org/#/c/575744/1 has a depends on an abandoned change - how do you want to proceed? | 19:52 |
| mordred | AJaeger: I made it depends-on the wrong patch because I'm dumb | 19:53 |
| mordred | AJaeger: I do think we should double-check that the new keypair got made everywhere before landing it though | 19:54 |
| AJaeger | ;) | 19:54 |
| *** heyongli has quit IRC | 19:55 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Use infra-root-keys-2018-06-15 in nodepool https://review.openstack.org/575744 | 19:55 |
| *** heyongli has joined #openstack-infra | 19:55 | |
| mordred | AJaeger: I fixed it | 19:56 |
| fungi | clarkb: sounds about right. i'd have expected complaints sooner otherwise | 19:56 |
| AJaeger | mordred: thanks, I let others review and check that it's there... | 19:57 |
| mordred | Im checking that it's there real quick | 19:57 |
| *** bobh has quit IRC | 19:58 | |
| openstackgerrit | Zane Bitter proposed openstack-infra/project-config master: Add Zuul config for openstack-service-broker https://review.openstack.org/573460 | 20:01 |
| clarkb | fungi: 1900 htcacheclean still running so ya this may be something we have to watch. I'm hoping it isnt though | 20:01 |
| *** lifeless has quit IRC | 20:02 | |
| *** lifeless has joined #openstack-infra | 20:03 | |
| *** dpawlik has joined #openstack-infra | 20:04 | |
| *** heyongli has quit IRC | 20:05 | |
| *** heyongli has joined #openstack-infra | 20:06 | |
| clarkb | fungi: its done now | 20:07 |
| clarkb | so about an hour I think | 20:07 |
| mordred | the new key is everywhere except for vexxhost | 20:07 |
| * mordred blames mnaser | 20:07 | |
| dhellmann | corvus : I think I see the issue with https://review.openstack.org/#/c/575801/1 and can propose an update, if you aren't already working on it | 20:09 |
| fungi | clarkb: maybe that'll be faster once it reaches a steady state again | 20:09 |
| *** dpawlik has quit IRC | 20:09 | |
| mordred | where do logs from cloud-launcher go? | 20:09 |
| clarkb | mordred: I think /var/log like puppet ansible | 20:09 |
| clarkb | but its own file? | 20:09 |
| mordred | ah - yes - there it is | 20:09 |
| mnaser | mordred: I wonder if it’s because it happens alphabetically lol | 20:11 |
| mordred | heh | 20:11 |
| mordred | nah - it was from the morning - it should have happend hours ago | 20:11 |
| mnaser | But anything our side? | 20:11 |
| mordred | I doubt it | 20:11 |
| corvus | dhellmann: by all means | 20:11 |
| mordred | it's just a nova keypair - it should be easy | 20:11 |
| dhellmann | corvus : done | 20:11 |
| mordred | yah - I don't see any errors in any files anywhere | 20:12 |
| mnaser | Heh. Alright. Let me know if I can assist in any way | 20:12 |
| mordred | mnaser: I will! | 20:13 |
| openstackgerrit | Merged openstack-infra/project-config master: Migrate the API-SIG to StoryBoard https://review.openstack.org/575120 | 20:15 |
| *** heyongli has quit IRC | 20:15 | |
| *** heyongli has joined #openstack-infra | 20:16 | |
| mordred | clarkb, mnaser: ok. I just added the key to vexxhost by hand | 20:17 |
| *** annp has quit IRC | 20:17 | |
| mordred | https://review.openstack.org/#/c/575744/ should be safe to land | 20:17 |
| EmilienM | so far so good I think | 20:19 |
| EmilienM | I see successful jobs in gate now | 20:19 |
| *** owalsh has quit IRC | 20:20 | |
| *** lseki has quit IRC | 20:22 | |
| *** heyongli has quit IRC | 20:26 | |
| *** heyongli has joined #openstack-infra | 20:26 | |
| clarkb | I'm still seeing the cache hit number go up on the limestone mirror | 20:26 |
| corvus | dhellmann: ah, thanks! :) | 20:26 |
| *** vtapia has joined #openstack-infra | 20:26 | |
| clarkb | corvus: did you see my note about a zuul release? is that still onyour radar? | 20:27 |
| *** r-daneel has joined #openstack-infra | 20:27 | |
| corvus | clarkb: yeah, i'd like to make one. i issued a recheck for a previously failing change as one last sanity check. should have the results in a few mins. | 20:29 |
| *** vtapia has quit IRC | 20:31 | |
| clarkb | cool | 20:33 |
| *** heyongli has quit IRC | 20:36 | |
| *** heyongli has joined #openstack-infra | 20:36 | |
| *** heyongli has quit IRC | 20:46 | |
| *** heyongli has joined #openstack-infra | 20:46 | |
| *** lifeless has quit IRC | 20:49 | |
| *** e0ne has quit IRC | 20:49 | |
| *** vtapia has joined #openstack-infra | 20:49 | |
| *** lifeless has joined #openstack-infra | 20:50 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/project-config master: Fix tripleo-docs doc job https://review.openstack.org/575849 | 20:51 |
| *** r-daneel has quit IRC | 20:54 | |
| *** r-daneel has joined #openstack-infra | 20:54 | |
| *** heyongli has quit IRC | 20:56 | |
| *** heyongli has joined #openstack-infra | 20:57 | |
| corvus | clarkb: it passed | 21:01 |
| corvus | clarkb, mnaser: i see a kata-runsh job queued; are there image availability problems? | 21:04 |
| clarkb | corvus: flavor availability + kata not actually ocmpiling so we have to hit the pre run failures 3 times before it unqueues | 21:06 |
| clarkb | corvus: kata only runs on vexxhost for now, I could move everything to run instead of doing their setup step in pre so that it only fails once | 21:06 |
| mnaser | I don’t have a ton of resources allocated to that flavour yet | 21:07 |
| *** heyongli has quit IRC | 21:07 | |
| mnaser | Are we hitting limits a lot? | 21:07 |
| *** heyongli has joined #openstack-infra | 21:07 | |
| clarkb | mnaser: its mostly nodepool not being built to provide resources to a single label type in one provider I think | 21:07 |
| clarkb | not optomized for that | 21:07 |
| clarkb | this will get better if 1) kata compiles work and 2) we stop running at capacity all the time (which should be better if we've made tripleo more reliable) | 21:09 |
| mnaser | clarkb: optimized to | 21:10 |
| mnaser | Run at high scale only :p | 21:10 |
| *** agopi has quit IRC | 21:14 | |
| *** agopi has joined #openstack-infra | 21:15 | |
| corvus | well, this is probably nodepool being fair | 21:15 |
| *** ldnunes has quit IRC | 21:15 | |
| *** rfolco has quit IRC | 21:15 | |
| corvus | each failure it goes back to the end of the line. so it's fairness at capacity combined with the pre-failures | 21:15 |
| *** yamahata has joined #openstack-infra | 21:16 | |
| corvus | (nodepool 0 was bad about this case, but nodepool 3 should be fine) | 21:16 |
| gtmanfred | mordred: or anyone else here, who manages os-client-config? I would really appreciate it if a new release could be made to include this https://github.com/openstack/os-client-config/commit/590b1723b5d70761dd14be132109173cfc1d9425 | 21:16 |
| *** heyongli has quit IRC | 21:17 | |
| *** heyongli has joined #openstack-infra | 21:17 | |
| clarkb | corvus: ya, I don't think its a big issue | 21:17 |
| *** kgiusti has left #openstack-infra | 21:18 | |
| *** slaweq has quit IRC | 21:18 | |
| *** slaweq has joined #openstack-infra | 21:19 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-cgit master: Add order to cgit package https://review.openstack.org/575852 | 21:21 |
| *** jtomasek_ has quit IRC | 21:21 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-cgit master: Update Gemfile for Zuulv3 https://review.openstack.org/571987 | 21:22 |
| *** slaweq has quit IRC | 21:24 | |
| *** heyongli has quit IRC | 21:27 | |
| *** heyongli has joined #openstack-infra | 21:27 | |
| *** edmondsw has quit IRC | 21:29 | |
| *** heyongli has quit IRC | 21:37 | |
| *** heyongli has joined #openstack-infra | 21:38 | |
| *** r-daneel has quit IRC | 21:39 | |
| *** boden has quit IRC | 21:44 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Improve kata-runsh job https://review.openstack.org/573748 | 21:45 |
| *** heyongli has quit IRC | 21:48 | |
| *** heyongli has joined #openstack-infra | 21:48 | |
| *** dbecker has quit IRC | 21:53 | |
| *** Swimingly has quit IRC | 21:53 | |
| *** Swimingly has joined #openstack-infra | 21:55 | |
| *** Swimingly has quit IRC | 21:55 | |
| *** Swimingly has joined #openstack-infra | 21:55 | |
| EmilienM | infra-root: can we move https://review.openstack.org/#/c/575855/ to the gate please (I hope it's the last request) | 21:57 |
| EmilienM | clarkb: things have improved from what I can tell | 21:58 |
| EmilienM | I'm looking at the logs and I haven't seen container failures since your change | 21:58 |
| *** yamamoto has joined #openstack-infra | 21:58 | |
| EmilienM | so far so good, I'll keep you posted here later today | 21:58 |
| *** heyongli has quit IRC | 21:58 | |
| *** heyongli has joined #openstack-infra | 21:58 | |
| clarkb | EmilienM: good to hear, thank you for checking | 21:59 |
| fungi | EmilienM: taken care of now | 21:59 |
| *** agopi has quit IRC | 22:00 | |
| EmilienM | thx | 22:00 |
| *** lifeless has quit IRC | 22:01 | |
| *** yamamoto has quit IRC | 22:04 | |
| clarkb | ok with tripleo happy and kata things debugged I'm in a spot to figure out what is next for packethost | 22:05 |
| clarkb | fungi: I think ianw had suggested we maybe juts manually increase max-servers to start? | 22:06 |
| clarkb | fungi: ya he says he thinks everything is as ready as it can be | 22:06 |
| clarkb | fungi: thoughts on setting max-servers to 10? | 22:06 |
| fungi | yeah, i can push up a patch now | 22:07 |
| *** felipemonteiro has joined #openstack-infra | 22:07 | |
| clarkb | the mirror there hasn't gotten the dockerhub fix | 22:07 |
| clarkb | I'm going to make sure that the ssh key for it is accepted | 22:07 |
| *** heyongli has quit IRC | 22:08 | |
| *** heyongli has joined #openstack-infra | 22:08 | |
| *** owalsh has joined #openstack-infra | 22:09 | |
| clarkb | hrm its / is also only ~80GB | 22:09 |
| clarkb | I'm guessing its on a similar flavor to the test nodes | 22:09 |
| clarkb | probably want to bump that up too. | 22:09 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Increase packethost max-servers to 10 https://review.openstack.org/575863 | 22:09 |
| ianw | clarkb: hmm, the mirror i brought up as m1.large, isn't that about what we usually use? | 22:09 |
| fungi | clarkb: ianw: ^ | 22:09 |
| clarkb | ianw: I think we try to allocate at least 100GB for afs and apache cache | 22:10 |
| *** hashar has quit IRC | 22:10 | |
| clarkb | more is better if we can but that is sort of the minimum? | 22:10 |
| fungi | ianw: we do but we add a cinder volume usually | 22:10 |
| clarkb | ya either via volume or bigger flavor | 22:10 |
| fungi | unless there is more available disk, right | 22:10 |
| fungi | or ephemeral or whatever | 22:10 |
| clarkb | Looks like 200GB cinder volume is common then we mount 100GB under apache cache and 100GB under openafs cache | 22:10 |
| fungi | depends on the provider obviously | 22:10 |
| *** slaweq has joined #openstack-infra | 22:11 | |
| clarkb | looking ta the flavors in that cloud avolume might be most appropriate | 22:13 |
| clarkb | unless we want to run an xlarge instance and have too much cpu and memory :) | 22:13 |
| ianw | hmm, it didn't like me trying to create a 200gb volume ... it's just in error status | 22:14 |
| ianw | | ed4686ae-c40d-4944-b153-d29ef187fbb4 | mirror | error | 204800 | | | 22:14 |
| fungi | i wonder if there's nothing backing cinder there | 22:15 |
| clarkb | if you volume show it does it tell you why it errored? sometimes that works on nova instances | 22:15 |
| clarkb | fungi: oh that could be | 22:15 |
| *** rlandy_ is now known as rlandy | 22:15 | |
| *** heyongli has quit IRC | 22:18 | |
| *** owalsh has quit IRC | 22:19 | |
| *** heyongli has joined #openstack-infra | 22:19 | |
| *** slaweq has quit IRC | 22:19 | |
| *** hemna_ has quit IRC | 22:20 | |
| ianw | no just error | 22:21 |
| fungi | the best kind og error | 22:22 |
| ianw | that was in openstackci-packethost project ... maybe the other one | 22:23 |
| ianw | works | 22:23 |
| fungi | ahh, so which one needed cinder quota got swapped backwards i guess? | 22:23 |
| clarkb | usually you get a quota error if it is a quota issue though rather than an object in an error state | 22:24 |
| clarkb | but worth testing | 22:24 |
| clarkb | in the short term maybe we just xlarge instance and then figure out cinder | 22:24 |
| *** yamahata has quit IRC | 22:24 | |
| clarkb | EmilienM: fungi I've also confirmed that a sha1 that was previously uncached and only redirected before had a cache miss to the backend then is served out of the cache | 22:25 |
| fungi | i don't suppose the volume create error was transient and might work when trying again? | 22:26 |
| ianw | oohh, hang on ... size is in gigabytes ... | 22:26 |
| fungi | yeah, so "200" | 22:26 |
| EmilienM | good to know | 22:27 |
| clarkb | er sha256 | 22:27 |
| fungi | we knew what you meant | 22:28 |
| ianw | | c1e49185-8d71-4049-ad8d-175c05b9173a | mirror | error | 200 | | | 22:28 |
| ianw | still errors with a 200gb volume, and also in the open-infra project | 22:28 |
| ianw | so yeah, i don't think it's setup | 22:28 |
| smcginnis | If they are running somewhat recent code, you may be able to get more info from "cinder message-list". Added in mv 3.3 I think. | 22:29 |
| *** heyongli has quit IRC | 22:29 | |
| smcginnis | It's a way to get async operation error messages. | 22:29 |
| *** heyongli has joined #openstack-infra | 22:29 | |
| ianw | if it doesn't obey OS_CLIENT_CONFIG_FILE it's dead to me :) jk | 22:31 |
| clarkb | its amazing how much easier clouds.yaml makes things | 22:32 |
| ianw | clarkb: i wonder if the cache sits at about 90% because it only needs about 90%, or it likes to keep some headroom? http://paste.openstack.org/show/723570/ | 22:39 |
| *** heyongli has quit IRC | 22:39 | |
| *** heyongli has joined #openstack-infra | 22:39 | |
| clarkb | Not sure, with apache we clean down to 50GB but it needs headroom as htcacheclean runs out of band | 22:40 |
| *** hongbin has quit IRC | 22:43 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Improve kata-runsh job https://review.openstack.org/573748 | 22:46 |
| *** dave-mccowan has quit IRC | 22:48 | |
| *** heyongli has quit IRC | 22:49 | |
| *** lifeless has joined #openstack-infra | 22:49 | |
| *** heyongli has joined #openstack-infra | 22:50 | |
| *** Goneri has quit IRC | 22:50 | |
| *** edmondsw has joined #openstack-infra | 22:50 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Log more information about what events trigger a reconfig https://review.openstack.org/575871 | 22:53 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Log more information about what events trigger a reconfig https://review.openstack.org/575871 | 22:54 |
| *** edmondsw has quit IRC | 22:55 | |
| *** heyongli has quit IRC | 22:59 | |
| *** heyongli has joined #openstack-infra | 23:00 | |
| *** yamamoto has joined #openstack-infra | 23:00 | |
| *** yamamoto has quit IRC | 23:06 | |
| *** heyongli has quit IRC | 23:10 | |
| *** heyongli has joined #openstack-infra | 23:10 | |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** hamzy_ has quit IRC | 23:15 | |
| *** slaweq has quit IRC | 23:16 | |
| *** owalsh_ has joined #openstack-infra | 23:16 | |
| *** caphrim007_ has joined #openstack-infra | 23:17 | |
| *** owalsh_ is now known as owalsh | 23:19 | |
| *** caphrim007 has quit IRC | 23:20 | |
| *** heyongli has quit IRC | 23:20 | |
| *** caphrim007_ has quit IRC | 23:20 | |
| *** heyongli has joined #openstack-infra | 23:20 | |
| *** gyee has quit IRC | 23:24 | |
| *** hamzy has joined #openstack-infra | 23:25 | |
| *** Swimingly has quit IRC | 23:26 | |
| *** Swimingly has joined #openstack-infra | 23:27 | |
| *** Swimingly has quit IRC | 23:27 | |
| *** Swimingly has joined #openstack-infra | 23:27 | |
| *** sthussey has quit IRC | 23:30 | |
| *** heyongli has quit IRC | 23:30 | |
| *** heyongli has joined #openstack-infra | 23:31 | |
| *** rpioso is now known as rpioso|afk | 23:36 | |
| *** gema has quit IRC | 23:37 | |
| *** heyongli has quit IRC | 23:40 | |
| *** tosky has quit IRC | 23:41 | |
| *** heyongli has joined #openstack-infra | 23:41 | |
| EmilienM | our tripleo-docs repo is stuck until https://review.openstack.org/#/c/575849/ gets merged, if someone can take a look! thankjs | 23:43 |
| *** sthussey has joined #openstack-infra | 23:43 | |
| *** Swimingly has quit IRC | 23:49 | |
| *** heyongli has quit IRC | 23:51 | |
| *** heyongli has joined #openstack-infra | 23:51 | |
| *** fried_rice is now known as efried | 23:53 | |
| *** agopi has joined #openstack-infra | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!