| *** s-shiono has joined #openstack-infra | 00:06 | |
| *** abelur_ has joined #openstack-infra | 00:15 | |
| *** ijw has joined #openstack-infra | 00:17 | |
| *** ijw has quit IRC | 00:18 | |
| *** wolverineav has joined #openstack-infra | 00:25 | |
| *** wolverineav has quit IRC | 00:30 | |
| *** tosky has quit IRC | 00:30 | |
| *** dave-mcc_ has joined #openstack-infra | 00:32 | |
| *** armax has joined #openstack-infra | 00:34 | |
| *** dave-mccowan has quit IRC | 00:35 | |
| *** dingyichen has joined #openstack-infra | 00:37 | |
| *** hongbin has joined #openstack-infra | 00:38 | |
| *** dingyichen has quit IRC | 00:39 | |
| *** shu-mutou has joined #openstack-infra | 00:39 | |
| *** rossella_s has quit IRC | 00:39 | |
| *** dingyichen has joined #openstack-infra | 00:40 | |
| *** rossella_s has joined #openstack-infra | 00:42 | |
| *** dingyichen has quit IRC | 00:43 | |
| *** threestrands has joined #openstack-infra | 00:43 | |
| *** threestrands has quit IRC | 00:43 | |
| *** threestrands has joined #openstack-infra | 00:43 | |
| *** dingyichen has joined #openstack-infra | 00:45 | |
| *** hongbin has quit IRC | 00:47 | |
| *** edmondsw has joined #openstack-infra | 00:52 | |
| *** edmondsw has quit IRC | 00:56 | |
| *** wolverineav has joined #openstack-infra | 00:57 | |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Zuul: Remove project name https://review.openstack.org/540466 | 01:05 |
|---|---|---|
| *** wolverineav has quit IRC | 01:06 | |
| *** ijw has joined #openstack-infra | 01:18 | |
| *** hongbin has joined #openstack-infra | 01:22 | |
| *** ijw has quit IRC | 01:23 | |
| *** rossella_s has quit IRC | 01:30 | |
| *** zhurong has joined #openstack-infra | 01:30 | |
| *** cuongnv has joined #openstack-infra | 01:30 | |
| *** rossella_s has joined #openstack-infra | 01:30 | |
| ianw | this stupid superbowl spam is one thing, but when it comes from "OVH" and makes me thing something must be up with ovh, then it's gone too far :) | 01:31 |
| *** gagehugo has quit IRC | 01:33 | |
| mnaser | Haha | 01:36 |
| *** claudiub|2 has quit IRC | 01:37 | |
| *** namnh has joined #openstack-infra | 01:44 | |
| *** kiennt26 has joined #openstack-infra | 01:56 | |
| *** xinliang has quit IRC | 01:56 | |
| *** mtreinish has quit IRC | 01:58 | |
| *** mtreinish has joined #openstack-infra | 01:59 | |
| *** threestrands has quit IRC | 02:01 | |
| *** wolverineav has joined #openstack-infra | 02:03 | |
| *** daidv has joined #openstack-infra | 02:06 | |
| *** wolverineav has quit IRC | 02:08 | |
| *** xinliang has joined #openstack-infra | 02:09 | |
| *** xinliang has quit IRC | 02:09 | |
| *** xinliang has joined #openstack-infra | 02:09 | |
| *** dhill_ has quit IRC | 02:11 | |
| *** threestrands has joined #openstack-infra | 02:16 | |
| *** ijw has joined #openstack-infra | 02:20 | |
| *** dingyichen has quit IRC | 02:26 | |
| *** ijw has quit IRC | 02:31 | |
| *** edmondsw has joined #openstack-infra | 02:40 | |
| *** zhurong has quit IRC | 02:43 | |
| *** MESCALINE_ has joined #openstack-infra | 02:44 | |
| *** edmondsw has quit IRC | 02:45 | |
| *** liujiong has joined #openstack-infra | 02:45 | |
| *** sshnaidm_ has joined #openstack-infra | 02:45 | |
| *** yamamoto has joined #openstack-infra | 02:47 | |
| *** sshnaidm has quit IRC | 02:47 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Use yarn and webpack to manage zuul-web javascript https://review.openstack.org/538099 | 03:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add babel transpiling enabling use of ES6 features https://review.openstack.org/538125 | 03:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add StandardJS linting and analysis https://review.openstack.org/538126 | 03:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Fix source_url handling for jobs view https://review.openstack.org/538127 | 03:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Fix StandardJS warnings and turn them to errors https://review.openstack.org/538128 | 03:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add bundle analysis to the lint target https://review.openstack.org/538129 | 03:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Inject url endpoint information https://review.openstack.org/538130 | 03:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Make bundle of build web content https://review.openstack.org/538131 | 03:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Remove use strict https://review.openstack.org/538132 | 03:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/jobs/{job_name} route https://review.openstack.org/535545 | 03:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add jobs graph rendering https://review.openstack.org/537869 | 03:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: /{tenant}/projects.json routes https://review.openstack.org/537870 | 03:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add project pipeline rendering https://review.openstack.org/537871 | 03:01 |
| *** annp has joined #openstack-infra | 03:05 | |
| *** agopi__ has joined #openstack-infra | 03:07 | |
| *** Adri2000 has quit IRC | 03:12 | |
| *** wolverineav has joined #openstack-infra | 03:14 | |
| *** ykarel has joined #openstack-infra | 03:14 | |
| *** wolverineav has quit IRC | 03:18 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Refactor run_handler to be generic https://review.openstack.org/535554 | 03:18 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Refactor NodeLauncher to be generic https://review.openstack.org/535555 | 03:18 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenContainer driver https://review.openstack.org/535556 | 03:18 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a Kubernetes driver https://review.openstack.org/535557 | 03:18 |
| *** zhenguo has joined #openstack-infra | 03:21 | |
| *** ramishra has joined #openstack-infra | 03:24 | |
| *** sshnaidm_ has quit IRC | 03:24 | |
| *** ramishra has quit IRC | 03:27 | |
| *** agopi__ is now known as agopi | 03:32 | |
| *** dave-mcc_ has quit IRC | 03:32 | |
| *** sree has joined #openstack-infra | 03:34 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Add /node-list to the webapp https://review.openstack.org/535562 | 03:35 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Add /label-list to the webapp https://review.openstack.org/535563 | 03:35 |
| *** gcb has joined #openstack-infra | 03:37 | |
| *** cshastri has joined #openstack-infra | 03:39 | |
| *** wolverineav has joined #openstack-infra | 03:50 | |
| *** olaph1 has joined #openstack-infra | 03:50 | |
| *** olaph has quit IRC | 03:51 | |
| *** gongysh has joined #openstack-infra | 03:54 | |
| *** wolverineav has quit IRC | 03:54 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Remove legacy-devstack-dsvm-py35-updown from devstack https://review.openstack.org/540707 | 03:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-devstack-dsvm-py35-updown https://review.openstack.org/540708 | 04:00 |
| *** ykarel has quit IRC | 04:06 | |
| *** psachin has joined #openstack-infra | 04:07 | |
| *** ykarel has joined #openstack-infra | 04:07 | |
| *** threestrands has quit IRC | 04:07 | |
| *** gongysh has quit IRC | 04:13 | |
| *** threestrands has joined #openstack-infra | 04:19 | |
| *** wolverineav has joined #openstack-infra | 04:27 | |
| *** edmondsw has joined #openstack-infra | 04:28 | |
| *** ramishra has joined #openstack-infra | 04:31 | |
| *** wolverineav has quit IRC | 04:32 | |
| *** edmondsw has quit IRC | 04:33 | |
| *** gongysh has joined #openstack-infra | 04:33 | |
| *** threestrands has quit IRC | 04:36 | |
| *** threestrands has joined #openstack-infra | 04:37 | |
| *** gongysh has quit IRC | 04:38 | |
| *** agopi_ has joined #openstack-infra | 04:42 | |
| *** ykarel_ has joined #openstack-infra | 04:42 | |
| *** agopi has quit IRC | 04:45 | |
| *** ykarel has quit IRC | 04:45 | |
| *** wolverineav has joined #openstack-infra | 04:59 | |
| *** rfolco|ruck|off has quit IRC | 05:02 | |
| *** pgadiya has joined #openstack-infra | 05:03 | |
| *** wolverineav has quit IRC | 05:03 | |
| ianw | covus / mordred : i feel like there's definitely something up with the streaming console output ... http://paste.openstack.org/show/661956/ | 05:06 |
| ianw | 2018-02-05 05:03:42.023493 | [e] Waiting on logger | 05:06 |
| ianw | 2018-02-05 05:03:42.026283 | [o] Waiting on logger | 05:06 |
| ianw | 2018-02-05 05:03:42.032497 | [c] Waiting on logger | 05:06 |
| ianw | 2018-02-05 05:03:42.037417 | [r] Waiting on logger | 05:06 |
| ianw | a random collection of single letters of the world "controller" it seems? | 05:06 |
| ianw | odd | 05:06 |
| *** threestrands has quit IRC | 05:07 | |
| AJaeger | odd ;( | 05:15 |
| *** rfolco|ruck|off has joined #openstack-infra | 05:15 | |
| *** coolsvap has joined #openstack-infra | 05:16 | |
| AJaeger | ianw, could you put https://review.openstack.org/#/c/540600/ https://review.openstack.org/#/c/540595/ and https://review.openstack.org/540596 on your review list, please? Those are part of getting rid of run-tox.sh | 05:16 |
| *** claudiub|2 has joined #openstack-infra | 05:20 | |
| ianw | covus: i'm fairly suspicous of https://review.openstack.org/#/c/540623/ (Use getfqdn instead of gethostname) seeing as the log line is 'self._log("[%s] Waiting on logger" % host,' | 05:21 |
| *** sshnaidm has joined #openstack-infra | 05:25 | |
| *** zhenguo has quit IRC | 05:31 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add buildset-artifacts-location https://review.openstack.org/530679 | 05:31 |
| *** wolverineav has joined #openstack-infra | 05:34 | |
| *** olaph has joined #openstack-infra | 05:36 | |
| *** olaph1 has quit IRC | 05:38 | |
| *** wolverineav has quit IRC | 05:38 | |
| *** zhurong has joined #openstack-infra | 05:38 | |
| *** jaosorior has joined #openstack-infra | 05:39 | |
| *** hongbin has quit IRC | 05:40 | |
| *** ykarel__ has joined #openstack-infra | 05:40 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add recursive-download https://review.openstack.org/540716 | 05:40 |
| *** threestrands has joined #openstack-infra | 05:42 | |
| *** ykarel_ has quit IRC | 05:43 | |
| *** links has joined #openstack-infra | 05:46 | |
| *** threestrands has quit IRC | 05:47 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add linters job and role https://review.openstack.org/530682 | 05:47 |
| *** threestrands has joined #openstack-infra | 05:47 | |
| *** links has quit IRC | 05:49 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-lint job https://review.openstack.org/532083 | 05:50 |
| *** links has joined #openstack-infra | 05:51 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-review job https://review.openstack.org/535223 | 05:52 |
| *** dhajare has joined #openstack-infra | 05:55 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-upload-to-galaxy job https://review.openstack.org/532084 | 05:57 |
| *** links has quit IRC | 05:59 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-spec job https://review.openstack.org/532085 | 05:59 |
| *** links has joined #openstack-infra | 06:00 | |
| AJaeger | tobiash: do you have any idea about the getfqdn issue that ianw mentions above? | 06:01 |
| *** threestrands_ has joined #openstack-infra | 06:04 | |
| *** threestrands has quit IRC | 06:06 | |
| *** wolverineav has joined #openstack-infra | 06:09 | |
| *** links has quit IRC | 06:14 | |
| *** wolverineav has quit IRC | 06:14 | |
| *** aeng has quit IRC | 06:15 | |
| *** links has joined #openstack-infra | 06:16 | |
| *** edmondsw has joined #openstack-infra | 06:17 | |
| *** olaph1 has joined #openstack-infra | 06:19 | |
| *** edmondsw has quit IRC | 06:21 | |
| *** olaph has quit IRC | 06:22 | |
| *** rossella_s has quit IRC | 06:31 | |
| *** rossella_s has joined #openstack-infra | 06:32 | |
| *** e0ne has joined #openstack-infra | 06:38 | |
| *** kiennt26 has quit IRC | 06:39 | |
| openstackgerrit | Hiroaki Kobayashi proposed openstack-infra/project-config master: Add api-ref-jobs to blazar jobs https://review.openstack.org/540728 | 06:40 |
| *** gongysh has joined #openstack-infra | 06:44 | |
| *** e0ne has quit IRC | 06:45 | |
| *** jaosorior has quit IRC | 06:50 | |
| *** ykarel__ is now known as ykarel | 06:52 | |
| *** pcaruana has joined #openstack-infra | 07:04 | |
| *** dsariel has joined #openstack-infra | 07:05 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix openstack-tox-py35-with-ovsdbapp-master for networking-ovn https://review.openstack.org/540647 | 07:07 |
| *** e0ne has joined #openstack-infra | 07:09 | |
| AJaeger | frickler, could you put https://review.openstack.org/#/c/540600/ https://review.openstack.org/#/c/540595/ and https://review.openstack.org/540596 and https://review.openstack.org/540658 on your review list, please? Those are part of getting rid of run-tox.sh | 07:10 |
| *** jaosorior has joined #openstack-infra | 07:12 | |
| *** alexchadin has joined #openstack-infra | 07:12 | |
| *** e0ne_ has joined #openstack-infra | 07:13 | |
| *** e0ne has quit IRC | 07:14 | |
| openstackgerrit | liusheng proposed openstack-infra/zuul master: Fix AttributeError when handle periodic job with github driver https://review.openstack.org/536645 | 07:16 |
| *** e0ne_ has quit IRC | 07:17 | |
| *** rcernin has quit IRC | 07:18 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: mqtt: add basic reporter https://review.openstack.org/535543 | 07:18 |
| *** xinliang has quit IRC | 07:18 | |
| *** agopi_ has quit IRC | 07:20 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: config: add statsd-server config parameter https://review.openstack.org/535560 | 07:21 |
| *** CrayZee has joined #openstack-infra | 07:26 | |
| *** makowals has joined #openstack-infra | 07:27 | |
| *** makowals has quit IRC | 07:28 | |
| *** e0ne has joined #openstack-infra | 07:29 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Add requirements to periodic kolla-k8s jobs https://review.openstack.org/540744 | 07:29 |
| *** e0ne has quit IRC | 07:29 | |
| *** mandre_afk is now known as mandre | 07:41 | |
| *** d0ugal has quit IRC | 07:42 | |
| *** alexchadin has quit IRC | 07:42 | |
| *** alexchadin has joined #openstack-infra | 07:43 | |
| *** AJaeger has quit IRC | 07:43 | |
| *** armaan has joined #openstack-infra | 07:44 | |
| *** AJaeger has joined #openstack-infra | 07:45 | |
| *** kjackal has joined #openstack-infra | 07:51 | |
| tobiash | AJaeger: the getfqdn should have fixed the issue that the executors only state ze01 as hostname instead of ze01.o.o | 07:51 |
| *** andreas_s has joined #openstack-infra | 07:54 | |
| tobiash | but that seems weird as _read_log connects to the node and not to the executor | 07:55 |
| tobiash | the hostname the executor reports should not matter here | 07:56 |
| *** kjackal has quit IRC | 07:58 | |
| *** b_bezak has joined #openstack-infra | 08:00 | |
| *** jaosorior has quit IRC | 08:00 | |
| tobiash | AJaeger, ianw: do you have the job at hand? | 08:00 |
| tobiash | I'd be interested in the ansible inventory | 08:00 |
| tobiash | http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/ansible/callback/zuul_stream.py#n208 | 08:02 |
| tobiash | this suggests that it retrieves the host from ansible_host in the inventory (and expects an ip) | 08:02 |
| *** iyamahat has joined #openstack-infra | 08:02 | |
| AJaeger | tobiash: http://logs.openstack.org/04/540704/2/check/devstack-platform-centos-7/fd901aa | 08:02 |
| AJaeger | tobiash: sorry, no logs there ;( | 08:03 |
| AJaeger | That was the location from the paste | 08:03 |
| tobiash | AJaeger: just wanted to say file not found ;) | 08:03 |
| *** eumel8 has joined #openstack-infra | 08:04 | |
| tobiash | AJaeger: found it: http://logs.openstack.org/04/540704/3/check/devstack-platform-centos-7/6d76fb4/ | 08:04 |
| *** edmondsw has joined #openstack-infra | 08:05 | |
| *** slaweq has joined #openstack-infra | 08:05 | |
| tobiash | but the inventory looks ok | 08:05 |
| AJaeger | tobiash: no idea - I was just messenger, perhaps ianw knows more... | 08:06 |
| *** d0ugal has joined #openstack-infra | 08:06 | |
| AJaeger | happens on both centos and openSUSE, e.g. http://logs.openstack.org/04/540704/3/check/devstack-platform-opensuse-423/9ac6e47/job-output.txt.gz | 08:07 |
| *** alexchadin has quit IRC | 08:08 | |
| tobiash | what's strange is that I don't find any other job with such a behavior | 08:09 |
| *** giblet is now known as gibi | 08:09 | |
| *** edmondsw has quit IRC | 08:09 | |
| *** iyamahat has quit IRC | 08:09 | |
| *** andreas_s has quit IRC | 08:10 | |
| *** hjensas has quit IRC | 08:11 | |
| AJaeger | neither can I ;( | 08:12 |
| *** hashar has joined #openstack-infra | 08:12 | |
| *** wolverineav has joined #openstack-infra | 08:12 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Refactor run_handler to be generic https://review.openstack.org/535554 | 08:12 |
| *** andreas_s has joined #openstack-infra | 08:15 | |
| *** ralonsoh has joined #openstack-infra | 08:16 | |
| *** wolverineav has quit IRC | 08:17 | |
| *** s-shiono has quit IRC | 08:17 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Convert ansible-role-functional jobs https://review.openstack.org/540600 | 08:18 |
| *** florianf has joined #openstack-infra | 08:19 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove jobs from project-config-example https://review.openstack.org/540658 | 08:20 |
| ianw | tobiash / corvus : haha, yep, on closer inspection it is me at fault (no surprise :) ... https://review.openstack.org/#/c/540704/3..4/.zuul.yaml | 08:24 |
| ianw | that became more obvious in the ara output when all the jobs collecting facts for "c" "o" "n" ... failed | 08:25 |
| AJaeger | ianw: ah, a list ;( | 08:25 |
| *** andreas_s has quit IRC | 08:25 | |
| ianw | still, maybe zuul should have rejected my bad config | 08:25 |
| AJaeger | ianw: fixed openSUSE caps quickly for you... | 08:25 |
| tobiash | nice catch :) | 08:25 |
| ianw | cool ... I *think* I'm understanding the hierarchy correctly to translate these jobs into native, feedback welcome. i want to put f27 jobs in, but don't want to bother copying out legacy jobs | 08:26 |
| *** d0ugal has quit IRC | 08:26 | |
| AJaeger | ianw: good approach | 08:27 |
| AJaeger | ianw, frickler, now https://review.openstack.org/#/c/540659/ is ready for review. Thanks! | 08:27 |
| *** tesseract has joined #openstack-infra | 08:28 | |
| *** efoley has joined #openstack-infra | 08:28 | |
| *** andreas_s has joined #openstack-infra | 08:30 | |
| *** andreas_s has quit IRC | 08:30 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/nodepool master: Convert nodepool-zuul-functional job https://review.openstack.org/540595 | 08:30 |
| *** andreas_s has joined #openstack-infra | 08:30 | |
| *** dsariel has quit IRC | 08:31 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 08:34 | |
| *** olaph has joined #openstack-infra | 08:37 | |
| *** amoralej|off is now known as amoralej | 08:38 | |
| *** dsariel has joined #openstack-infra | 08:40 | |
| *** olaph1 has quit IRC | 08:40 | |
| *** e0ne has joined #openstack-infra | 08:43 | |
| *** jpena|off is now known as jpena | 08:45 | |
| *** dsariel has quit IRC | 08:46 | |
| *** d0ugal has joined #openstack-infra | 08:47 | |
| *** threestrands_ has quit IRC | 08:54 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Status branch protection checking for github https://review.openstack.org/535680 | 08:55 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/nodepool master: Convert nodepool-zuul-functional job https://review.openstack.org/540595 | 08:58 |
| *** olaph has quit IRC | 09:01 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Status branch protection checking for github https://review.openstack.org/535680 | 09:03 |
| openstackgerrit | Hiroaki Kobayashi proposed openstack-infra/project-config master: Add publish-openstack-sphinx-docs to blazar-dashboard jobs https://review.openstack.org/540771 | 09:04 |
| *** wolverineav has joined #openstack-infra | 09:04 | |
| *** kjackal has joined #openstack-infra | 09:05 | |
| openstackgerrit | Hiroaki Kobayashi proposed openstack-infra/project-config master: Add publish-openstack-sphinx-docs to blazar-dashboard jobs https://review.openstack.org/540771 | 09:05 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix github connection for standalone debugging https://review.openstack.org/540772 | 09:06 |
| *** shardy has joined #openstack-infra | 09:07 | |
| *** efoley has quit IRC | 09:08 | |
| *** Adri2000 has joined #openstack-infra | 09:09 | |
| *** wolverineav has quit IRC | 09:09 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Enhance github debugging script for apps https://review.openstack.org/540774 | 09:11 |
| *** liujiong has quit IRC | 09:11 | |
| *** timothyb89 has joined #openstack-infra | 09:11 | |
| *** liujiong has joined #openstack-infra | 09:12 | |
| *** pblaho has joined #openstack-infra | 09:13 | |
| *** ramishra has quit IRC | 09:13 | |
| *** ramishra has joined #openstack-infra | 09:13 | |
| *** wolverineav has joined #openstack-infra | 09:16 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: DNM: Fix ansible-role-functional https://review.openstack.org/540778 | 09:16 |
| *** wolverineav has quit IRC | 09:20 | |
| *** derekh has joined #openstack-infra | 09:20 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Revert "Convert ansible-role-functional jobs" https://review.openstack.org/540782 | 09:22 |
| *** jpich has joined #openstack-infra | 09:22 | |
| *** dbecker has joined #openstack-infra | 09:27 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy project-config jobs https://review.openstack.org/540659 | 09:27 |
| *** ykarel has quit IRC | 09:29 | |
| *** ykarel has joined #openstack-infra | 09:30 | |
| *** kopecmartin has joined #openstack-infra | 09:32 | |
| *** alexchadin has joined #openstack-infra | 09:35 | |
| *** alexchadin has quit IRC | 09:39 | |
| *** alexchadin has joined #openstack-infra | 09:40 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: DNM: Fix ansible-role-functional https://review.openstack.org/540778 | 09:47 |
| *** wolverineav has joined #openstack-infra | 09:55 | |
| *** dsariel has joined #openstack-infra | 09:56 | |
| *** wolverineav has quit IRC | 09:59 | |
| *** askb has quit IRC | 10:00 | |
| *** abelur_ has quit IRC | 10:00 | |
| *** dtantsur|afk is now known as dtantsur | 10:05 | |
| *** sree has quit IRC | 10:05 | |
| *** sree has joined #openstack-infra | 10:06 | |
| *** annp has quit IRC | 10:09 | |
| *** cuongnv has quit IRC | 10:10 | |
| *** sree has quit IRC | 10:14 | |
| *** sshnaidm is now known as sshnaidm|afk | 10:14 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: DNM: Fix ansible-role-functional https://review.openstack.org/540778 | 10:18 |
| *** abelur_ has joined #openstack-infra | 10:19 | |
| *** pcichy has quit IRC | 10:20 | |
| *** askb has joined #openstack-infra | 10:20 | |
| *** liujiong has quit IRC | 10:22 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: DNM: Fix ansible-role-functional https://review.openstack.org/540778 | 10:25 |
| *** stakeda has quit IRC | 10:26 | |
| *** zoli is now known as zoli|lunch | 10:33 | |
| *** zoli|lunch is now known as zoli | 10:33 | |
| *** namnh has quit IRC | 10:33 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Fix ansible-role-functional jobs https://review.openstack.org/540778 | 10:36 |
| *** ldnunes has joined #openstack-infra | 10:37 | |
| *** ykarel is now known as ykarel|away | 10:40 | |
| *** jaosorior has joined #openstack-infra | 10:43 | |
| AJaeger | ianw, frickler, my earlier change for ansible-role-functional was untested and thus broken ;( Sorry. 540778 above should fix it properly - and is tested now. | 10:44 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Replace legacy ansible-role func jobs https://review.openstack.org/540602 | 10:44 |
| AJaeger | config-core, please review 540778 | 10:44 |
| *** ykarel|away has quit IRC | 10:44 | |
| AJaeger | - and followup 540602 as well. | 10:45 |
| *** tosky has joined #openstack-infra | 10:51 | |
| *** tosky has quit IRC | 10:52 | |
| *** tosky has joined #openstack-infra | 10:52 | |
| *** lucas-afk is now known as lucasagomes | 10:52 | |
| *** rfolco|ruck|off is now known as rfolco|ruck | 10:57 | |
| *** sshnaidm|afk is now known as sshnaidm|lnch | 10:58 | |
| *** sambetts|afk is now known as sambetts | 11:00 | |
| *** shardy has quit IRC | 11:10 | |
| *** wolverineav has joined #openstack-infra | 11:14 | |
| *** andreas_s has quit IRC | 11:16 | |
| *** andreas_s has joined #openstack-infra | 11:17 | |
| *** gongysh has quit IRC | 11:17 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Enhance github debugging script for apps https://review.openstack.org/540774 | 11:22 |
| *** gfidente has joined #openstack-infra | 11:23 | |
| *** gfidente has joined #openstack-infra | 11:23 | |
| tosky | AJaeger (and others): is the requirements-check job supposed to fail for a project (python-tempestconf) not listed in projects.txt of requirements.git? | 11:23 |
| tosky | namely http://logs.openstack.org/35/538835/3/gate/requirements-check/3497f40/job-output.txt.gz#_2018-02-04_21_31_14_173300 | 11:23 |
| *** b_bezak has left #openstack-infra | 11:26 | |
| *** andreas_s has quit IRC | 11:26 | |
| *** kjackal has quit IRC | 11:30 | |
| *** kjackal has joined #openstack-infra | 11:30 | |
| *** andreas_s has joined #openstack-infra | 11:31 | |
| *** armaan has quit IRC | 11:32 | |
| AJaeger | tosky: sure - it fails for every project that adds the check-requirements template | 11:40 |
| *** andreas_s has quit IRC | 11:40 | |
| tosky | AJaeger: thanks; so we should remove that for python-tempestconf, I guess | 11:41 |
| *** edmondsw has joined #openstack-infra | 11:41 | |
| tosky | is there a "softer" job that checks for incompatibility, but not strict matches? (for example, in this case the version from requirements is a subset, there is a not-empty intersection) | 11:41 |
| AJaeger | tosky: there's no softer job. But let me look at the change... | 11:42 |
| *** alexchadin has quit IRC | 11:44 | |
| *** claudiub has joined #openstack-infra | 11:44 | |
| AJaeger | tosky: I would update all requirements for the repo to be in sync with what we have in requirements repo. | 11:44 |
| tosky | AJaeger: no, because tempestconf can work for more versions | 11:45 |
| *** alexchadin has joined #openstack-infra | 11:45 | |
| tosky | so it makes sense to have it out of requirements | 11:45 |
| AJaeger | tosky: then remove the check-requirements template | 11:45 |
| tosky | ack, perfect :) | 11:45 |
| *** edmondsw has quit IRC | 11:45 | |
| *** andreas_s has joined #openstack-infra | 11:46 | |
| AJaeger | tosky: since it's not listed in requirements repo, you can easily remove it. | 11:46 |
| tosky | going to send a review | 11:46 |
| *** claudiub|2 has quit IRC | 11:47 | |
| *** panda|off is now known as panda | 11:49 | |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: tempestconf is not in requirements, remove the check job https://review.openstack.org/540836 | 11:50 |
| tosky | chandankumar, arxcruz, kopecmartin: ^^ | 11:50 |
| AJaeger | tosky: talk with chandankumar about https://review.openstack.org/#/c/538838/ | 11:50 |
| tosky | AJaeger: I think he mostly wanted the publish job | 11:51 |
| *** pgadiya has quit IRC | 11:51 | |
| AJaeger | tosky, chandankumar that change introduced the check-requirements and it merged 3 days ago | 11:51 |
| *** armaan has joined #openstack-infra | 11:51 | |
| AJaeger | tosky: might be - let him confirm... | 11:51 |
| * tosky suspect a copy-paste | 11:51 | |
| AJaeger | tosky: could be - it's not mentioned in commit message. | 11:52 |
| *** tpsilva has joined #openstack-infra | 11:52 | |
| AJaeger | tosky: oh, mentioned in title - so intended | 11:53 |
| tosky | AJaeger: not the reason | 11:53 |
| tosky | still suspect a copy-paste, let's see | 11:53 |
| chandankumar | tosky: AJaeger i wanted to publish the package to pypi so that later on i can add in requirements so that it can be consumed with refstack-cliemt | 11:54 |
| tosky | chandankumar: I think there is a misunderstanding here | 11:54 |
| tosky | chandankumar: refstack-client itself is not in projects.txt of requirements.git | 11:54 |
| tosky | chandankumar: where did you see about the requirement of requirements.git (!) for the usage in refstack-client ? | 11:55 |
| *** andreas_s has quit IRC | 11:55 | |
| *** andreas_s has joined #openstack-infra | 11:56 | |
| chandankumar | tosky: sorry requirements is not used in refstack-client, but later on needed, i can submit a patch to requirements project | 11:58 |
| tosky | chandankumar: why is it needed? I disagree | 11:59 |
| tosky | chandankumar: if we want to use tempestconf for more OpenStack releases, we can't put it in requirements.git | 11:59 |
| tosky | my question still stand: where does the requirement of being in requirements.git come from? | 12:00 |
| tosky | given that even refstack-client is not there? | 12:00 |
| tosky | we can't have both: usage for more releases, and tracked in requirements.git | 12:01 |
| *** aviau has quit IRC | 12:01 | |
| *** aviau has joined #openstack-infra | 12:01 | |
| chandankumar | I have added check-jobs for updating requirements.txt for the tempestconf project. | 12:01 |
| AJaeger | chandankumar: and that one fails now on your change 538835 ;) | 12:02 |
| chandankumar | but i was not knowing that i need to add it to requirments project for the same | 12:02 |
| *** pcaruana has quit IRC | 12:02 | |
| AJaeger | chandankumar: if you use check-requirements, it means that you can only use requirements as listed in http://git.openstack.org/cgit/openstack/requirements/tree/global-requirements.txt | 12:03 |
| tosky | chandankumar: no, that's not what you want :) | 12:03 |
| tosky | chandankumar: we don't want to automatically update requirements.txt - we can't | 12:03 |
| AJaeger | chandankumar: change 538835 adds a line that is not in global-requirements | 12:03 |
| *** andreas_s has quit IRC | 12:04 | |
| *** andreas_s has joined #openstack-infra | 12:04 | |
| AJaeger | chandankumar: you might still be able to *add* python-tempestconf to the global-requirements file so that others can use it -> best to discuss on #openstack-requirements. But if the repo needs to support several OpenStack releases, you might not be able to use the requirements sync as tosky says | 12:05 |
| chandankumar | tosky: but updating the requirements may check whether what is getting broken on a new release of dependencies. | 12:07 |
| tosky | chandankumar: we can't. | 12:08 |
| tosky | it's either ... or | 12:08 |
| tosky | either you are in requirements.txt and your dependencies are bumped to follow the last openstack, or you update them manually and you are compatible with multiple releases | 12:08 |
| tosky | what's missing is the "soft checker" that checks if your manually updated dependencies are incompabile with one or more openstack releases | 12:09 |
| chandankumar | so if i get correctly first tempestconf should be consumed as a dependency by any other openstack project (which are already in global requirmenets) | 12:10 |
| *** jaosorior has quit IRC | 12:16 | |
| AJaeger | rcarrillocruz: could you review https://review.openstack.org/#/c/540596/ - infra-ansible change, please? | 12:16 |
| *** gongysh has joined #openstack-infra | 12:17 | |
| *** dtantsur is now known as dtantsur|brb | 12:17 | |
| AJaeger | chandankumar: best to discuss the options with requirements team. I suggest for now we remove the job - and if needed, we can add it back again later. | 12:17 |
| chandankumar | AJaeger: sure | 12:17 |
| *** panda is now known as panda|lunch | 12:18 | |
| tosky | I don't think that having other projects consume tempestconf has nothing to do with the usage of requirements.git by tempestconf itself | 12:19 |
| *** jpena is now known as jpena|lunch | 12:25 | |
| *** sshnaidm|lnch is now known as sshnaidm|afk | 12:29 | |
| *** gcb has quit IRC | 12:30 | |
| openstackgerrit | Merged openstack-infra/project-config master: tempestconf is not in requirements, remove the check job https://review.openstack.org/540836 | 12:35 |
| *** amoralej is now known as amoralej|lunch | 12:44 | |
| *** fultonj has joined #openstack-infra | 12:46 | |
| *** dbecker has quit IRC | 12:55 | |
| *** psachin has quit IRC | 12:56 | |
| *** panda|lunch is now known as panda | 12:57 | |
| *** gongysh has quit IRC | 12:58 | |
| *** yamamoto has quit IRC | 13:01 | |
| *** gongysh has joined #openstack-infra | 13:02 | |
| *** sshnaidm|afk is now known as sshnaidm | 13:02 | |
| *** alexchadin has quit IRC | 13:03 | |
| *** alexchadin has joined #openstack-infra | 13:04 | |
| *** jlabarre has joined #openstack-infra | 13:05 | |
| *** edmondsw has joined #openstack-infra | 13:13 | |
| *** amoralej|lunch is now known as amoralej | 13:16 | |
| *** yamamoto has joined #openstack-infra | 13:19 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: config: add statsd-server config parameter https://review.openstack.org/535560 | 13:21 |
| *** cshastri has quit IRC | 13:21 | |
| *** gongysh has quit IRC | 13:25 | |
| *** dhajare has quit IRC | 13:26 | |
| *** dhajare has joined #openstack-infra | 13:26 | |
| *** jpena|lunch is now known as jpena | 13:27 | |
| AJaeger | pabelanger: good morning! Before you recheck the ansible-role-zuul, please review https://review.openstack.org/540778 - I broke the jobs ;( | 13:29 |
| AJaeger | pabelanger: and https://review.openstack.org/537789 (windmill change) is ready now as well | 13:29 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: mqtt: add basic reporter https://review.openstack.org/535543 | 13:30 |
| *** coolsvap has quit IRC | 13:34 | |
| *** edmondsw has quit IRC | 13:36 | |
| *** edmondsw has joined #openstack-infra | 13:36 | |
| *** jaosorior has joined #openstack-infra | 13:37 | |
| *** edmondsw has quit IRC | 13:41 | |
| *** edmondsw has joined #openstack-infra | 13:44 | |
| *** armaan has quit IRC | 13:48 | |
| *** armaan has joined #openstack-infra | 13:48 | |
| *** edmondsw has quit IRC | 13:50 | |
| *** jcoufal has joined #openstack-infra | 13:51 | |
| *** edmondsw has joined #openstack-infra | 13:51 | |
| *** arxcruz is now known as arxcruz|brb | 13:52 | |
| *** d0ugal has quit IRC | 13:55 | |
| *** edmondsw has quit IRC | 13:55 | |
| *** edmondsw has joined #openstack-infra | 13:59 | |
| *** agopi_ has joined #openstack-infra | 13:59 | |
| *** dhajare has quit IRC | 14:00 | |
| *** edmondsw has quit IRC | 14:01 | |
| *** edmondsw has joined #openstack-infra | 14:01 | |
| *** dtantsur|brb is now known as dtantsur | 14:07 | |
| *** dave-mccowan has joined #openstack-infra | 14:07 | |
| *** sree has joined #openstack-infra | 14:13 | |
| *** sshnaidm is now known as sshnaidm|rover | 14:13 | |
| *** jamesmcarthur has joined #openstack-infra | 14:14 | |
| pabelanger | AJaeger: sure | 14:15 |
| *** sree has quit IRC | 14:17 | |
| *** coolsvap has joined #openstack-infra | 14:21 | |
| *** ihrachys has joined #openstack-infra | 14:23 | |
| *** dhill_ has joined #openstack-infra | 14:23 | |
| dhellmann | I'm seeing an error I don't understand during one of the release jobs. http://logs.openstack.org/4e/4e510b15c1b13fde307039e7472ed02a421856d1/release/release-openstack-python/2725b29/ara/result/d18ad8b7-6265-46f4-a054-c556df69f76d/ | 14:28 |
| pabelanger | looking | 14:29 |
| dhellmann | something to do with the way git is managing the local repo? | 14:29 |
| *** d0ugal has joined #openstack-infra | 14:29 | |
| pabelanger | dhellmann: we did merge some changes into zuul-mergers over the weekend | 14:29 |
| *** david-lyle has quit IRC | 14:30 | |
| *** lucasagomes is now known as lucas-hungry | 14:30 | |
| dhellmann | pabelanger : my mental model of how the git repos are managed is pretty simple, and likely to be inaccurate. I don't know what "pushed to a non-bare repo" means here | 14:31 |
| *** dklyle has quit IRC | 14:32 | |
| pabelanger | git checkout branch. | 14:32 |
| pabelanger | doesn't look right | 14:32 |
| pabelanger | that is for 'openstack/murano-agent' | 14:33 |
| *** shardy has joined #openstack-infra | 14:33 | |
| AJaeger | tobiash: were you involved in the updated git changes for zuul ? ^ | 14:34 |
| AJaeger | does this ring a bell? | 14:34 |
| *** Goneri has joined #openstack-infra | 14:34 | |
| *** claudiub|2 has joined #openstack-infra | 14:34 | |
| *** efoley has joined #openstack-infra | 14:34 | |
| pabelanger | dhellmann: I'm not too familiar with this logic myself, I think corvus was the one who originally created the role. But is related to how we now mirror repos to the node | 14:34 |
| tobiash | AJaeger: there was a change which removed zuul refs from the mergers | 14:35 |
| pabelanger | dhellmann: I want to say there is a syntax issue with the shell command, but unsure why git command isn't correct | 14:36 |
| *** claudiub has quit IRC | 14:37 | |
| pabelanger | dhellmann: http://logs.openstack.org/4e/4e510b15c1b13fde307039e7472ed02a421856d1/release/release-openstack-python/2725b29/ara/result/8c444c0d-c39c-47bd-80ef-f2ad32933fa3/ | 14:37 |
| pabelanger | dhellmann: echo 'git.openstack.org/openstack/murano-agent:' $(git status | head -1 |awk '{ print $NF }') appears to be the issue | 14:37 |
| *** hrw has quit IRC | 14:37 | |
| tobiash | it's doing 'git checkout branch.' | 14:38 |
| pabelanger | yah, appears to be the issue, but unsure why | 14:39 |
| *** dave-mcc_ has joined #openstack-infra | 14:39 | |
| *** mriedem has joined #openstack-infra | 14:39 | |
| dhellmann | perhaps the git status output is not what we expect it to be | 14:40 |
| dhellmann | is there some way to see what that output is producing? | 14:40 |
| *** hrw has joined #openstack-infra | 14:40 | |
| pabelanger | I'm trying locally to see myself | 14:41 |
| *** gongysh has joined #openstack-infra | 14:41 | |
| *** kgiusti has joined #openstack-infra | 14:41 | |
| *** dave-mccowan has quit IRC | 14:42 | |
| dhellmann | pabelanger : the first line of "git status" when I run it on a non-bare repo is "On branch master" | 14:44 |
| dhellmann | I don't see "branch." with the "." anywhere in that output | 14:44 |
| dhellmann | but I'm sure my repo is not set up like the one on this node because I've just done a git clone | 14:44 |
| mordred | dhellmann: "checkout": "3.4.0", | 14:45 |
| mordred | dhellmann: search for "key": "git.openstack.org/openstack/murano-agent", | 14:45 |
| *** rosmaita has joined #openstack-infra | 14:45 | |
| mordred | and in that dict, see that the ref in question for murano-agent to check out is 3.4.0 | 14:45 |
| *** jamesmcarthur has quit IRC | 14:45 | |
| *** jamesmcarthur has joined #openstack-infra | 14:46 | |
| dhellmann | "HEAD detached at 3.4.0 | 14:46 |
| dhellmann | nothing to commit, working directory clean" | 14:46 |
| *** gcb has joined #openstack-infra | 14:46 | |
| dhellmann | still no "branch." in that output | 14:46 |
| tobiash | this command is probably there to check which branch is checked out? | 14:46 |
| AJaeger | git status |head -1|awk '{ print $NF}' shows '3.4.0' | 14:47 |
| dhellmann | oh, wait, it's horizon | 14:47 |
| mordred | it is? | 14:47 |
| tobiash | maybe we shouldn't try to parse 'git status'? | 14:47 |
| tobiash | what about 'git rev-parse --abbrev-ref HEAD'? | 14:47 |
| tobiash | would that work? | 14:47 |
| dhellmann | mordred : the command run at the bottom of http://logs.openstack.org/4e/4e510b15c1b13fde307039e7472ed02a421856d1/release/release-openstack-python/2725b29/ara/result/8c444c0d-c39c-47bd-80ef-f2ad32933fa3/ is echo 'git.openstack.org/openstack/horizon:' $(git status | head -1 |awk '{ print $NF }') | 14:47 |
| dhellmann | what is that command trying to do? | 14:48 |
| mordred | tobiash: IIRC we tried git rev-parse --abbrev-ref HEAD and that didn't work for some cases for some reason | 14:48 |
| dhellmann | tobiash : yeah, I've found parsing the porcelain output of git to be unstable. That's why I converted reno to use dulwich. | 14:48 |
| AJaeger | dhellmann: 'git.openstack.org/openstack/murano-agent: branch.' is at the end of the link | 14:48 |
| pabelanger | http://paste.openstack.org/show/662495/ are logs from zuul-executor, might help | 14:48 |
| dhellmann | AJaeger : oops, I didn't scroll far enough | 14:49 |
| tobiash | the repo returning branch. is murano-agent: http://logs.openstack.org/4e/4e510b15c1b13fde307039e7472ed02a421856d1/release/release-openstack-python/2725b29/job-output.txt.gz#_2018-02-05_14_10_10_274254 | 14:49 |
| *** gongysh has quit IRC | 14:49 | |
| mordred | tobiash: the thing we're trying to do there is determine the local heads that exist on the executor so that we can reset them properly on the remote side after pushing the refs | 14:49 |
| pabelanger | since tobiash and mordred seem to have a handle what is going on, I'm going to bow out for now. But learn :) Don't fully understand the logic myself | 14:50 |
| mordred | well - I don't know that I havea handle on anything yet - I still don't understand where the word 'branch' is coming from | 14:50 |
| *** esberglu has joined #openstack-infra | 14:52 | |
| mordred | also, locally, echo $(git status | head -1 |awk '{ print $NF }') prints '3.4.0' in murano-agent after a git checkout 3.4.0 | 14:52 |
| mordred | which makes me even more confused | 14:52 |
| AJaeger | mordred: want to review my snafus' instead? I broke some jobs and https://review.openstack.org/540778 is the fix (and https://review.openstack.org/540602 and https://review.openstack.org/540603 are followups) | 14:52 |
| pabelanger | mordred: yah, that's what I also got | 14:52 |
| dhellmann | mordred : I see this message in the git source code: "Cannot setup tracking information; starting point '%s' is not a branch." | 14:53 |
| dhellmann | also these 2, but that one seems more likely to be what's going on ("Can't add a note on empty branch.", "Can't tag an empty branch.") | 14:54 |
| mordred | dhellmann: wow - you'd thikn that message would occur when the branch command was run | 14:54 |
| mordred | or the checkout, rather | 14:54 |
| *** dizquierdo has joined #openstack-infra | 14:55 | |
| dhellmann | it is in branch.c; let me see if I can figure out what operaiton | 14:55 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Blackhole messages to FBC owner alias on lists https://review.openstack.org/540876 | 14:55 |
| dhellmann | mordred : it does seem to be in the branch creation code | 14:55 |
| fungi | #status log temporarily added lists.openstack.org to the emergency maintenance list pending merger of https://review.openstack.org/540876 | 14:56 |
| openstackstatus | fungi: finished logging | 14:56 |
| *** armaan has quit IRC | 14:58 | |
| *** armaan has joined #openstack-infra | 14:59 | |
| dhellmann | mordred : maybe the thing to do here is to update that playbook to dump more debug info before this step and run the job again? | 14:59 |
| *** myoung|bbl is now known as myoung | 15:00 | |
| dhellmann | because reading the git source code to try to figure out what's going on seems like madness, or a path to it | 15:00 |
| mordred | dhellmann: yah - I agree. that logic runs on every job too - so I'm loathe to start changing things without actually understanding the situation | 15:02 |
| *** zeus has quit IRC | 15:02 | |
| mordred | dhellmann: probably just adding a shell step that runs the git status command by itself would be useful | 15:02 |
| *** hongbin has joined #openstack-infra | 15:02 | |
| dhellmann | if we're not seeing constant failures, maybe a node has a bad cache? we've had that happen before | 15:02 |
| dhellmann | right | 15:03 |
| *** r-daneel has joined #openstack-infra | 15:03 | |
| dhellmann | that's in zuul-jobs? | 15:03 |
| *** dsariel has quit IRC | 15:04 | |
| *** zeus has joined #openstack-infra | 15:05 | |
| mordred | dhellmann: yah - patch coming | 15:05 |
| *** zeus is now known as Guest60045 | 15:05 | |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/openstack-zuul-jobs master: Replace legacy tempest-dsvm-py35 https://review.openstack.org/539497 | 15:05 |
| dhellmann | mordred : if I propose it you can vote, hang on | 15:05 |
| mordred | dhellmann: good call | 15:05 |
| andreaf | AJaeger: rebased https://review.openstack.org/#/c/539497/ | 15:05 |
| *** olaph has joined #openstack-infra | 15:06 | |
| andreaf | tobiash: thanks for your comments on https://review.openstack.org/#/c/540485/ - I tried using blocks but it didn't help - the with_dict always complains even if the task is in a block that should be skipped... any idea? | 15:07 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/zuul-jobs master: add debug info to mirror-workspace-git-repos https://review.openstack.org/540880 | 15:07 |
| dhellmann | mordred : ^^ | 15:07 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Fix ansible-role-functional jobs https://review.openstack.org/540778 | 15:08 |
| AJaeger | andreaf: I'll try not to break it again ;) | 15:08 |
| mordred | dhellmann: lgtm ... tobiash ^^ ? | 15:09 |
| AJaeger | andreaf: +2 | 15:09 |
| *** oidgar has joined #openstack-infra | 15:09 | |
| andreaf | AJaeger thanks | 15:09 |
| AJaeger | config-core, next step in fixing ansible-role-functional, please review https://review.openstack.org/#/c/540602/ | 15:09 |
| mordred | andreaf: a when on a block in ansible gets expanded essentially by just adding the when condition from the block to each commadn in the block | 15:10 |
| pabelanger | mordred: dhellmann: left comment on 540880 with +3 | 15:11 |
| andreaf | mordred | 15:11 |
| andreaf | mordred: doh! | 15:11 |
| andreaf | mordred: ok so it only helps with readability really - thanks for the explanation | 15:11 |
| pabelanger | mordred: dhellmann: but more tricky since it would have to be a dict of results | 15:11 |
| dhellmann | pabelanger : oh, good thought. I assumed we would take the debug logic back out, rather than capture it and then try to parse it separately. | 15:12 |
| *** bobh has joined #openstack-infra | 15:12 | |
| tobiash | andreaf: you also could move that task into its own file and do a conditional include, that way ansible shouldn't even parse it | 15:14 |
| *** alexchadin has quit IRC | 15:14 | |
| mordred | andreaf: what you COULD try there is to use a larger jinja block - something like: {%- if extensions_to_txt | type_debug == 'list' %} {{ extensions_to_txt | join('|') | default('__do_not_replace__') }} {% else %} {% for item in extensions_to text ... | 15:14 |
| *** katkapilatova has joined #openstack-infra | 15:14 | |
| mordred | andreaf: or you could do what tobiash said, which would likely be more readable | 15:14 |
| *** shardy has quit IRC | 15:15 | |
| mordred | andreaf: but an example of using complex jinja in set_fact: http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/multi-node-firewall/tasks/main.yaml | 15:16 |
| *** dizquierdo has quit IRC | 15:16 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Redirect status.o.o/zuul to https https://review.openstack.org/540883 | 15:16 |
| andreaf | mordred, tobiash: well I just wanted to support list until three other patches are merged and then ditch it so maybe this is becoming a bit too much - the alternatives are I break file renaming for a short bit or I use a different variable name | 15:17 |
| *** dsariel has joined #openstack-infra | 15:19 | |
| AJaeger | mordred, tobiash, I agree with andreaf - https://review.openstack.org/#/c/540485/ is short lived. andreaf what do you think of pushing the followup change directly - with depends-on those three changes? That way we can discuss the final solution and might accept temporary not so perfect code | 15:19 |
| andreaf | AJaeger: uhm yeah ok that would be simpler, but setting a dict in those three changes might break POST - well I can try | 15:21 |
| AJaeger | andreaf: I'm not sure you understood my suggestion. I suggested to leave 540485 as is - and push a change stacked on top that removes the list handling. | 15:22 |
| *** lucas-hungry is now known as lucasagomes | 15:22 | |
| AJaeger | andreaf: so, show us the end result and we can discuss that instead argue about the intermediate step. | 15:23 |
| andreaf | AJaeger ah ok | 15:23 |
| smcginnis | dhellmann: Did we want to wait for that debugging patch, or trust that it was a transient problem and proceed? | 15:26 |
| openstackgerrit | Thierry Carrez proposed openstack/ptgbot master: Pre-allocate random colors https://review.openstack.org/540886 | 15:27 |
| dhellmann | smcginnis : I think I want to wait for the debugging patch and then try the murano-agent jobs again | 15:27 |
| *** dizquierdo has joined #openstack-infra | 15:28 | |
| smcginnis | dhellmann: I don't get the error on there. | 15:29 |
| dhellmann | smcginnis : I cannot reproduce it or understand it, either. | 15:29 |
| smcginnis | recheck? | 15:29 |
| dhellmann | wait, which are we talking about? | 15:29 |
| smcginnis | dhellmann: https://review.openstack.org/#/c/540880/ | 15:30 |
| dhellmann | oh, I hadn't seen that error yet | 15:30 |
| andreaf | mordred, tobiash, pabelanger: switching py3 leagcy integration gate to the new zuulv3 native one - https://review.openstack.org/#/c/539497/ | 15:31 |
| dhellmann | yeah, let's see what a recheck does | 15:31 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Increase test timeout https://review.openstack.org/540889 | 15:31 |
| openstackgerrit | Andrea Frittoli proposed openstack-infra/zuul-jobs master: Remove support for extensions as lists https://review.openstack.org/540890 | 15:32 |
| *** armax has quit IRC | 15:32 | |
| *** caphrim007 has quit IRC | 15:33 | |
| andreaf | AJaeger ^^^ | 15:33 |
| *** armax has joined #openstack-infra | 15:33 | |
| AJaeger | tobiash, mordred ^ | 15:33 |
| * AJaeger needs to step out for a bit... | 15:33 | |
| *** yujunz has quit IRC | 15:34 | |
| *** yujunz has joined #openstack-infra | 15:34 | |
| *** gagehugo has joined #openstack-infra | 15:39 | |
| tobiash | andreaf: if you want to ditch it afterwards, just leave the ignore_errors | 15:39 |
| tobiash | I was reviewing from a long term maintenance view | 15:40 |
| tobiash | andreaf: or with the depends on in your latest ps ;) | 15:41 |
| *** david-lyle has joined #openstack-infra | 15:42 | |
| *** agopi__ has joined #openstack-infra | 15:42 | |
| *** lbragstad_ is now known as lbragstad | 15:43 | |
| *** slaweq has quit IRC | 15:43 | |
| *** slaweq has joined #openstack-infra | 15:43 | |
| *** slaweq has quit IRC | 15:43 | |
| *** HeOS has joined #openstack-infra | 15:44 | |
| *** agopi_ has quit IRC | 15:44 | |
| *** gcb has quit IRC | 15:47 | |
| *** ramishra has quit IRC | 15:48 | |
| *** armaan has quit IRC | 15:49 | |
| *** jcoufal has quit IRC | 15:50 | |
| *** xarses has joined #openstack-infra | 15:55 | |
| *** askb has quit IRC | 15:55 | |
| *** xarses_ has joined #openstack-infra | 15:56 | |
| *** sree has joined #openstack-infra | 15:57 | |
| *** ijw has joined #openstack-infra | 15:58 | |
| *** jcoufal has joined #openstack-infra | 15:59 | |
| *** xarses has quit IRC | 15:59 | |
| *** efoley_ has joined #openstack-infra | 16:00 | |
| *** efoley has quit IRC | 16:00 | |
| *** sree has quit IRC | 16:02 | |
| *** katkapilatova has quit IRC | 16:04 | |
| *** agopi__ has quit IRC | 16:04 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: add debug info to mirror-workspace-git-repos https://review.openstack.org/540880 | 16:06 |
| *** florianf has quit IRC | 16:07 | |
| *** armax has quit IRC | 16:07 | |
| *** florianf has joined #openstack-infra | 16:08 | |
| *** armax has joined #openstack-infra | 16:09 | |
| *** askb has joined #openstack-infra | 16:10 | |
| dhellmann | mordred, smcginnis , fungi : now that https://review.openstack.org/#/c/540880/ has merged can we re-try the murano-agent job(s) that failed? | 16:11 |
| fungi | dhellmann: i missed the murano-agent discussion but will try to catch up after my current meeting | 16:12 |
| dhellmann | fungi : ack. I have notes in https://etherpad.openstack.org/p/0HI0LZLSvt | 16:12 |
| *** eumel8 has quit IRC | 16:13 | |
| fungi | thanks! | 16:13 |
| corvus | dhellmann: i can run the cmd | 16:13 |
| corvus | fungi: ^ i'll take it, ok? | 16:13 |
| *** armax has quit IRC | 16:13 | |
| dhellmann | corvus : I used our script to generate some zuul commands and pasted them at the bottom of that etherpad | 16:14 |
| *** armax has joined #openstack-infra | 16:14 | |
| corvus | dhellmann: should we just start with just one? which? | 16:14 |
| fungi | corvus: appreciated | 16:14 |
| *** armax has left #openstack-infra | 16:14 | |
| *** slaweq has joined #openstack-infra | 16:14 | |
| dhellmann | corvus : we only had an error for the release queue job, afaict, so maybe just that one? | 16:14 |
| dhellmann | line 11 | 16:14 |
| corvus | dhellmann: done | 16:14 |
| dhellmann | corvus : thanks | 16:15 |
| *** camunoz has joined #openstack-infra | 16:15 | |
| corvus | i'm resuming uploads of zuulv3-project changes | 16:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Add unit test for multiple launchers https://review.openstack.org/540916 | 16:16 |
| corvus | dhellmann: http://zuul.openstack.org/stream.html?uuid=9a217084748b4a4e839135d9aef60a0e&logfile=console.log | 16:16 |
| *** armax has joined #openstack-infra | 16:16 | |
| smcginnis | I saw an error scroll past. | 16:17 |
| dhellmann | murano-agent is showing "not currently on any branch" as part of the git status output | 16:17 |
| dhellmann | is something assuming that every ref being checked out will be on a branch? does that hold true when we check out a tag? | 16:18 |
| *** sree has joined #openstack-infra | 16:18 | |
| *** slaweq has quit IRC | 16:18 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Add unit test for multiple launchers https://review.openstack.org/540916 | 16:19 |
| dhellmann | and which version of git is being used? I don't get that message with git 2.7.4 | 16:19 |
| mordred | dhellmann, corvus: my local git is 2.15.1 - and I also don't get that error when having checked out a tag | 16:19 |
| dhellmann | I do see that message in one of the message catalogs in the git source tree | 16:20 |
| mordred | dhellmann: that string certainly does sound like a string that would cause the 'branch' issue we were seeing | 16:20 |
| dhellmann | yep | 16:20 |
| *** jcoufal has quit IRC | 16:21 | |
| dhellmann | that step in the playbook "Determine local HEAD of projects" hasn't changed for a while | 16:22 |
| dhellmann | it's not really clear what it's trying to do though | 16:22 |
| mordred | dhellmann: zuul prepares the git repos on the executor | 16:23 |
| dhellmann | "prepares"? | 16:23 |
| mordred | dhellmann: yah - it does all of the speculative merging and then checks out the branches into the appropriate state for testing | 16:23 |
| dhellmann | ok | 16:23 |
| mordred | dhellmann: but then those git repo states need to be pushed to the remote nodes where the tests run | 16:23 |
| *** sree has quit IRC | 16:23 | |
| mordred | the 'determine local head' step is looking at the state of the git repos on disk to find the head states so it can set those head states in the remote git repos on the test nodes | 16:24 |
| dhellmann | what "state" is that? the sha of the commit? | 16:24 |
| dhellmann | or the branch name? | 16:25 |
| mordred | dhellmann: in this case, branch name - it needs to know which branch to check out in the repo on the remote node | 16:25 |
| *** jcoufal has joined #openstack-infra | 16:26 | |
| dhellmann | so maybe : git branch --contains | grep -v detached | 16:26 |
| mordred | dhellmann: so that src/git.openstack.org/openstack/murano-agent has a filesystem tree in the correct state for the test | 16:26 |
| dhellmann | although that command can return multiple names for some commits, when we create branches | 16:26 |
| *** dizquierdo has quit IRC | 16:27 | |
| dhellmann | the only changes this year in that zuul role main.yaml are from me in that debugging patch, so I guess something else somewhere else changed? | 16:27 |
| mordred | yah - I'm guessing a git version bump or something | 16:28 |
| dhellmann | pabelanger mentioned other changes to git handling | 16:28 |
| corvus | several things in the zuul merger changed recently | 16:28 |
| *** oidgar has quit IRC | 16:29 | |
| dhellmann | so for a tag like this, what would the expected output of that git status parsing be? the tag itself? | 16:32 |
| mordred | dhellmann: yes. | 16:35 |
| corvus | yeah, i *think* we expect 'HEAD detached at 3.4.0' in that case | 16:35 |
| dhellmann | yeah, that's what I'm getting locally | 16:35 |
| mordred | yes - which is what I get locally too | 16:35 |
| dhellmann | like I said, I abandoned parsing git porcelain output because it changes in surprising ways, so I'm wondering if we need a different way to determine the right state value for this case | 16:36 |
| dhellmann | but it's not clear what all the possible expected values are | 16:36 |
| dhellmann | just branch names or tag names? or sometimes a sha? | 16:36 |
| *** yamamoto has quit IRC | 16:36 | |
| *** andreas_s has quit IRC | 16:36 | |
| corvus | (i'm going to try to recreate what would be in the executor's work dir... if i fail, i'll re-run that job with --keep) | 16:36 |
| dhellmann | ok | 16:37 |
| *** andreas_s has joined #openstack-infra | 16:37 | |
| *** links has quit IRC | 16:37 | |
| dhellmann | corvus : are the commands to reproduce the work dir in the log? like probably as a bunch of individual steps right now but is that how you're reproducing it? or is there some shortcut? | 16:37 |
| *** vhosakot has joined #openstack-infra | 16:38 | |
| tbarron | do we have python3 support in the infra CentOS mirrors now? (so no EPEL needed) | 16:38 |
| corvus | dhellmann: there is not a sequence of git commands to reproduce it, zuul uses gitpython to create the repo | 16:38 |
| dhellmann | corvus : ah. | 16:38 |
| tbarron | last time we looked at migrating our CentOS gate jobs to python3 I think that was an issue | 16:39 |
| dhellmann | having a (zuul?) command to reproduce the workspace would be useful | 16:39 |
| corvus | dhellmann: yes there's a story about it. afaik no one is working on it | 16:40 |
| corvus | nevermind, i'm just going to run with keep | 16:40 |
| dhellmann | yeah, we have that problem in several areas of the community these days :-) | 16:40 |
| clarkb | tbarron: I'm not sure I understand the question, but we mirror upstream centos 7 so if upstream centos7 has python3 packages in its repos we should too | 16:40 |
| tbarron | clarkb: I think pure upstream centos 7 does not have python3 yet but one can get it via EPEL repos | 16:41 |
| clarkb | tbarron: we mirror epel too | 16:41 |
| *** claudiub has joined #openstack-infra | 16:41 | |
| tbarron | clarkb: ok, if epel is mirrored then I think we're OK, iirc there was a time when we were | 16:42 |
| *** dizquierdo has joined #openstack-infra | 16:42 | |
| tbarron | clarkb: looking at migrating our check/gate jobs to python3 and lack of CentOS support was an issue | 16:42 |
| pabelanger | tbarron: I've just been testing on fedora-26 nodes for python3 | 16:43 |
| tbarron | clarkb: maybe that was a mis-understanding, in any case it seems not to be a current blocker | 16:43 |
| corvus | http://zuul.openstack.org/stream.html?uuid=daac5cd5aacc4b86917bd7b46fc16eb5&logfile=console.log is next run | 16:43 |
| clarkb | tbarron: thats more of an upstream issue than an infra issue aiui. We will happily mirror upstraem centos and epel and if that includes python3 then you are set | 16:43 |
| pabelanger | you can try software collections for python3, if you want | 16:43 |
| pabelanger | but not all libraries are packaged | 16:43 |
| tbarron | pabelanger: yeah, me too, but the jobs are CentOS and there's no will to change them to Fedora | 16:43 |
| *** pcichy has joined #openstack-infra | 16:44 | |
| corvus | hey this is pretty cool -- we have an executor backlog! | 16:44 |
| mordred | corvus: neat! | 16:44 |
| corvus | (this is an improvement on executors overloading themselves) | 16:44 |
| *** claudiub|2 has quit IRC | 16:45 | |
| corvus | there are currently 32 jobs waiting to run on executors | 16:45 |
| tbarron | clarkb: thanks! | 16:45 |
| pabelanger | tbarron: I know dhellmann suggested it might be a topic at PTG to update devstack to python3 only. But the effort for centos and python3 likey won't happed IMO, too much work. Fedora26+ would be my suggestions | 16:45 |
| dhellmann | python3 first, not only | 16:45 |
| tbarron | dhellmann ack | 16:46 |
| mordred | pabelanger, tbarron: I think it would be great if someone put together a 'this is what python3 support on centos looks like' story | 16:46 |
| *** andreas_s has quit IRC | 16:46 | |
| dhellmann | I know I'm confused about that, but maybe that's because I keep seeing conflicting email threads about it internally | 16:46 |
| mordred | and then we can figure out how best to support that story with infrastructure | 16:46 |
| mordred | yah | 16:46 |
| mordred | basically there does not seem to be a definitive answer, which makes supporting it in infra essentially impossible | 16:47 |
| tbarron | pabelanger: mordred probably I'll bring this up in TripleO since they depend so heavily on CentOS and | 16:47 |
| mordred | tbarron: ++ | 16:47 |
| mordred | tbarron: please do - I think it would benefit many people to have such a story sorted out | 16:47 |
| pabelanger | agree, story would be nice, but everytime I looked into it (internally) software collections was the only way. However, that is missing a lot of python packages and also has different RPM spec format, IIRC | 16:47 |
| tbarron | it seems to me that real deployments runninig with python3 like kolla and tripelo | 16:47 |
| *** shardy has joined #openstack-infra | 16:48 | |
| tbarron | are where the rubber really meets the road | 16:48 |
| *** e0ne has quit IRC | 16:48 | |
| pabelanger | since then, I've just switched to fedora and holding out for centos8 | 16:48 |
| tbarron | we can run everything in gate on xenial and fedora but ... | 16:48 |
| tbarron | if there is a year or so runway to move to python3 ... | 16:49 |
| *** yamamoto has joined #openstack-infra | 16:49 | |
| *** yamamoto has quit IRC | 16:49 | |
| smcginnis | 2020 is coming up quick. | 16:50 |
| *** yamamoto has joined #openstack-infra | 16:50 | |
| *** yamamoto has quit IRC | 16:50 | |
| *** edmondsw has quit IRC | 16:50 | |
| fungi | 2020 is end of _upstream_ support for 2.7.x... not end of distro support for it | 16:52 |
| fungi | granted, would be nice to already support py3k everywhere in two years | 16:52 |
| smcginnis | Still.. | 16:52 |
| fungi | seems achievable | 16:52 |
| *** gfidente is now known as gfidente|not | 16:52 | |
| mtreinish | fungi: well unless they push it back again :p | 16:54 |
| mordred | it would honetly be nice to SWITCH to py3k everywhere by then from an openstack perspective | 16:54 |
| corvus | dhellmann, mordred: if i take the repo i got from the work dir, and 'git checkout master; git checkout 3.4.0' i get what's expected. so i think something that bears on the issue is perhaps that this repo has never checked out a branch before. like, i don't know how get decides if it's on a branch, but presumably, once you've checked out a branch, and then checkout a tag, that changes something | 16:54 |
| corvus | that 'git status' looks at. | 16:54 |
| *** kjackal has quit IRC | 16:54 | |
| *** panda is now known as panda|bbl | 16:55 | |
| mordred | corvus: I thought we had the merger check out the target branch/ref | 16:55 |
| dhellmann | interesting | 16:55 |
| *** gyee has joined #openstack-infra | 16:55 | |
| mordred | corvus: so that the repo would be in the post-branch-checkout state | 16:55 |
| mordred | corvus: oh - wait | 16:55 |
| corvus | mordred: we have it checkout the tag, which it does | 16:55 |
| mordred | corvus: I grok what you're syaing | 16:55 |
| dhellmann | this job is triggered by the tag, does that change the input assumptions? | 16:55 |
| *** edmondsw has joined #openstack-infra | 16:56 | |
| corvus | but... and i'm going to be vague here because i'm delirious... something about checking out a branch is sticky. so checking out a branch, or not, followed by checking out a tag, somehow make the repo different. | 16:56 |
| dhellmann | it feels hacky to have it check out master then the tag just to set the repo status | 16:56 |
| mordred | I agree with both of you | 16:56 |
| corvus | i'm going to try to figure out what specifically is different | 16:56 |
| mordred | corvus: ++ | 16:57 |
| dhellmann | is the state we're trying to capture in this step ever going to be anything other than a tag or a branch name? | 16:57 |
| *** katkapilatova has joined #openstack-infra | 16:58 | |
| mordred | corvus: also - we have the zuul.projects variable now - I wonder if we can't just use it instead of inspecting repo state to determine what to check out on the remote node | 16:58 |
| *** gfidente has joined #openstack-infra | 16:58 | |
| fungi | also not necessarily just master. a stable point release tag would usually reference a commit which only exists on a stable branch | 16:58 |
| mordred | corvus: IIRC when we wrote this sync logic originally zuul.projects didnt' exist yet? | 16:58 |
| mordred | fungi: correct. and also, some repos don't have a master branch | 16:58 |
| dhellmann | fungi : a tag at a point where we have a branch will show up on both, as well | 16:58 |
| fungi | yep | 16:59 |
| dhellmann | so if we just need to know what commit head is, we can do that. If we're trying to track head on a branch if it is on *1* branch we can do that. I'm worried the current step isn't doing either in a "safe" way | 16:59 |
| *** edmondsw has quit IRC | 17:00 | |
| dhellmann | mordred : oh, if we already have this information somewhere else, why are we trying to recompute it at all? | 17:00 |
| mordred | dhellmann: well - we didn't used to have the information somewhere else | 17:00 |
| corvus | mordred: i feel like zuul.projects is an avenue worth exploring | 17:00 |
| dhellmann | ah, sure | 17:00 |
| *** kopecmartin has quit IRC | 17:00 | |
| fungi | makes sense to me | 17:00 |
| *** pbourke has quit IRC | 17:00 | |
| mordred | corvus: k. I'll take a quick stab at the zuul.projects avenue | 17:00 |
| *** caphrim007 has joined #openstack-infra | 17:01 | |
| corvus | mordred: i think we need to look at the scheduler source and make sure that it matches in all cases | 17:01 |
| dhellmann | do we just want to save item.value['checkout']? | 17:02 |
| *** caphrim007 has quit IRC | 17:02 | |
| *** caphrim007_ has joined #openstack-infra | 17:02 | |
| *** pbourke has joined #openstack-infra | 17:02 | |
| corvus | the thing that determines whether 'git status' outputs 'not on any branch' or 'detached head' is .git/logs/HEAD | 17:02 |
| corvus | i was not expecting that. | 17:02 |
| mordred | dhellmann: yah - we don't even need to save it - we can simplify this a lot ... | 17:03 |
| mordred | dhellmann: do you have the failing log link from earlier handy? | 17:03 |
| dhellmann | it's in https://etherpad.openstack.org/p/0HI0LZLSvt | 17:03 |
| dhellmann | I have to step away for a bit, but I'll be back | 17:04 |
| *** efoley_ has quit IRC | 17:06 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Use item.checkout from zuul.projects when mirroring https://review.openstack.org/540934 | 17:06 |
| mordred | dhellmann, corvus: ^^ | 17:06 |
| mordred | +1,-32 ... almost always seems like a win :) | 17:06 |
| smcginnis | ++ | 17:07 |
| *** caphrim007_ has quit IRC | 17:07 | |
| *** efoley has joined #openstack-infra | 17:07 | |
| *** slaweq has joined #openstack-infra | 17:08 | |
| corvus | mordred: that looks very nice. i've given the code a once-over and think it's correct. | 17:08 |
| *** dhajare has joined #openstack-infra | 17:09 | |
| *** trown is now known as trown|lunch | 17:09 | |
| corvus | mordred, dhellmann: we changed several things about the merger with the restart over the weekend, however, none of them are jumping out at me as being consistent with this. however, i did one other thing, which was to delete all of the cached git repos on all of the zuul hosts. that means zuul has been cloning new copies. it's possible that the old copies had branch checkouts, but the new | 17:10 |
| corvus | clones don't. | 17:10 |
| corvus | mordred, dhellmann: if *that* is the cause, then this was a lurking error which would hit new projects, but not old ones, and we just exposed it by clearing out the cache. | 17:10 |
| dhellmann | corvus, mordred : we had some similar issues with the git caches not quite being what we expected when we moved some of the tagging jobs onto dynamically created nodes | 17:12 |
| *** caphrim007 has joined #openstack-infra | 17:13 | |
| mordred | corvus: OH! you know what? the cached repos were originally pre-populated by us not by zuul because we pre-warmed the cache with normal git clones | 17:13 |
| corvus | mordred: yep | 17:13 |
| mordred | corvus: so I do think that, in fact, this was a lurking error | 17:13 |
| corvus | only nova is like that now ;) | 17:13 |
| *** dhill_ has quit IRC | 17:13 | |
| *** slaweq has quit IRC | 17:14 | |
| mordred | heh | 17:14 |
| corvus | in general, i think we're continuing to move in the direction of eliminating assumptions and becoming more robust. | 17:14 |
| mordred | ++ | 17:14 |
| *** dhill_ has joined #openstack-infra | 17:14 | |
| mordred | corvus: how do we want to test that change? | 17:15 |
| *** d0ugal has quit IRC | 17:15 | |
| *** slaweq has joined #openstack-infra | 17:15 | |
| mordred | corvus: I think the git mirroring is covered in the zuul-jobs functional tests, right? | 17:15 |
| corvus | dhellmann, smcginnis, mordred: mordred's fix will make the repos a more faithful copy of what's on the executor, which is now a repo with no branch checkout history, so we may need to keep an eye out for errors further down the line in jobs that operate on tags. | 17:15 |
| corvus | mordred: yeah i think so, we should examine its output closely | 17:16 |
| mordred | ++ | 17:16 |
| *** caphrim007 has quit IRC | 17:18 | |
| *** kjackal has joined #openstack-infra | 17:20 | |
| *** dsariel has quit IRC | 17:20 | |
| *** d0ugal has joined #openstack-infra | 17:23 | |
| *** dougwig has joined #openstack-infra | 17:24 | |
| fungi | do we want to schedule that approval for a specific time or is now good? | 17:24 |
| corvus | let's wait to examine the check output before approving | 17:25 |
| fungi | well, at least that yes | 17:25 |
| fungi | but also could be _massively_ disruptive and potentially require bypassing ci to merge any revert | 17:25 |
| corvus | fungi: yeah, though the problem is solves is pretty disruptive too. so i'd be inclined to merge it asap and watch. | 17:26 |
| fungi | agreed | 17:26 |
| smcginnis | If we were to wait and schedule it, it would need to be scheduled realtively soon anyway. | 17:26 |
| fungi | i'm around all day, so any time is good for me | 17:26 |
| fungi | including asap | 17:26 |
| corvus | running http://zuul.openstack.org/stream.html?uuid=ff4f2569400540f0a48ffc52bb7005cf&logfile=console.log | 17:27 |
| *** oidgar has joined #openstack-infra | 17:27 | |
| *** arxcruz|brb is now known as arxcruz | 17:28 | |
| fungi | HEAD is now at ce1bf1f Use getfqdn instead of gethostname | 17:28 |
| corvus | oh, er, i don't think it speculatively ran that, because it's in a trusted pre playbook | 17:28 |
| fungi | ahh, yep | 17:28 |
| corvus | i think if we want to pre-merge test that, we'll need to do the base-test dance | 17:28 |
| *** slaweq has quit IRC | 17:28 | |
| corvus | (copy the role, use the copy in base-test, then promote to prod) | 17:29 |
| corvus | i'll get started on that. | 17:30 |
| fungi | right, looks like the checkout of zuul-jobs output "Already on 'master'" | 17:30 |
| fungi | so i agree it doesn't seem to have checked out that change | 17:30 |
| *** shardy has quit IRC | 17:31 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Test Use item.checkout from zuul.projects when mirroring https://review.openstack.org/540945 | 17:32 |
| corvus | that can be merged immediately | 17:32 |
| *** jpich has quit IRC | 17:35 | |
| corvus | did base_serial get merged without going through base-test? | 17:35 |
| johnsom | Question, I noticed that the nova-specs repo on github is out of sync with git.openstack.org. Is that expected? | 17:36 |
| johnsom | https://git.openstack.org/cgit/openstack/nova-specs/tree/specs/queens/approved vs https://github.com/openstack/nova-specs/tree/master/specs/queens/approved | 17:37 |
| corvus | oh, no... it just hasn't been migrated to base yet | 17:37 |
| pabelanger | corvus: oh, we can revert that. It doesn't actually work as expected | 17:37 |
| fungi | johnsom: yes, it's corrupt and probably needs an offline git fsck before it will update correctly on gh | 17:37 |
| pabelanger | corvus: I can push up patch now if you'd like | 17:37 |
| corvus | pabelanger: no i'll do it | 17:37 |
| johnsom | fungi Ok, thanks | 17:37 |
| pabelanger | kk | 17:37 |
| corvus | i need to untangle it from the thing i'm doing | 17:37 |
| fungi | johnsom: we simply haven't had an appropriate outage where we remembered to try fixing it | 17:38 |
| *** slaweq has joined #openstack-infra | 17:38 | |
| corvus | pabelanger: you have another half-completed thing in there with fetch-zuul-cloner, what's the deal with that? | 17:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Resync base-test to base https://review.openstack.org/540947 | 17:40 |
| corvus | that can be approved immediately ^ | 17:40 |
| *** armax has quit IRC | 17:41 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: base-test: test new mirror-workspace role https://review.openstack.org/540948 | 17:42 |
| pabelanger | corvus: that was https://review.openstack.org/513506/ but we are currently -2. Maybe we should revert from base-test for now? | 17:42 |
| corvus | pabelanger: that's what i did above | 17:43 |
| *** slaweq has quit IRC | 17:43 | |
| *** olaph1 has joined #openstack-infra | 17:43 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Use item.checkout from zuul.projects when mirroring https://review.openstack.org/540934 | 17:43 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config master: Revert "base-test: test new mirror-workspace role" https://review.openstack.org/540949 | 17:44 |
| *** sambetts is now known as sambetts|afk | 17:45 | |
| *** olaph has quit IRC | 17:45 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Revert "Test Use item.checkout from zuul.projects when mirroring" https://review.openstack.org/540950 | 17:45 |
| *** dizquierdo has quit IRC | 17:47 | |
| *** suhdood has joined #openstack-infra | 17:47 | |
| *** hashar is now known as hasharAway | 17:47 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: DNM: Test new mirror-workspace-git-repos role https://review.openstack.org/540952 | 17:49 |
| corvus | pabelanger, fungi, mordred, dhellmann: i think i've staged the whole test + cleanup for the role update under the topic zuul-checkout. | 17:50 |
| mordred | awesome. looking now | 17:50 |
| *** gfidente|not has quit IRC | 17:50 | |
| corvus | pabelanger, fungi, mordred, dhellmann: https://review.openstack.org/540952 is the change we need to get to in order to evaluate. i think i got the depends-ons right | 17:50 |
| *** yamamoto has joined #openstack-infra | 17:50 | |
| corvus | nope got one of those wrong :) | 17:50 |
| corvus | oh, no i think it's right. nm. | 17:51 |
| mordred | https://review.openstack.org/#/c/540945/ adds test role - can be safely landed | 17:51 |
| *** gfidente|not has joined #openstack-infra | 17:52 | |
| mordred | https://review.openstack.org/#/c/540948/ depends on it and updates base-test to use it - can be safely landed now | 17:52 |
| *** sshnaidm|rover is now known as sshnaidm|bbl | 17:52 | |
| mordred | then we wait for it to land | 17:52 |
| *** dtantsur is now known as dtantsur|afk | 17:53 | |
| *** e0ne has joined #openstack-infra | 17:53 | |
| *** caphrim007 has joined #openstack-infra | 17:53 | |
| mordred | and recheck https://review.openstack.org/#/c/540952/ | 17:53 |
| *** olaph has joined #openstack-infra | 17:54 | |
| dhellmann | this feels like a round-about way to test job changes. is that number of patches normal? I guess it's because some of the repos are flagged so that speculative changes aren't used when building up the jobs? | 17:54 |
| fungi | correct | 17:54 |
| fungi | we basically have to try them out using shadow copies of the trusted playbooks | 17:55 |
| * dhellmann nods | 17:55 | |
| *** efoley has quit IRC | 17:55 | |
| fungi | which yes does seem convoluted, but is still vastly better than under zuul v2 where we basically couldn't perform speculative testing of job configuration changes at all | 17:55 |
| *** olaph1 has quit IRC | 17:55 | |
| dhellmann | true | 17:56 |
| corvus | very few things need this kind of testing, and they don't change very often | 17:56 |
| fungi | and the idea here is that we don't touch these parts of the base configuration often, and with decreasing frequency over time as zuul v3 stabilizes | 17:56 |
| fungi | yeah, what corvus said | 17:56 |
| dhellmann | that also makes sense | 17:56 |
| *** caphrim007 has quit IRC | 17:57 | |
| AJaeger | config-core, could you review https://review.openstack.org/#/c/540602/ and https://review.openstack.org/#/c/540603/ to finish conversion of ansible-role jobs, please? | 17:58 |
| dhellmann | so how do we trigger the release job to see if the clone command worked properly with all of these patches in place? | 17:58 |
| mnaser | AJaeger: +2 first patch, will +2+w once we get it merged :> | 17:59 |
| AJaeger | mnaser: thanks - just +2 the second one and I can +W myself after recheck | 17:59 |
| mordred | dhellmann: I think this is mainly to test that this cahnge in logic doesn't break the other stuff | 17:59 |
| corvus | dhellmann: we'll have to do it after it's landed for real i think. under the assumption that if ... what mordred just said. | 17:59 |
| dhellmann | ah, ok | 17:59 |
| mordred | dhellmann: it can't possibly break the release jobs more than they are already broken | 17:59 |
| mnaser | AJaeger: deal :) | 17:59 |
| dhellmann | mordred : never say never | 17:59 |
| mordred | dhellmann: very good point | 18:00 |
| smcginnis | :) | 18:00 |
| fungi | the odds of it breaking the release jobs more than they currently are broken are low | 18:00 |
| mordred | yes | 18:00 |
| *** efoley has joined #openstack-infra | 18:00 | |
| *** derekh has quit IRC | 18:00 | |
| AJaeger | thanks, clarkb and mnaser for reviewing | 18:00 |
| mnaser | :) | 18:01 |
| *** david-lyle has quit IRC | 18:01 | |
| tobiash | corvus: would it make sense to move the repo setup stuff into its own untrusted pre playbook? | 18:05 |
| *** yamamoto has quit IRC | 18:06 | |
| tobiash | That way we could reduce the hard-to-test parts of zhe jobs | 18:06 |
| corvus | tobiash: we want it in a base job, and those can only be in config-projects | 18:07 |
| openstackgerrit | Merged openstack-infra/project-config master: Replace legacy ansible-role func jobs https://review.openstack.org/540602 | 18:07 |
| *** ralonsoh has quit IRC | 18:07 | |
| *** dprince has joined #openstack-infra | 18:07 | |
| tobiash | Well, base is just a default parent name. It also could be defined in a untrusted repo and have a trusted 'root' job which does the trusted setup parts | 18:09 |
| tobiash | That would cost one inhrritance layer per job but make essential parts easier testable | 18:10 |
| *** gfidente has quit IRC | 18:12 | |
| corvus | tobiash: true, but we do actually want it in a base job because we don't want people to be able to change this | 18:12 |
| *** trown|lunch is now known as trown | 18:12 | |
| *** d0ugal has quit IRC | 18:13 | |
| pabelanger | sorry, had to AFK due to internet install. | 18:13 |
| *** gfidente|not is now known as gfidente | 18:14 | |
| *** gfidente has quit IRC | 18:14 | |
| *** gfidente has joined #openstack-infra | 18:14 | |
| *** caphrim007 has joined #openstack-infra | 18:16 | |
| *** caphrim007 has quit IRC | 18:16 | |
| clarkb | re python2 EOL in 2020 every time I bring it up in other circles others are quick to point out to me that only Cpython2 is EOLing. Other python2 implementations apparnetly have no plans to EOL (pypy and jython in particular) | 18:16 |
| clarkb | now I don' | 18:16 |
| clarkb | er | 18:17 |
| *** caphrim007 has joined #openstack-infra | 18:17 | |
| clarkb | now I don't think we operate on either of those alternate implementation so its not super helpful for us but in the more general python conversation is important to remember | 18:17 |
| tobiash | Ah ok, so it's a policy decision to make also the repo handling mandatory | 18:17 |
| *** jpena is now known as jpena|off | 18:19 | |
| tosky | well, it's not an excuse to not migrate IMHO | 18:22 |
| *** dizquierdo has joined #openstack-infra | 18:22 | |
| *** olaph has quit IRC | 18:22 | |
| *** olaph has joined #openstack-infra | 18:23 | |
| clarkb | tosky: ya I'm firmly on the python3 is good for you camp (its much nicer as a language adn fixes a lot of warts we've lived with for a long time) | 18:23 |
| clarkb | I think mostly the pypy folks want to avoid confusion of their users thinking they must update right away too | 18:24 |
| dmsimard | RHEL7 will support py2 beyond EOL for a long while still :) | 18:24 |
| *** d0ugal has joined #openstack-infra | 18:26 | |
| openstackgerrit | Merged openstack-infra/project-config master: Resync base-test to base https://review.openstack.org/540947 | 18:27 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Test Use item.checkout from zuul.projects when mirroring https://review.openstack.org/540945 | 18:27 |
| AJaeger | team, converting the Zuul / nodepool integration test with https://review.openstack.org/#/c/540595/ I get a failure with "ln -s /tmp/nodepool/log $WORKSPACE/logs" - see http://logs.openstack.org/95/540595/4/experimental/nodepool-zuul-functional/5f79fc8/job-output.txt.gz#_2018-02-05_09_13_44_459281 . Any ideas how to fix? | 18:28 |
| *** tosky has quit IRC | 18:29 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/storyboard master: Add MQTT notification publisher https://review.openstack.org/538575 | 18:30 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/storyboard master: Add configurable notification subscriber and mqtt driver https://review.openstack.org/540958 | 18:30 |
| *** efoley has quit IRC | 18:30 | |
| *** Swami has joined #openstack-infra | 18:32 | |
| mtreinish | fungi: ^^^ you might want to take a look at that stack | 18:33 |
| *** dhajare has quit IRC | 18:34 | |
| *** olaph1 has joined #openstack-infra | 18:38 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Allow a few more starting builds https://review.openstack.org/540965 | 18:38 |
| *** myoung is now known as myoung|dr | 18:38 | |
| *** olaph has quit IRC | 18:38 | |
| dhellmann | clarkb : it's not just cpython, though. I anticipate a lot of library authors dropping support, too, regardless of what other interpreters maintain. | 18:39 |
| *** armaan has joined #openstack-infra | 18:42 | |
| *** olaph1 is now known as olaph | 18:42 | |
| corvus | zuul and nodepool are now py3 only; it was pretty easy for us to (intentionally) fall into that due to library support. | 18:42 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul master: Update nodepool-integration for Zuul v3 https://review.openstack.org/540967 | 18:44 |
| AJaeger | testing this now ^ | 18:44 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/nodepool master: Convert nodepool-zuul-functional job https://review.openstack.org/540595 | 18:45 |
| *** wolverineav has quit IRC | 18:45 | |
| *** wolverineav has joined #openstack-infra | 18:46 | |
| AJaeger | infra-root, here's a change for a Zuul v3 native job for infra-ansible, could I get a second +2, please? https://review.openstack.org/540596 | 18:50 |
| clarkb | AJaeger: done | 18:55 |
| AJaeger | thanks! | 18:57 |
| *** david-lyle has joined #openstack-infra | 18:57 | |
| openstackgerrit | Merged openstack-infra/project-config master: base-test: test new mirror-workspace role https://review.openstack.org/540948 | 18:57 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove obsolete ansible-func jobs https://review.openstack.org/540603 | 18:57 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove legacy infra-ansible jobs https://review.openstack.org/540970 | 18:59 |
| *** olaph has quit IRC | 18:59 | |
| *** olaph has joined #openstack-infra | 18:59 | |
| *** d0ugal has quit IRC | 18:59 | |
| fungi | okay, i've rechecked the "DNM: Test new mirror-workspace-git-repos role" (540952) change now that its deps are merged | 18:59 |
| *** pcichy has quit IRC | 19:00 | |
| fungi | will probably be a few minutes still before it gets nodes scheduled | 19:00 |
| fungi | i think the recheck comment is still in the events queue at the moment | 19:01 |
| openstackgerrit | Merged openstack-infra/zuul master: Increase test timeout https://review.openstack.org/540889 | 19:02 |
| mordred | fungi: ++ | 19:02 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy infra-ansible job https://review.openstack.org/540971 | 19:02 |
| *** dhajare has joined #openstack-infra | 19:03 | |
| *** harlowja has joined #openstack-infra | 19:03 | |
| *** coolsvap has quit IRC | 19:04 | |
| openstackgerrit | Merged openstack-infra/infra-ansible master: Import infra-ansible job https://review.openstack.org/540596 | 19:07 |
| fungi | hrm, i coulda sworn i saw jobs running for it a moment ago but now they're back to queued. doesn't bode well :( | 19:10 |
| fungi | expecting we'll see a retry_limit failure reported shortly on all of them | 19:10 |
| *** suhdood has quit IRC | 19:12 | |
| *** suhdood has joined #openstack-infra | 19:12 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Allow a few more starting builds https://review.openstack.org/540965 | 19:12 |
| AJaeger | first two jobs running | 19:12 |
| fungi | yeah | 19:13 |
| fungi | i was streaming one which just got aborted/requeued | 19:13 |
| * AJaeger captured trace | 19:14 | |
| AJaeger | fungi, mordred, corvus, http://paste.openstack.org/show/662570/ | 19:14 |
| AJaeger | I have the full trace if you need | 19:14 |
| *** tosky has joined #openstack-infra | 19:14 | |
| fungi | though build-sphinx-docs succeeded | 19:14 |
| *** amoralej is now known as amoralej|off | 19:14 | |
| corvus | fungi: i reparented unittests, it may not go through that path | 19:15 |
| AJaeger | fungi: tox-py35-zuul failed | 19:15 |
| clarkb | ianw: thank you for the review on https://review.openstack.org/#/c/535879/ one thing I don't understand is why we get the full devstack log in the job-output.txt and in the devstack log. Shouldn't the switchover to the devstack log file stop logging output in the job-output.txt? | 19:15 |
| fungi | AJaeger: i didn't see it fail, just abort and get requeued (again) | 19:15 |
| AJaeger | fungi: see the paste, that's the reason | 19:15 |
| fungi | AJaeger: yep | 19:15 |
| *** katkapilatova has left #openstack-infra | 19:16 | |
| *** katkapilatova has quit IRC | 19:16 | |
| AJaeger | fungi: I shouldn't have used "fail" here ;( | 19:16 |
| fungi | "'dict object' has no attribute 'checkout'" | 19:16 |
| corvus | item.value i think... fixing | 19:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Fix error in test-mirror-workspace-git-repos https://review.openstack.org/540977 | 19:17 |
| corvus | fungi, AJaeger, mordred: ^ | 19:17 |
| *** dhajare has quit IRC | 19:17 | |
| *** suhdood has quit IRC | 19:17 | |
| fungi | hooray for testing! | 19:18 |
| AJaeger | \o/ | 19:18 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/nodepool master: Convert nodepool-zuul-functional job https://review.openstack.org/540595 | 19:19 |
| AJaeger | corvus: +2a | 19:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Use item.checkout from zuul.projects when mirroring https://review.openstack.org/540934 | 19:21 |
| corvus | that's the 'real' change updated to match | 19:22 |
| *** jamesmcarthur has quit IRC | 19:22 | |
| *** d0ugal has joined #openstack-infra | 19:22 | |
| AJaeger | config-core, two quick reviews to remove legacy infra-ansible jobs: https://review.openstack.org/540970 and https://review.openstack.org/540971, please | 19:23 |
| *** slaweq has joined #openstack-infra | 19:23 | |
| openstackgerrit | Merged openstack-infra/zuul master: Update nodepool-integration for Zuul v3 https://review.openstack.org/540967 | 19:24 |
| *** oidgar has quit IRC | 19:26 | |
| *** dprince has quit IRC | 19:27 | |
| *** edmondsw has joined #openstack-infra | 19:29 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/nodepool master: Convert nodepool-zuul-functional job https://review.openstack.org/540595 | 19:29 |
| Shrews | AJaeger: i think the embedded single quotes are going to hurt you there | 19:30 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/nodepool master: Convert nodepool-zuul-functional job https://review.openstack.org/540595 | 19:30 |
| AJaeger | Shrews: yeah, fixed ^ | 19:30 |
| AJaeger | Better? | 19:30 |
| Shrews | *nod* | 19:30 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Add requirements to periodic kolla-k8s jobs https://review.openstack.org/540744 | 19:31 |
| * AJaeger waits for test results and thansk Shrews for review | 19:31 | |
| *** e0ne has quit IRC | 19:31 | |
| *** slaweq_ has joined #openstack-infra | 19:32 | |
| Shrews | AJaeger: where is zuul_work_dir used? | 19:32 |
| AJaeger | Shrews: by tox | 19:32 |
| AJaeger | Shrews: http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/tox/tasks/main.yaml#n36 | 19:33 |
| AJaeger | we need to override the default | 19:33 |
| pabelanger | Have to run into town for an errand. Should be back in time for weekly zuul meeting | 19:33 |
| clarkb | pabelanger: fungi I'm working on a response to https://github.com/kata-containers/packaging/issues/5 with info on how we've done package building in the past. Were the debian package builds hosted on tarballs or somewhere else (afs repo)? | 19:34 |
| fungi | clarkb: the recent ones were hosted on tarballs.o.o | 19:34 |
| fungi | if you're talking about ancient history, we hosted deb builds on lp ppas | 19:35 |
| clarkb | fungi: thinking of the more recent stuff | 19:35 |
| clarkb | that zigo et al did | 19:35 |
| *** tesseract has quit IRC | 19:36 | |
| *** slaweq_ has quit IRC | 19:36 | |
| fungi | clarkb: https://tarballs.openstack.org/packaging-deb/ | 19:36 |
| clarkb | infra-root basically kata is looking at package building (and hosting?) options though their exact needs aren't super clear to me they apparently had issues using OBS. I was going to explain how if they hosted their packaging on Gerrit they could get package build jobs and possibly if hosting on github too (though I think we are still working through that) which I guess would then be hosted on | 19:37 |
| clarkb | tarballs and could optionally be tested as well | 19:37 |
| dmsimard | OBS ? | 19:37 |
| clarkb | dmsimard: suse;s package build system | 19:37 |
| dmsimard | The only acronym that shortens to in my land is openshift build service | 19:38 |
| dmsimard | ah | 19:38 |
| clarkb | open build system or something like that | 19:38 |
| dmsimard | in RDO land, we use DLRN to build new packages (and repositories) each time a new commit lands in one of the packaged projects https://trunk.rdoproject.org/centos7-master/report.html | 19:39 |
| *** lucasagomes is now known as lucas-afk | 19:39 | |
| *** slaweq has quit IRC | 19:39 | |
| dmsimard | It's fairly generic if not for the fact that it's RPM, it's been used to test building ansible and systemd packages before | 19:40 |
| clarkb | ya I'm not sure what package formats they are interested in. More looking to give them a high level of "this is what we can do that is similar to OBS" | 19:40 |
| *** ijw has quit IRC | 19:41 | |
| fungi | obs sounds like the suse equivalent of launchpad ppa build automation | 19:41 |
| clarkb | ya | 19:41 |
| corvus | we have an event-driven arbitrary-code-execution machine, so... anything. :) | 19:41 |
| clarkb | corvus: indeed :) | 19:41 |
| dmsimard | Happy to share RDO's perspective although I realize that we're not very aligned with the rpm-packaging OpenStack project | 19:41 |
| fungi | not only that, but we have a fair amount of prior art in this area | 19:41 |
| clarkb | dmsimard: I think rdo is orthogonal? | 19:41 |
| clarkb | you all have a specific distro you target | 19:41 |
| clarkb | and consume upstreams | 19:41 |
| fungi | rdo uses dlrn to autobuild their packages? | 19:41 |
| dmsimard | clarkb: well, the tool that builds packages is not RDO specific | 19:42 |
| clarkb | this is an upstream wanting to make pacakges for downstreams to pull | 19:42 |
| *** dizquierdo has quit IRC | 19:42 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy infra-ansible jobs https://review.openstack.org/540970 | 19:42 |
| dmsimard | clarkb: it's really more or less a wrapper around mock/rpmbuild which is the toolchain to build packages for fedora/centos/rhel | 19:42 |
| clarkb | right but choice of $buildtool is also orthogonal I think? this is more about the build service and hosting? (I actually don't completely know because the info is lacking on that issue) | 19:42 |
| clarkb | I don't want to suggest they use a specific build tool and instead want to let them know they can run arbitrary build tools on our infra | 19:42 |
| clarkb | basically what corvus said :) | 19:43 |
| dmsimard | Ah, I didn't realize we were lacking details -- might be worth asking for more info | 19:43 |
| fungi | there is often a bit of tension between distros and upstreams who want to build distro packages. generalizing package builds across distros often requires a number of compromises for the sake of consistency, and the distro package maintainers tend to view it at best as a nuisance/distraction or a useless waste of duplicated effort | 19:43 |
| clarkb | and was going to use the debian stuff from before as an example since we've done it there | 19:43 |
| *** jamesmcarthur has joined #openstack-infra | 19:43 | |
| dmsimard | fungi: that's mostly the story between rdo and rpm-packaging :( | 19:43 |
| *** jamesmcarthur has quit IRC | 19:43 | |
| dmsimard | unfortunately | 19:43 |
| *** jamesmcarthur has joined #openstack-infra | 19:44 | |
| fungi | so we're in a somewhat better position as a general event-driven automation platform than the numerous various distro-specific package build automation systems | 19:44 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul master: Fix nodepool integration tests https://review.openstack.org/540983 | 19:44 |
| clarkb | AJaeger: and yes learning more about why OBS doesn't work for them would likely be useful | 19:45 |
| clarkb | at least for suse and possibly others | 19:45 |
| dmsimard | fungi: I'll hit "subscribe" on that issue and I'll wait for more info :) | 19:45 |
| dmsimard | I'm mostly curious as to what "packaging" means for them | 19:46 |
| fungi | agreed. their effort is in a somewhat orthogonal space to traditional software so normal distro packages may not make sense depending on what it is they're producing | 19:48 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix error in test-mirror-workspace-git-repos https://review.openstack.org/540977 | 19:48 |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Remove [webapp] section from v3 config https://review.openstack.org/540597 | 19:49 |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Zuulv3: Convert all proxy rules to zuul-web https://review.openstack.org/540598 | 19:49 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove unused xstatic_check_version.py https://review.openstack.org/540608 | 19:50 |
| openstackgerrit | Merged openstack-infra/project-config master: Add api-ref-jobs to blazar jobs https://review.openstack.org/540728 | 19:50 |
| corvus | rechecked 540952 | 19:51 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/project-config master: gerritbot: Add queens and rocky to ironic IRC notifications https://review.openstack.org/540986 | 19:54 |
| *** florianf has quit IRC | 19:57 | |
| *** edmondsw_ has joined #openstack-infra | 19:57 | |
| fungi | jobs are running for 540952 again | 19:57 |
| openstackgerrit | Merged openstack-infra/project-config master: Iotronic service and Lightning-rod have been moved to python3.5 in order to use the asyncio instead of Twisted https://review.openstack.org/539900 | 19:57 |
| fungi | though the linters job console stream is empty for me | 19:58 |
| fungi | oh, succeeded | 19:58 |
| fungi | it's the py35 job console stream which is empty i think | 19:58 |
| clarkb | infra-root how does https://etherpad.openstack.org/p/dmmTbgoFEM look? particular want to amek sure I'm not overpromising zuul features and github integration | 19:58 |
| corvus | clarkb: that looks reasonable | 19:59 |
| dmsimard | lgtm | 20:00 |
| *** edmondsw has quit IRC | 20:00 | |
| fungi | clarkb: looks great | 20:01 |
| openstackgerrit | Merged openstack-infra/project-config master: Add ansible-role-k8s-cinder https://review.openstack.org/534594 | 20:01 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/openstack-zuul-jobs master: [WIP] Add default_network extension drivers https://review.openstack.org/540990 | 20:03 |
| AJaeger | config-core, could you review https://review.openstack.org/538342 and https://review.openstack.org/538344 - to allow removing run-tox.sh and also the old unused jenkins-sudo-grep , please? | 20:04 |
| *** armaan_ has joined #openstack-infra | 20:04 | |
| *** armaan has quit IRC | 20:04 | |
| *** armaan_ has quit IRC | 20:04 | |
| *** armaan has joined #openstack-infra | 20:05 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use non-legacy job results in neutron-dynamic-routing grafana https://review.openstack.org/538073 | 20:05 |
| openstackgerrit | Merged openstack-infra/project-config master: Use non-legacy periodic job results of neutron-dynamic-routing in neutron grafana https://review.openstack.org/538084 | 20:06 |
| *** slaweq has joined #openstack-infra | 20:10 | |
| clarkb | AJaeger: re those changes we don't have a jenkins user at all anymore right? | 20:10 |
| clarkb | so no risk of this letting sudo use slip by | 20:10 |
| *** Goneri has quit IRC | 20:11 | |
| *** slaweq_ has joined #openstack-infra | 20:11 | |
| clarkb | AJaeger: left a comment on the first one | 20:12 |
| *** ijw has joined #openstack-infra | 20:12 | |
| AJaeger | clarkb: we have http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/playbooks/legacy/zuul-sudo-grep.sh | 20:13 |
| *** rockyg has joined #openstack-infra | 20:13 | |
| fungi | corvus: 540952 actually ran all jobs successfully... looking at logs now | 20:13 |
| jlvillal | Review request: https://review.openstack.org/#/c/499377/ Adding initial unit test framework to gerritbot. | 20:14 |
| *** vhosakot has quit IRC | 20:14 | |
| jlvillal | Been languishing for about 2+ months now. But has one +2 :) | 20:14 |
| AJaeger | clarkb: can we do the removal in a followup? I'd like to do it in two steps. | 20:14 |
| *** e0ne has joined #openstack-infra | 20:15 | |
| clarkb | AJaeger: the removal of the dib element file? ya I think that is fine as long as we do clean it out | 20:15 |
| clarkb | AJaeger: I'll update the review | 20:15 |
| *** ldnunes has quit IRC | 20:15 | |
| AJaeger | clarkb: ok, preparing change... | 20:16 |
| *** olaph1 has joined #openstack-infra | 20:16 | |
| *** slaweq_ has quit IRC | 20:16 | |
| *** slaweq has quit IRC | 20:16 | |
| clarkb | jlvillal: do you want to add the unittests to that change via a .zuul.yaml? | 20:16 |
| clarkb | jlvillal: will make the chagne self testing | 20:16 |
| AJaeger | clarkb: see also https://review.openstack.org/#/c/514485/ | 20:17 |
| *** olaph has quit IRC | 20:18 | |
| jlvillal | clarkb, Uh. How do I do that? I'm not sure how to add a unit test to .zuul.yaml. Any examples out there. | 20:18 |
| jlvillal | clarkb, And I would be happy to. | 20:18 |
| clarkb | AJaeger: I guess that will conflict with your change but is an easy rebase | 20:18 |
| clarkb | jlvillal: ya let me find an example | 20:18 |
| clarkb | or | 20:18 |
| jlvillal | clarkb, Awesome :) | 20:18 |
| clarkb | AJaeger: are we trying to keep the unittest jobs in project-config? or is it best to put that in the projects? | 20:18 |
| mordred | clarkb: I thikn we're trying to keep PTI jobs in project-config | 20:19 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove jenkins-sudo https://review.openstack.org/540994 | 20:19 |
| AJaeger | clarkb: ^ | 20:19 |
| *** rfolco|ruck is now known as rfolco|off | 20:19 | |
| fungi | corvus: i'm still not sure how to tell from http://logs.openstack.org/52/540952/1/check/tox-py35-on-zuul/1693b61/job-output.txt.gz#_2018-02-05_19_56_20_196686 whether it checked out the right state for zuul-jobs... i only see what looks like the zuul repo checkout stdout there? | 20:20 |
| clarkb | mordred: looks like nodepool does it in tree not in project-config | 20:20 |
| clarkb | jlvillal: https://git.openstack.org/cgit/openstack-infra/nodepool/tree/.zuul.yaml is an example from nodepool | 20:20 |
| AJaeger | clarkb: yeah, that change is in merge-conflict, let's do it piecewise. We had some fear doing that one - but this one should be ok. | 20:20 |
| mordred | clarkb: yah - nodepol isn't governed by the PTI | 20:20 |
| AJaeger | clarkb: yeah, we recommend but don't enforece it | 20:20 |
| jlvillal | clarkb, thanks | 20:20 |
| AJaeger | mordred: why not? It's an official OpenStack project | 20:20 |
| jlvillal | clarkb, So let me know which way to go. I can propose a project-config change or zuul.yaml | 20:20 |
| clarkb | jlvillal: you basically just need the stuff after line 34 and below | 20:21 |
| clarkb | and s/35/27/ | 20:21 |
| fungi | AJaeger: infra projects get a pass on a lot of that, and also it's about to become part of a separate community with its own governance rules fairly soon | 20:21 |
| *** slaweq has joined #openstack-infra | 20:21 | |
| clarkb | considering nodepool does it this way already I'm happy for gerritbot to do so as well | 20:22 |
| ianw | clarkb: hmm, i think it clones all the fd's ... it was all written for a time when the output of the stack.sh wasn't captured (by d-g) at all? | 20:23 |
| AJaeger | fungi, clarkb, yeah | 20:23 |
| fungi | _if_ we end up with a similar rule for mandated jobs across all infra projects, then i can see including them in project-config (or whatever equivalent we have) at that point | 20:23 |
| mordred | fungi: ++ | 20:23 |
| AJaeger | mordred, clarkb, you both gave +2 on https://review.openstack.org/538342 - shall I +A? Or wait for something? | 20:23 |
| clarkb | ianw: oh so it is writing to all the fds not doing a switchover | 20:23 |
| mordred | AJaeger: I just didn't +A because I was on the phone for the last couple of hours and wasn't sure I had all the context | 20:23 |
| mordred | AJaeger: I think feel free to +A | 20:23 |
| clarkb | ianw: maybe in this case we can/should consider dropping the devstack log proper and just have it all be in job output? | 20:24 |
| corvus | remember, for jobs to use irrelevant-files effectively, those need to be added when the job is added to the project. so mandating those jobs be in project-config means we're in the business of reviewing 'skip-if' changes. | 20:24 |
| AJaeger | mordred: ok, will do. | 20:24 |
| corvus | i for one, am *out* of the business of reviewing skip-if changes. :) | 20:24 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/gerritbot master: Add unit test framework and one unit test https://review.openstack.org/499377 | 20:24 |
| fungi | would be ideal if we could just say that "mandatory" jobs are really mandatory for all changes to their respective projects and so anything defined in project-config can't be "skipped" | 20:25 |
| jlvillal | Is there a known issue with: http://zuul.openstack.org/ ? | 20:25 |
| ianw | clarkb: https://git.openstack.org/cgit/openstack-dev/devstack/tree/stack.sh#n472 ... the outfilter timestamping thing is useful i think, for the not running under ansible case | 20:25 |
| fungi | jlvillal: plenty of known issues. what particular problem are you seeing though? | 20:25 |
| jlvillal | fungi, An almost empty page... | 20:25 |
| fungi | do you have a filter defined? | 20:26 |
| fungi | i see a very not-empty page there | 20:26 |
| jlvillal | All I see is: | 20:26 |
| jlvillal | Zuul Dashboard | 20:26 |
| jlvillal | Status | 20:26 |
| jlvillal | Jobs | 20:26 |
| jlvillal | Builds | 20:26 |
| clarkb | ianw: ya, maybe we just don't collect the log file in the job then? | 20:26 |
| corvus | jlvillal, fungi: oh, yep, i think i broke that | 20:27 |
| *** dsariel has joined #openstack-infra | 20:27 | |
| fungi | jlvillal: hrm, the stuff under there renders once the page manages to load the status.json | 20:27 |
| jlvillal | fungi, I've tried two different browsers | 20:27 |
| fungi | i wonder why it's not breaking for me. maybe i have some of it cached | 20:27 |
| jlvillal | corvus, Oh okay. Thanks. | 20:27 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Revert "Zuulv3: Convert all proxy rules to zuul-web" https://review.openstack.org/540995 | 20:27 |
| fungi | force refresh did indeed break it for me too | 20:27 |
| jlvillal | fungi, Cool, not just me then :) | 20:28 |
| fungi | ahh, yeah i guess we can't rewrite everything to the dashboard since there are static files we also need to serve | 20:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Zuul v3: handle github paylods with zuul-web https://review.openstack.org/540996 | 20:29 |
| *** dave-mcc_ has quit IRC | 20:29 | |
| corvus | yeah, not yet. soon. | 20:29 |
| * fungi totally failed to consider that in reviewing the original change | 20:29 | |
| corvus | fungi: ^ that's the minimum we need for now. | 20:29 |
| dhellmann | corvus : how'd the testing with that change go? | 20:29 |
| fungi | dhellmann: it succeeded but i'm not sure how to tell from http://logs.openstack.org/52/540952/1/check/tox-py35-on-zuul/1693b61/job-output.txt.gz#_2018-02-05_19_56_20_196686 whether it actually checked out the correct repo state | 20:30 |
| fungi | i only see stdout for the zuul repo checkout, not for zuul-jobs | 20:30 |
| dhellmann | wasn't the point of that series to run some unit tests with it? | 20:31 |
| corvus | fungi, dhellmann: yeah, it seems like we're missing one chunk of output from that, though it did iterate twice. i'vo noticed that on the old variant of the role too, so i think it's okay. maybe we're missing something in the output capture/log system. | 20:31 |
| fungi | the point was to determine whether we're going to break most jobs | 20:31 |
| fungi | which we almost did with the original change and only managed to catch by testing this way | 20:31 |
| dhellmann | ah, I thought someone mentioned some unit tests or something, my mistake | 20:31 |
| ianw | clarkb: for single node ... ok, and we can even drop the outfilter under ansible to stop the double timestamp. my only concern is that it's a fair bit easier in the multinode case to see the logs not mingled together in the one big file | 20:32 |
| dhellmann | I mean in addition to that testing | 20:32 |
| clarkb | ianw: hrm thats a good point | 20:32 |
| fungi | dhellmann: well, in theory that log i linked is a tox job which should have used the modified task | 20:32 |
| clarkb | ianw: maybe we stop indexing the devstack log file with e-r then and only index the job output (since it will mix them all together there anyways) | 20:32 |
| *** onovy has quit IRC | 20:33 | |
| smcginnis | corvus: I see a few of the zuul-checkout patches have merged. Are things looking OK so far? Any idea when we might be able to process some releases? | 20:35 |
| openstackgerrit | Merged openstack-infra/project-config master: Use zuul-sudo-grep https://review.openstack.org/538342 | 20:35 |
| dhellmann | fungi : ok. I'm also unsure how to tell from any of that git output whether it did the right thing | 20:35 |
| ianw | clarkb: in terms of indexing for searching, yeah i'd agree just the job-output.txt will have it all | 20:35 |
| fungi | smcginnis: current state is looking at the speculative run in a do-not-merge change | 20:35 |
| clarkb | ianw: ok last thing is the 2>&1 required in the job? I think ansible is getting both of them and interleaving them the same? | 20:36 |
| ianw | clarkb: i feel like if we're tricky, we might be able to send the output-to-log-file via the outfilter.py for timestamps, but leave the stdout/stderr without that for no timestamps | 20:36 |
| smcginnis | fungi: Is that one in that topic series? | 20:36 |
| dhellmann | it feels like we could do with a set -x in the "Update remote repository state correctly" task | 20:36 |
| clarkb | but maybe doing 2>&1 will make the interleaving of stderr into stdout more reliable | 20:36 |
| fungi | smcginnis: http://logs.openstack.org/52/540952/1/check/tox-py35-on-zuul/1693b61/job-output.txt.gz#_2018-02-05_19_56_20_196686 | 20:36 |
| openstackgerrit | Manoj Kumar proposed openstack-infra/project-config master: Third stage of removing trove-integration https://review.openstack.org/540997 | 20:37 |
| fungi | dhellmann: but at least better than the previous iteration, which would have resulted in almost every job failing out to a retry_limit state due to an otherwise uncaught typo | 20:37 |
| dhellmann | fungi : oh, yeah, no doubt | 20:37 |
| dhellmann | just thinking out loud while I try to make sense of the output | 20:37 |
| corvus | fungi: like here: http://logs.openstack.org/34/540934/3/check/tox-py35-on-zuul/55ffbb3/job-output.txt.gz#_2018-02-05_19_29_42_865799 you can see that ran 2 items, but we're missing output from one. (that's a build using the old role) | 20:37 |
| ianw | clarkb: hmm, yeah not 100% sure on that ... one for the ansible experts on what's best practice there i guess? if they skew out of sync wildly without 2>&1 , that would be annoying | 20:37 |
| AJaeger | config-core, now https://review.openstack.org/#/c/538344/ (remove zuul-sudo-grep setting) and https://review.openstack.org/540971 (openstack-zuul-jobs removal of unused infra-ansible job) are ready to merge, please review | 20:38 |
| *** jamesmcarthur has quit IRC | 20:38 | |
| clarkb | ianw: I'll reupdate the change to add the 2>&1 for safety and just clean out the devstack early stuff that is no longer needed | 20:38 |
| *** jamesmcarthur has joined #openstack-infra | 20:38 | |
| fungi | corvus: yeah, so it seems sane, we're unfortunately just missing output from the one bit of that loop we care about in this case? | 20:39 |
| *** onovy has joined #openstack-infra | 20:39 | |
| mordred | fungi, corvus: you can see it in the json log | 20:40 |
| dhellmann | corvus : could that conditional on the task have caused that? "when: item.1.stat.exists" | 20:40 |
| dhellmann | is item.1 the location of the cache? | 20:41 |
| dhellmann | oh, I guess not, ansible reports "skipping" for the next step when the conditional is false | 20:42 |
| mordred | http://logs.openstack.org/52/540952/1/check/tox-py35-on-zuul/1693b61/job-output.json.gz search for "Update remote repository" | 20:43 |
| corvus | mordred, fungi, dhellmann: yeah, play 1, task 33 seems to have the expected output in http://logs.openstack.org/52/540952/1/check/tox-py35-on-zuul/1693b61/job-output.json.gz | 20:43 |
| mordred | corvus: ++ I concur | 20:43 |
| openstackgerrit | Manoj Kumar proposed openstack-infra/project-config master: Third stage of removing trove-integration https://review.openstack.org/540997 | 20:45 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove zuul-sudo-grep lines from nodepool https://review.openstack.org/541000 | 20:45 |
| corvus | mordred, fungi, dhellmann: i'm +2 on https://review.openstack.org/540934 | 20:45 |
| AJaeger | clarkb: another split up from the big jenkins change ^ | 20:45 |
| fungi | corvus: mordred: dhellmann: i concur. approving | 20:46 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/gerritbot master: Add unit test framework and one unit test https://review.openstack.org/499377 | 20:47 |
| fungi | we'll still want to watch super closely as soon as 540934 lands | 20:47 |
| fungi | but at least we have a viable canary we can reenqueue | 20:48 |
| *** jcoufal has quit IRC | 20:48 | |
| *** slaweq_ has joined #openstack-infra | 20:48 | |
| *** dbecker has joined #openstack-infra | 20:49 | |
| andreaf | mordred: any reason for holding https://review.openstack.org/#/c/539497/? | 20:49 |
| fungi | infra-root: if i can get a couple reviews on the 540876 "Blackhole messages to FBC owner alias on lists" change to system-config i can take lists.o.o back out of the emergency maintenance file | 20:51 |
| *** efoley has joined #openstack-infra | 20:52 | |
| fungi | dmsimard: do you still need static.o.o in the emergency maintenance file? | 20:52 |
| fungi | looks like it's had puppet held since the filesystem recovery work | 20:52 |
| pabelanger | clarkb: ++ (kata) | 20:53 |
| *** rloo has joined #openstack-infra | 20:53 | |
| *** slaweq_ has quit IRC | 20:53 | |
| rloo | hi, should this URL exist? https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=stable/queens | 20:54 |
| rloo | or will it exist later? it is failing http://logs.openstack.org/82/538082/1/check/ironicclient-dsvm-functional/4cc4237/job-output.txt.gz#_2018-02-01_16_18_56_188050 | 20:54 |
| rloo | from this bot's patch: https://review.openstack.org/#/c/538082/ | 20:55 |
| AJaeger | rloo: best to discus with release team IMHO - #openstack-release channel | 20:55 |
| rloo | AJaeger: oh, sorry. wrong channel. thx! | 20:55 |
| AJaeger | rloo: I expect it will exist | 20:55 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/openstack-zuul-jobs master: [WIP] Add default_network extension drivers https://review.openstack.org/540990 | 20:55 |
| fungi | yeah, that's most likely pre-emptive for when the branch on the requirements repo gets created | 20:55 |
| AJaeger | mordred: I'll +A andreaf's https://review.openstack.org/#/c/539497 now - I assume the same comment as before applies (was on phone, didn't want to +A) | 20:56 |
| *** jamesmcarthur has quit IRC | 20:56 | |
| mordred | AJaeger: ya - I think so | 20:57 |
| *** jamesmcarthur has joined #openstack-infra | 20:57 | |
| AJaeger | done | 20:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Revert "Test Use item.checkout from zuul.projects when mirroring" https://review.openstack.org/540950 | 20:59 |
| dmsimard | fungi: we can take it out, yes | 20:59 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Use item.checkout from zuul.projects when mirroring https://review.openstack.org/540934 | 20:59 |
| corvus | fungi, mordred, dhellmann: i will re-enqueue that release item now | 20:59 |
| fungi | watching | 21:00 |
| * mordred is provisionally excited | 21:00 | |
| * fungi is provincially excited | 21:00 | |
| fungi | oh, not watching... the zuul dashboard is broken and i've uncached my static files :/ | 21:00 |
| corvus | i'll see if i can dig up a link from the log | 21:01 |
| openstackgerrit | John L. Villalovos proposed openstack-infra/gerritbot master: Add unit test framework and one unit test https://review.openstack.org/499377 | 21:01 |
| dmsimard | infra-root: Do we want to salvage any data from the log data we formatted ? It is still attached to a temporary server. I took the liberty of at least doing a "du" on all the data so we can see if there are jobs using up more disk space than others, maybe you have other idas | 21:01 |
| dmsimard | s/idas/ideas/ | 21:01 |
| *** jamesmcarthur has quit IRC | 21:02 | |
| fungi | since we haven't been under nearly as much block/inode pressure on that fs recently, i can't think of much pressing to analyze there | 21:02 |
| corvus | fungi, dhellmann, mordred: it will be build 49406003603641b1afb14efbb01cd2e7 not started yet | 21:03 |
| fungi | #status log removed static.openstack.org from emergency maintenance list | 21:03 |
| openstackstatus | fungi: finished logging | 21:03 |
| fungi | thanks corvus! | 21:03 |
| corvus | starting on ze02 | 21:03 |
| pabelanger | dmsimard: I've been meaning to try and rsync over the 00 folder again, as jenkins user | 21:03 |
| dmsimard | FWIW here's the top 100 sorted by disk usage http://paste.openstack.org/raw/662587/ | 21:04 |
| corvus | ze02 has no stream worker running now, so we can't stream it :( | 21:04 |
| *** ijw has quit IRC | 21:04 | |
| fungi | that probably explains the empty stream i saw for another job i was trying to check a little while ago | 21:04 |
| corvus | yeah, i've started discussing that in #zuul | 21:05 |
| dmsimard | pabelanger: do we have a need for it ? I guess we're nearing the expiration of those logs anyway | 21:05 |
| fungi | looks like ze02 has another oom event as recently as 2.5 hours ago | 21:05 |
| pabelanger | dmsimard: I mean, we could expire the logs on temp server, then rsync what is left, to help reduce size | 21:06 |
| AJaeger | team, getting the final pieces together for removing run-tox.sh: https://review.openstack.org/#/c/540595 (nodepool); https://review.openstack.org/#/c/540971/ (infra-ansible job in openstack-zuul-jobs); https://review.openstack.org/538344 (jenkins-sudo-grep); | 21:06 |
| *** ijw has joined #openstack-infra | 21:06 | |
| AJaeger | plus for pabelanger : windmill https://review.openstack.org/537789 | 21:07 |
| AJaeger | afterwards: https://review.openstack.org/540609 - removing run-tox.sh | 21:07 |
| * AJaeger calls it a day now and says thanks for all review love | 21:08 | |
| fungi | thanks AJaeger! | 21:08 |
| corvus | fungi, dhellmann, mordred: http://logs.openstack.org/4e/4e510b15c1b13fde307039e7472ed02a421856d1/release/release-openstack-python/4940600/ | 21:08 |
| fungi | pabelanger: if we wait much longer there won't be much to sync over anyway | 21:08 |
| dmsimard | pabelanger: I apologize for not being around to troubleshoot it but I didn't quite understand how the permissions on the other end ended up not matching the source despite "-a" -- the uid/gid did not exist on the temporary server but they matched the ones of jenkins on logs.o.o | 21:08 |
| AJaeger | pabelanger: ok, will quickly do your windmill suggestion | 21:09 |
| dhellmann | corvus : it looks like it ran ok this time | 21:09 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Replace legacy tempest-dsvm-py35 https://review.openstack.org/539497 | 21:09 |
| mordred | corvus, dhellmann: http://tarballs.openstack.org/murano-agent/murano-agent-3.4.0.tar.gz exists | 21:09 |
| pabelanger | fungi: agree | 21:09 |
| mordred | and https://pypi.python.org/pypi/murano-agent has 3.4.0 | 21:09 |
| mordred | so yay | 21:09 |
| fungi | dmsimard: rsync tries to match up user and group names on the assumption that uid and gid will be disparate | 21:09 |
| corvus | mordred, dhellmann, fungi: yay! things got simpler and better! | 21:10 |
| fungi | dmsimard: there is a different option for matching on uid/gid rather than user/group name i think, checking | 21:10 |
| pabelanger | dmsimard: np, one though was the permission changed happend after contents we're rsync'd. However, if we do it again, we should just use the jenkins user account, and setup require ssh keys on temp server. | 21:10 |
| mordred | corvus: love it when that happens | 21:10 |
| dhellmann | corvus , mordred , fungi : thanks! | 21:10 |
| dhellmann | smcginnis : I think we're ready to release things again ^^ | 21:10 |
| *** jamesmcarthur has joined #openstack-infra | 21:10 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove windmill-buildimages https://review.openstack.org/541009 | 21:10 |
| fungi | dmsimard: per the rsync manpage, that's --numeric-ids | 21:10 |
| openstackgerrit | Ian Wienand proposed openstack-infra/devstack-gate master: Check for virtualenv binary https://review.openstack.org/541010 | 21:11 |
| fungi | "don't map uid/gid values by user/group name" | 21:11 |
| smcginnis | Do we need to rerun the murano-agent release? | 21:11 |
| fungi | smcginnis: that's http://logs.openstack.org/4e/4e510b15c1b13fde307039e7472ed02a421856d1/release/release-openstack-python/4940600/job-output.txt.gz | 21:12 |
| AJaeger | pabelanger: updated https://review.openstack.org/537789 but need https://review.openstack.org/541009 first for that | 21:12 |
| dmsimard | fungi: -a implies -o but the man for o is a bit vague as to what the actual result ends up being -- it does mention it falls back to --numeric-ids in "some circumstances" :/ | 21:12 |
| smcginnis | fungi: Oh, great. I hadn't seen that yet. | 21:12 |
| pabelanger | fungi: other thing, our HDD usage on logs.o.o will no be lower that tripleo-test-cloud-rh1 jobs are gone. So we can likely start increasing our retention period again | 21:12 |
| fungi | pabelanger: right | 21:12 |
| pabelanger | AJaeger: +2 | 21:13 |
| fungi | pabelanger: dmsimard: on a related note, i have 526511 sitting awaiting approval for a while | 21:14 |
| *** bobh has quit IRC | 21:14 | |
| fungi | ahh, pabelanger already reviewed that one | 21:15 |
| *** jamesmcarthur has quit IRC | 21:15 | |
| dmsimard | pabelanger: if there hasn't been much -- if any complaints -- about missing logs in the past few days I'm not against spending the effort we'd spend on this elsewhere | 21:15 |
| dmsimard | btw in case it wasn't obvious, I'm no longer afk.. sorry about last week | 21:16 |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Revert "Zuulv3: Convert all proxy rules to zuul-web" https://review.openstack.org/540995 | 21:16 |
| jlvillal | clarkb, So I got it done with the zuul.yaml and tests pass: https://review.openstack.org/499377 Though it does run pep8 twice... | 21:18 |
| pabelanger | dmsimard: I only know of cmurphy asking for old logs so far | 21:18 |
| openstackgerrit | Merged openstack-infra/system-config master: Blackhole messages to FBC owner alias on lists https://review.openstack.org/540876 | 21:19 |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Zuul v3: handle github paylods with zuul-web https://review.openstack.org/540996 | 21:19 |
| dmsimard | pabelanger: I guess we could retrieve logs on a need basis until they "expire" on the temporary server, mostly concerned about the cost of keeping that temporary server up | 21:19 |
| fungi | i'm sure others noticed. i certainly ran into a few i just ended up rechecking to get fresh logs | 21:19 |
| dmsimard | fungi: yeah, in cases where you can recheck that's probably the best course of action -- in other circumstances (like tag/release/post) it's not as straightforward | 21:20 |
| fungi | though i also encountered one occasion where the world read permissions for a tree of logs was missing and had to be manuallt repaired | 21:20 |
| clarkb | jlvillal: ya we'll need to remove pep8 from the project in project-config we can do that after your change merges (I'll review it now) | 21:20 |
| *** dave-mccowan has joined #openstack-infra | 21:20 | |
| pabelanger | fungi: dmsimard: yah, I think if we expire temp logs, then confirm logs under 00 were rsync properly as jenkins user, we could then do the rest with zero impact | 21:21 |
| cmurphy | pabelanger: dmsimard yeah i noticed they were missing while trying to track some failures, i'm not sure it's obvious to everyone that if we complain hard enough we might be able to get them back | 21:25 |
| cmurphy | but the failures i was tracking reappeared elsewhere so i'm not personally that worried about it | 21:25 |
| jlvillal | clarkb, Thanks | 21:26 |
| *** rcernin has joined #openstack-infra | 21:26 | |
| *** panda|bbl is now known as panda | 21:26 | |
| dmsimard | cmurphy: the situation this time around was quite exceptional in order to avoid impacting the release as much as possible, I certainly hope we don't have to do this again | 21:28 |
| clarkb | and we've got ideas on how to make it better going forward | 21:28 |
| clarkb | we just need to nail down a plan and implement it | 21:29 |
| cmurphy | awesome | 21:29 |
| pabelanger | ++ | 21:29 |
| clarkb | I think mordred was going to report new swift things | 21:29 |
| mordred | yes, I am. I need to finish writing thatup | 21:30 |
| clarkb | mordred: I'm mostly curious to how you intend on solving the index problem :) | 21:33 |
| openstackgerrit | Merged openstack-infra/puppet-openstackci master: Stop expiring contents for docs-draft.o.o https://review.openstack.org/526511 | 21:34 |
| *** kgiusti has left #openstack-infra | 21:34 | |
| clarkb | as an INAP datapoint I don't see any in use insatnces older than 2 hours | 21:35 |
| clarkb | so I think the updates may have helped | 21:36 |
| *** e0ne has quit IRC | 21:36 | |
| *** rockyg has quit IRC | 21:39 | |
| dmsimard | clarkb: I recall the software factory implementation using zuul's swift uploader in the past but I don't have any details on how they did it. It had directory indexes as far as I remember. | 21:39 |
| *** e0ne has joined #openstack-infra | 21:41 | |
| *** Goneri has joined #openstack-infra | 21:42 | |
| dmsimard | tristanC: was there any black magic involved in ^ ? | 21:42 |
| mordred | clarkb: that part is easy - now that we have the zuul build history dashboard, I assert that we no longer need to be able to browse *between* builds -only between files in an individual build | 21:43 |
| mordred | clarkb: we only need to be able to browse files inside of a given build. but that's actually pretty easy - since we have all the files available when we upload at the individual build level | 21:44 |
| openstackgerrit | Ruby Loo proposed openstack-infra/project-config master: Add description to update_upper_constraints patches https://review.openstack.org/541027 | 21:44 |
| mordred | clarkb: anywho - I've got a bunch of notes in unintelligible form I need to make into a thing other people can read | 21:46 |
| *** panda is now known as panda|off | 21:48 | |
| *** myoung|dr is now known as myoung | 21:51 | |
| *** olaph has joined #openstack-infra | 21:53 | |
| *** olaph1 has quit IRC | 21:54 | |
| *** e0ne has quit IRC | 21:54 | |
| jhesketh | Morning | 21:55 |
| clarkb | mordred: ya I guess the dashboard does help a lot there. The big thing we ran into last we used swift was sorting periodic job results | 21:56 |
| clarkb | which the dashboard sort of does today | 21:56 |
| *** hasharAway has quit IRC | 21:56 | |
| clarkb | its hard to get older jobs from dashboard but we don't retain those on not swift anyways | 21:56 |
| *** edmondsw_ is now known as edmondsw | 21:57 | |
| *** dbecker has quit IRC | 21:58 | |
| mordred | clarkb: yah | 21:59 |
| corvus | zuul meeting time in #openstack-meeting-alt | 22:00 |
| *** abelur_ has quit IRC | 22:01 | |
| *** greghaynes has quit IRC | 22:03 | |
| smcginnis | dhellmann: We got another release job failure with a timeout. | 22:06 |
| smcginnis | http://logs.openstack.org/1c/1cd731f7c2cfc935fe9dddcb34d89193da3c7858/release-post/tag-releases/e356a3d/ara/ | 22:06 |
| smcginnis | Another one where it timed out doing a git fetch. | 22:06 |
| smcginnis | Maybe after the zuul meeting someone from infra can rerun that one. | 22:06 |
| *** jamesmcarthur has joined #openstack-infra | 22:07 | |
| dhellmann | smcginnis : the release worked, it just failed to create the branch? | 22:08 |
| dhellmann | oh, no, the branch is there too | 22:08 |
| dhellmann | it just failed to propose updates on the branch | 22:08 |
| smcginnis | Hmm, and I saw the requirements proposal. | 22:08 |
| dhellmann | smcginnis : we can handle the recovery for that ourselves | 22:08 |
| smcginnis | So I guess we are OK? | 22:08 |
| *** greghaynes has joined #openstack-infra | 22:09 | |
| dhellmann | it failed to propose the .gitreview update and the constraints update in tox.ini | 22:09 |
| smcginnis | dhellmann: Yeah, those are easy enough to do manually. Who wants it? | 22:10 |
| *** dave-mccowan has quit IRC | 22:10 | |
| dhellmann | on it | 22:10 |
| *** gfidente has quit IRC | 22:11 | |
| smcginnis | Thanks | 22:12 |
| *** slaweq_ has joined #openstack-infra | 22:12 | |
| *** efoley has quit IRC | 22:12 | |
| *** slaweq_ has quit IRC | 22:16 | |
| dhellmann | smcginnis : it appears they already have that .gitreview update | 22:16 |
| *** bobh has joined #openstack-infra | 22:16 | |
| dhellmann | and the constraint update | 22:16 |
| smcginnis | I'm confused, but if everything is happy... OK? | 22:17 |
| dhellmann | oh | 22:17 |
| dhellmann | they branched at the previous release | 22:17 |
| dhellmann | this is an FFE update | 22:17 |
| *** kjackal has quit IRC | 22:17 | |
| dhellmann | so the job did time out, but it didn't need to do the work it failed to do | 22:17 |
| dhellmann | and it would have figured that out | 22:17 |
| fungi | still, would be good to figure out why these git timeouts are endemic to release jobs... is it due to having to rely on git remote operations because we can't easily leverage zuul to prep the right repos as those end up determined at job runtime? | 22:18 |
| *** kjackal has joined #openstack-infra | 22:18 | |
| fungi | are the git remote operations wrapped in retry loops? we used to do that with git operations in old jobs because <unreliable cloud networks> | 22:19 |
| dhellmann | fungi : these calls aren't using the local cache, so that could well be it | 22:20 |
| dhellmann | I don't think they're in retry loops | 22:20 |
| fungi | based on past observations, that does sound like a recipe for semi-frequent job instability | 22:21 |
| pabelanger | git fetch origin --tags looks to have hung, maybe we should add timeout to gitconfig for HTTPS | 22:22 |
| *** slaweq has quit IRC | 22:22 | |
| *** CrayZee has quit IRC | 22:23 | |
| *** slaweq has joined #openstack-infra | 22:23 | |
| *** bobh has quit IRC | 22:23 | |
| *** wolverineav has quit IRC | 22:23 | |
| fungi | oh, yeah actually timing those out would be a good first step. at least then the job will properly fail because the fetch broke rather than sitting until the playbook timeout is reached | 22:24 |
| ianw | clarkb: ahh, i knew i had deja-vu on devstack output ... i've already re-jiggered outfilter.py to have a "bare" mode in diskimage-builder that drops the timestamp, for basically the same reason | 22:24 |
| fungi | but ultimately i think this means it'll need to be wrapped in a retry as well | 22:24 |
| dhellmann | fungi , pabelanger : how do we add a timeout like that? | 22:24 |
| fungi | dhellmann: it can be done via an envvar or in clobal or local git config, i believe | 22:24 |
| fungi | not sure if we already have a reusable role to set that | 22:25 |
| *** slaweq has quit IRC | 22:25 | |
| *** slaweq has joined #openstack-infra | 22:25 | |
| *** armaan has quit IRC | 22:26 | |
| *** armaan has joined #openstack-infra | 22:27 | |
| *** slaweq_ has joined #openstack-infra | 22:28 | |
| clarkb | ianw: I don't want to overthink this I mostly started out realizing the devstack early logs were missing and that we were double logging. Looks like at least half of that is an easy fix for the other we might just ignore it while we work on transition to the new job? | 22:30 |
| *** niska has quit IRC | 22:31 | |
| *** jlabarre has quit IRC | 22:33 | |
| *** slaweq_ has quit IRC | 22:33 | |
| openstackgerrit | Merged openstack/diskimage-builder master: upgrade pip before using -c option https://review.openstack.org/535983 | 22:34 |
| ianw | clarkb: yeah, I've just started migrating the centos/fedora/suse jobs to native in -> https://review.openstack.org/540704 ; and had half the changes for the filter (py3 compat) 1/2 unreviewed too https://review.openstack.org/474833 | 22:34 |
| ianw | so a good chance to bring it all together | 22:34 |
| *** slaweq has quit IRC | 22:35 | |
| *** niska has joined #openstack-infra | 22:36 | |
| *** olaph1 has joined #openstack-infra | 22:39 | |
| *** olaph has quit IRC | 22:41 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add git timeout setting for clone_repo.sh https://review.openstack.org/541050 | 22:41 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add a retry loop to clone_repo.sh https://review.openstack.org/541051 | 22:41 |
| dhellmann | smcginnis , fungi , pabelanger : ^^ | 22:41 |
| openstackgerrit | Merged openstack/diskimage-builder master: Create rescue user on ironic agent https://review.openstack.org/506579 | 22:43 |
| fungi | dhellmann: those look super helpful | 22:45 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard master: set up tests to run with sqlite https://review.openstack.org/539308 | 22:49 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard master: remove unused variable in test-cleanup.sh https://review.openstack.org/539695 | 22:49 |
| *** wolverineav has joined #openstack-infra | 22:57 | |
| *** wolverineav has quit IRC | 22:59 | |
| *** wolverineav has joined #openstack-infra | 23:00 | |
| *** lathiat has quit IRC | 23:00 | |
| *** bobh has joined #openstack-infra | 23:01 | |
| fungi | #status log removed lists.openstack.org from the emergency maintenance file | 23:02 |
| openstackstatus | fungi: finished logging | 23:02 |
| pabelanger | dhellmann: we parted that with GIT_HTTP_LOW_SPEED_LIMIT for zuul-mergers, might be worth also exposing it to control speed limit | 23:04 |
| pabelanger | paired* | 23:04 |
| dhellmann | I have no idea what a reasonable value for that might be | 23:05 |
| *** wolverineav has quit IRC | 23:05 | |
| pabelanger | maybe keep them the same as zuul-executor: https://docs.openstack.org/infra/zuul/admin/components.html?highlight=git_http_low_speed_limit#attr-merger.git_http_low_speed_limit | 23:06 |
| dhellmann | makes sense | 23:06 |
| fungi | double-plug good | 23:06 |
| fungi | plus | 23:06 |
| * fungi has a double-plug usb cable though, and it too is good | 23:07 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add git timeout setting for clone_repo.sh https://review.openstack.org/541050 | 23:07 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: add a retry loop to clone_repo.sh https://review.openstack.org/541051 | 23:07 |
| *** edmondsw has quit IRC | 23:07 | |
| dhellmann | knocking-off time here; I'll check in on those patches tomorrow morning | 23:08 |
| *** edmondsw has joined #openstack-infra | 23:08 | |
| *** tpsilva has quit IRC | 23:10 | |
| *** edmondsw has quit IRC | 23:12 | |
| *** ijw has quit IRC | 23:12 | |
| *** yamahata has joined #openstack-infra | 23:16 | |
| *** wolverin_ has joined #openstack-infra | 23:18 | |
| *** threestrands has joined #openstack-infra | 23:19 | |
| *** threestrands has quit IRC | 23:19 | |
| *** threestrands has joined #openstack-infra | 23:19 | |
| *** wolverin_ has quit IRC | 23:20 | |
| *** wolverin_ has joined #openstack-infra | 23:20 | |
| *** wolverin_ has quit IRC | 23:21 | |
| *** wolverineav has joined #openstack-infra | 23:22 | |
| ianw | clarkb: http://zuul.openstack.org/stream.html?uuid=81a0fd188a9a41f89884127824aac4dc&logfile=console.log is scolling by now without the double logs | 23:23 |
| ianw | s/logs/timestamps/ | 23:23 |
| clarkb | nice thats implementing the dib fix in devstack? | 23:24 |
| ianw | yep, that's the top of the stack -> https://review.openstack.org/541067 | 23:25 |
| pabelanger | clarkb: have we started search for PTG team dinner? | 23:25 |
| *** bobh has quit IRC | 23:26 | |
| *** yamahata has quit IRC | 23:26 | |
| ianw | i'd welcome any reviews on https://review.openstack.org/#/c/474833/5 from people for whom "python3" "unbuffered text i/o" doesn't strike fear into their hearts :) | 23:26 |
| clarkb | pabelanger: no, I've pinged the foundation planners for info on whether or not there would be any guide info this time around. I seem to recall them saying a lot of locations are very small | 23:26 |
| clarkb | but there are a few that could handle large groups | 23:26 |
| pabelanger | k | 23:27 |
| pabelanger | I can see about getting up which data/time work for people, they find location based on that | 23:28 |
| clarkb | I'm guessing our options will likely be pub A, pub B, and pub C (I'm ok with this) | 23:28 |
| pabelanger | ++ | 23:28 |
| *** wolverin_ has joined #openstack-infra | 23:29 | |
| *** wolverineav has quit IRC | 23:31 | |
| *** ijw has joined #openstack-infra | 23:32 | |
| *** wolverin_ has quit IRC | 23:35 | |
| *** ijw has quit IRC | 23:37 | |
| *** olaph has joined #openstack-infra | 23:37 | |
| *** slaweq has joined #openstack-infra | 23:38 | |
| *** bobh has joined #openstack-infra | 23:38 | |
| *** olaph1 has quit IRC | 23:38 | |
| *** yamahata has joined #openstack-infra | 23:42 | |
| *** slaweq has quit IRC | 23:42 | |
| *** yamahata has quit IRC | 23:43 | |
| *** yamahata has joined #openstack-infra | 23:44 | |
| corvus | clarkb, fungi, pabelanger: i'm starting to think that the bug clarkb found, https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842 may be related to our oom-killer issues | 23:44 |
| openstack | Launchpad bug 1655842 in linux-aws (Ubuntu Xenial) ""Out of memory" errors after upgrade to 4.4.0-59" [Undecided,Confirmed] | 23:44 |
| corvus | we're running a later kernel than that, but that bug does suggest that some workloads may need further fixes | 23:45 |
| clarkb | specifically if you have more than one numa node | 23:45 |
| clarkb | not sure if we have more than one though | 23:45 |
| corvus | i'm guessing that's indicated by "Node 0" ? in which case we only have one? | 23:46 |
| corvus | in general, it seems like a lot of stuff was changing around when pages get reclaimed, etc. and with the bulk of our available memory in 'recoverable slab' space, it seems like maybe it's not a stretch that we're running into a situation where that isn't happening. | 23:49 |
| *** HeOS has quit IRC | 23:50 | |
| *** hongbin has quit IRC | 23:53 | |
| *** ijw has joined #openstack-infra | 23:55 | |
| openstackgerrit | Luz Cazares proposed openstack-infra/project-config master: Added NodeJS v6 jobs to check pipeline https://review.openstack.org/537634 | 23:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/openstack-zuul-jobs master: Zuul: Remove project name https://review.openstack.org/541078 | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!