| ianw | i'm presuming we need to manually pull the updated cron running script because, chicken meet egg. so i've done that; watching the logs for next run | 00:05 |
|---|---|---|
| *** sthussey has quit IRC | 00:11 | |
| fungi | yes, and as noted in my review comment on the earlier change, once it's running regularly changes to that script will always lag one pulse behind | 00:12 |
| fungi | if we could move the git pull into the cron command itself or get that script to re-exec itself after the git pull, it could be solved of course | 00:13 |
| ianw | t minus 1 minute? | 00:14 |
| ianw | oh | 00:15 |
| ianw | well, you actually have to run it to get ansible to update the crontab | 00:15 |
| ianw | hrm | 00:15 |
| fungi | yeah, changes to the crontab will similarly run one pulse behind | 00:16 |
| ianw | except currently it's dead :) we should get the paddles out with a manual ansible-playbook run? | 00:16 |
| ianw | just a run of "ansible-playbook -f 10 /opt/system-config/playbooks/bridge.yaml" should bring cron into sync? | 00:18 |
| *** gyee has quit IRC | 00:19 | |
| *** felipemonteiro has joined #openstack-infra | 00:20 | |
| fungi | the -f10 is likely unnecessary | 00:21 |
| fungi | telling it to run against 10 hosts at a time when you're only updating one anyway | 00:21 |
| ianw | yeah, anyway i've done that, and the root crontab looks correct now. so t-8 minutes | 00:22 |
| *** harlowja has quit IRC | 00:23 | |
| ianw | fungi: https://review.openstack.org/#/c/594568 might be a good little test of the resyncing | 00:24 |
| ianw | ok, it's running ... | 00:30 |
| *** longkb has joined #openstack-infra | 00:33 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Generate roles documentation https://review.openstack.org/593478 | 00:40 |
| mordred | ianw: woot! | 00:42 |
| mordred | ianw: I was just checking in to see how that was going | 00:42 |
| ianw | mordred: does this finish within 15 minutes? | 00:42 |
| mordred | golly no | 00:42 |
| mordred | we have some work to do on optimization :) | 00:42 |
| mordred | https://review.openstack.org/#/c/594173/ is one patch that helps some | 00:42 |
| ianw | oh i guess it's flocked | 00:42 |
| mordred | yeah | 00:42 |
| mordred | fungi: ^^ also you if you get a sec - I tested the -f 20 this morning and it runs fine - and shaves the ansible run down to 10 minutes (for base.py) | 00:43 |
| ianw | hrm, no native logrotate support in ansible? | 00:45 |
| ianw | oh there's openstack/ansible-role-logrotate | 00:46 |
| openstackgerrit | Merged openstack-infra/system-config master: run_all.sh: add timestamp https://review.openstack.org/594568 | 00:50 |
| *** felipemonteiro has quit IRC | 00:59 | |
| ianw | mordred: interesting ... root 24043 24042 0 00:30 ? 00:00:00 /bin/sh -c flock -n /var/run/ansible/run_all.lock | 01:04 |
| ianw | it's still alive | 01:04 |
| ianw | root 25251 25250 0 00:48 ? 00:00:00 ... ssh ... 2001:4800:7819:103:be76:4eff:fe04:9229 /bin/sh -c '/usr/bin/python && sleep 0' | 01:05 |
| ianw | that seems to be review.o.o | 01:05 |
| ianw | review01.o.o to be precise | 01:06 |
| ianw | have we not accepted the host key? | 01:07 |
| fungi | it's hitting it as review.o.o because that's still in the inventory i think? | 01:10 |
| fungi | we noted we still need to delete the old review.o.o server | 01:10 |
| ianw | fungi: no sorry it was my mistake, fe04:9229 appears to be review01.o.o | 01:11 |
| ianw | it's moved on now, but that certainly seemed to get stuck | 01:11 |
| ianw | disappearing for some lunch. getting the -f20 and timestamp changes through will help validate if things are getting stuck | 01:19 |
| *** felipemonteiro has joined #openstack-infra | 01:21 | |
| *** jmorgan1 has joined #openstack-infra | 01:21 | |
| *** mrsoul has joined #openstack-infra | 01:21 | |
| *** dingyichen has joined #openstack-infra | 01:21 | |
| *** dmellado has quit IRC | 01:22 | |
| *** rcernin_ has joined #openstack-infra | 01:23 | |
| *** gouthamr has quit IRC | 01:23 | |
| *** stevebaker has quit IRC | 01:24 | |
| *** rcernin has quit IRC | 01:25 | |
| openstackgerrit | Merged openstack-infra/system-config master: Move tox basepython to top-level https://review.openstack.org/593982 | 01:26 |
| *** felipemonteiro has quit IRC | 01:31 | |
| fungi | looks like they're in merge conflict | 01:35 |
| fungi | i'll rebase if you'll approve when you get back | 01:35 |
| *** HenryG has quit IRC | 01:36 | |
| openstackgerrit | Filippo Inzaghi proposed openstack-infra/project-config master: import zuul job settings from project-config https://review.openstack.org/594621 | 01:36 |
| openstackgerrit | Chuck Short proposed openstack-infra/elastic-recheck master: Drop mox usage https://review.openstack.org/594642 | 01:40 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Increase the run_all forks count to 20 https://review.openstack.org/594173 | 01:43 |
| *** felipemonteiro has joined #openstack-infra | 01:45 | |
| *** HenryG has joined #openstack-infra | 01:46 | |
| *** HenryG has quit IRC | 01:48 | |
| *** rcernin has joined #openstack-infra | 02:00 | |
| *** felipemonteiro has quit IRC | 02:01 | |
| *** rcernin_ has quit IRC | 02:03 | |
| *** felipemonteiro has joined #openstack-infra | 02:05 | |
| *** felipemonteiro has quit IRC | 02:06 | |
| *** HenryG has joined #openstack-infra | 02:09 | |
| *** onovy has quit IRC | 02:12 | |
| openstackgerrit | Merged openstack-infra/git-review master: Testing getting specific patchset https://review.openstack.org/541892 | 02:21 |
| *** hongbin has joined #openstack-infra | 02:22 | |
| *** yamahata has quit IRC | 02:23 | |
| *** rlandy has quit IRC | 02:25 | |
| ianw | lgtm | 02:25 |
| *** apetrich has quit IRC | 02:26 | |
| *** zhangfei has joined #openstack-infra | 02:28 | |
| *** dhill_ has quit IRC | 02:34 | |
| *** gouthamr has joined #openstack-infra | 02:36 | |
| *** dmellado has joined #openstack-infra | 02:38 | |
| *** rpioso is now known as rpioso|afk | 02:43 | |
| *** edmondsw has quit IRC | 02:45 | |
| openstackgerrit | Merged openstack-infra/system-config master: Increase the run_all forks count to 20 https://review.openstack.org/594173 | 02:47 |
| *** stevebaker has joined #openstack-infra | 02:48 | |
| *** psachin has joined #openstack-infra | 02:51 | |
| *** yamahata has joined #openstack-infra | 03:04 | |
| *** jamesmcarthur has joined #openstack-infra | 03:08 | |
| *** jamesmcarthur has quit IRC | 03:13 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Generate roles documentation https://review.openstack.org/593478 | 03:13 |
| *** nicolasbock has quit IRC | 03:32 | |
| *** hongbin has quit IRC | 03:33 | |
| *** larainema has joined #openstack-infra | 03:33 | |
| *** diablo_rojo has quit IRC | 03:34 | |
| openstackgerrit | melissaml proposed openstack-infra/devstack-gate master: Remove the duplicated word https://review.openstack.org/594781 | 03:49 |
| *** felipemonteiro has joined #openstack-infra | 03:49 | |
| *** onovy has joined #openstack-infra | 03:53 | |
| *** felipemonteiro has quit IRC | 03:54 | |
| openstackgerrit | melissaml proposed openstack-infra/elastic-recheck master: Remove the duplicated word https://review.openstack.org/594783 | 03:57 |
| openstackgerrit | Merged openstack-infra/system-config master: Add bridge to cacti https://review.openstack.org/593965 | 03:57 |
| *** urbinaalex17 has joined #openstack-infra | 04:00 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Use doc bindep for tox-docs https://review.openstack.org/594790 | 04:10 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: Use doc bindep for tox-docs https://review.openstack.org/594792 | 04:12 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul master: Update docs build job https://review.openstack.org/594510 | 04:12 |
| *** Bhujay has joined #openstack-infra | 04:16 | |
| AJaeger | config-core, Could I get some reviews for https://review.openstack.org/#/c/593938/ and https://review.openstack.org/594159, please? | 04:20 |
| AJaeger | Also, https://review.openstack.org/#/q/status:open+++topic:tox-docs should be ready... | 04:20 |
| *** felipemonteiro has joined #openstack-infra | 04:24 | |
| openstackgerrit | Keiichi Hikita proposed openstack-infra/project-config master: Add translation job to zuul.d/projects.yaml https://review.openstack.org/594070 | 04:25 |
| *** onovy has quit IRC | 04:27 | |
| openstackgerrit | Keiichi Hikita proposed openstack-infra/project-config master: Add translation job to zuul.d/projects.yaml https://review.openstack.org/594070 | 04:27 |
| *** felipemonteiro has quit IRC | 04:28 | |
| *** annp has joined #openstack-infra | 04:30 | |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for sahara repositories https://review.openstack.org/593938 | 04:33 |
| openstackgerrit | melissaml proposed openstack-infra/infra-manual master: Remove the duplicated word https://review.openstack.org/594794 | 04:33 |
| ianw | AJaeger: i think tox runs tools/test_setup.sh which wants to start mysql, but it's not been bindeped in? | 04:34 |
| ianw | for 594510 i mena | 04:34 |
| ianw | oh heh, your comment :) | 04:34 |
| AJaeger | ianw: exactly. | 04:34 |
| AJaeger | yes, build-sphinx did not run tools/test_setup - but tox-docs does since all the tox jobs do it. | 04:35 |
| AJaeger | Now, question is: What is the desired behaviour in this case? I think we should not use it... | 04:35 |
| ianw | yeah, i don't think we need mysql and all the db's setup to build the docs | 04:36 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: Update tox-docs to follow build-sphinx better https://review.openstack.org/594792 | 04:48 |
| AJaeger | ianw: let's try this ^ | 04:48 |
| ianw | mordred: sed -n '/begin run/,/end run/p;/end run/q' run_all_cron.log | wc -c | 04:49 |
| ianw | 714056 | 04:49 |
| ianw | so i make that out as (714056 * ((24*60)/15)) / 1024) / 1024 ~= 65MiB a day of logs in that file. so yeah, rotation may not be a bad idea | 04:50 |
| ianw | (math may be wrong) | 04:50 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Update openstack-tox-docs to follow build-sphinx better https://review.openstack.org/594790 | 04:50 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: playbooks/roles/exim: remove "true" filter plugin https://review.openstack.org/593998 | 04:50 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: roles: Add README.rst and lint https://review.openstack.org/594800 | 04:50 |
| AJaeger | yeah, 594790 works for zuul repo (tox-docs job passing) | 04:54 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: Update tox-docs to follow build-sphinx better https://review.openstack.org/594792 | 04:57 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Update openstack-tox-docs to follow build-sphinx better https://review.openstack.org/594790 | 04:57 |
| *** sdake has quit IRC | 05:13 | |
| *** sdake has joined #openstack-infra | 05:13 | |
| *** onovy has joined #openstack-infra | 05:15 | |
| AJaeger | ianw: thanks for comment. Let me try that... | 05:16 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: roles: Add README.rst and lint https://review.openstack.org/594800 | 05:19 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove tagging rights for Adjutant team https://review.openstack.org/591836 | 05:19 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: Update tox-docs to follow build-sphinx better https://review.openstack.org/594802 | 05:20 |
| AJaeger | ianw: better? ^ | 05:20 |
| ianw | AJaeger: yeah, i think that looks like a more correct environment to be running tox -e docs in | 05:24 |
| *** mrsoul is now known as mschuppert | 05:25 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Update openstack-tox-docs to follow build-sphinx better https://review.openstack.org/594803 | 05:25 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul master: Update docs build job https://review.openstack.org/594510 | 05:27 |
| AJaeger | ianw: and then we need this as well ^ | 05:27 |
| AJaeger | I meant the one before | 05:27 |
| *** edmondsw has joined #openstack-infra | 05:34 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: Update tox-docs to follow build-sphinx better https://review.openstack.org/594802 | 05:34 |
| AJaeger | ianw: now enabled bindep again ^. Let's see whether that works... | 05:35 |
| *** felipemonteiro has joined #openstack-infra | 05:36 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Move exim role to top-level https://review.openstack.org/590753 | 05:37 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add kerberos-client role https://review.openstack.org/593486 | 05:42 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add openafs-client role https://review.openstack.org/593487 | 05:42 |
| AJaeger | doesn't work ;( | 05:48 |
| *** onovy has quit IRC | 05:49 | |
| *** sdake has quit IRC | 05:50 | |
| *** felipemonteiro has quit IRC | 05:50 | |
| adriant | AJaeger: I created the rocky branch manually, because it shouldn't be counted in any release stuff, but for Stein I'll submit a proper patch to releases for Adjutant | 05:51 |
| *** sdake has joined #openstack-infra | 05:51 | |
| *** ykarel has joined #openstack-infra | 05:54 | |
| adriant | also, any clue why https://review.openstack.org/#/c/589551/ is failing? although I can run it locally and the coverage tests pass, and the html is indeed created? | 05:54 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: Update tox-docs to follow build-sphinx better https://review.openstack.org/594802 | 05:55 |
| AJaeger | adriant: understood | 05:57 |
| AJaeger | adriant: http://logs.openstack.org/51/589551/1/check/openstack-tox-cover/da677aa/job-output.txt.gz#_2018-08-22_05_45_40_108192 - not sure why | 05:57 |
| adriant | yeah, I don't really understand. | 05:57 |
| adriant | I mean, I don't think we even really need the pike branch, but it's just odd | 05:57 |
| AJaeger | I think coverage_output_src looks at the wrong place | 05:59 |
| adriant | it works on all the other branches, and they all export to the same place :/ | 05:59 |
| adriant | it's so weird | 05:59 |
| AJaeger | adriant: compare tox.ini cover section, the pike one misses | 06:00 |
| AJaeger | "coverage html -d cover | 06:00 |
| AJaeger | added that and all is fine... | 06:00 |
| adriant | oh | 06:00 |
| adriant | I think I was looking in the wrong place | 06:00 |
| adriant | I thought it ran that | 06:01 |
| adriant | my bad, I'm just blind | 06:01 |
| * adriant will fix | 06:01 | |
| *** sdake has quit IRC | 06:02 | |
| *** sdake has joined #openstack-infra | 06:04 | |
| adriant | AJaeger: Huh I could have sworn I checked out the right branch, but it appears not. Yeah, in queens I updated tox.ini and split 'cover' and 'cover_report' so the default tox env included cover but didn't generate html. | 06:05 |
| *** onovy has joined #openstack-infra | 06:08 | |
| *** dpawlik has joined #openstack-infra | 06:08 | |
| AJaeger | ianw: yeah, seems to work finally - now waiting for all tests to finish... | 06:11 |
| AJaeger | pabelanger: http://logs.openstack.org/10/594510/4/check/tox-docs/83e8c70/job-output.txt.gz#_2018-08-22_06_08_49_906212 contains "ansible world writeable directory" - did you miss one system? | 06:11 |
| *** dpawlik has quit IRC | 06:13 | |
| openstackgerrit | Merged openstack-infra/infra-manual master: Remove the duplicated word https://review.openstack.org/594794 | 06:13 |
| *** dpawlik has joined #openstack-infra | 06:15 | |
| openstackgerrit | Nguyen Hai proposed openstack-infra/project-config master: remove job settings for karbor repositories https://review.openstack.org/594834 | 06:19 |
| *** ccamacho has joined #openstack-infra | 06:20 | |
| *** ccamacho has quit IRC | 06:20 | |
| *** ccamacho has joined #openstack-infra | 06:20 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Remove legacy-install-afs-with-puppet role https://review.openstack.org/590636 | 06:21 |
| AJaeger | ianw: I hope this is fine now, if you want to review the rest of https://review.openstack.org/#/q/status:open+++topic:tox-docs it would be nice. | 06:22 |
| * AJaeger takes a break now... | 06:22 | |
| *** diablo_rojo has joined #openstack-infra | 06:33 | |
| *** pcaruana has joined #openstack-infra | 06:41 | |
| *** gagehugo has quit IRC | 06:49 | |
| *** zhangfei has quit IRC | 07:03 | |
| *** dmellado has quit IRC | 07:04 | |
| *** ginopc has joined #openstack-infra | 07:06 | |
| *** dmellado has joined #openstack-infra | 07:06 | |
| *** rcernin has quit IRC | 07:07 | |
| *** mschuppert has quit IRC | 07:07 | |
| *** noama has joined #openstack-infra | 07:08 | |
| *** dmellado has quit IRC | 07:12 | |
| *** tosky has joined #openstack-infra | 07:13 | |
| *** longkb has quit IRC | 07:15 | |
| *** longkb has joined #openstack-infra | 07:16 | |
| *** sshnaidm|afk is now known as sshnaidm | 07:17 | |
| *** dmellado has joined #openstack-infra | 07:19 | |
| *** jpich has joined #openstack-infra | 07:20 | |
| *** dtantsur|afk is now known as dtantsur | 07:22 | |
| *** hashar has joined #openstack-infra | 07:23 | |
| AJaeger | ianw: https://review.openstack.org/#/c/594510 runs under python3, I commented | 07:23 |
| *** florianf has joined #openstack-infra | 07:23 | |
| *** evrardjp has joined #openstack-infra | 07:26 | |
| *** gagehugo has joined #openstack-infra | 07:26 | |
| *** alexchadin has joined #openstack-infra | 07:31 | |
| *** ccamacho has quit IRC | 07:33 | |
| *** shardy has joined #openstack-infra | 07:38 | |
| *** jtomasek has joined #openstack-infra | 07:41 | |
| *** tosky has quit IRC | 07:43 | |
| *** tosky has joined #openstack-infra | 07:43 | |
| *** jpena|off is now known as jpena | 07:47 | |
| ianw | AJaeger: yeah, it's a weirdish error and i'm not sure i'm that much more informed after reading the tox commit about it :) | 07:54 |
| *** e0ne has joined #openstack-infra | 08:01 | |
| AJaeger | ianw: yeah, it looks confusing... | 08:03 |
| *** ccamacho has joined #openstack-infra | 08:16 | |
| *** electrofelix has joined #openstack-infra | 08:16 | |
| *** ccamacho has quit IRC | 08:25 | |
| *** ccamacho has joined #openstack-infra | 08:26 | |
| *** holser_ has joined #openstack-infra | 08:26 | |
| *** pcaruana has quit IRC | 08:28 | |
| *** pcaruana has joined #openstack-infra | 08:30 | |
| *** diablo_rojo has quit IRC | 08:48 | |
| *** tosky has quit IRC | 08:59 | |
| *** tosky has joined #openstack-infra | 09:00 | |
| *** sambetts has joined #openstack-infra | 09:03 | |
| *** dingyichen has quit IRC | 09:03 | |
| ssbarnea|ruck | hi! i have a question regarding zuul timeouts..., as I seen lots of them recently. | 09:04 |
| *** elod has quit IRC | 09:06 | |
| *** jamesmcarthur has joined #openstack-infra | 09:09 | |
| ssbarnea|ruck | i investigated average execution time of one specific job and made few grim discoveries, its average succesfull execution duration is very close to the 180 cutout, which would explain why I got ~20% failure rate. I tried to find ways to speed it up, but is in the end, it is normal for an update job to be "longer". | 09:11 |
| *** elod has joined #openstack-infra | 09:12 | |
| *** jamesmcarthur has quit IRC | 09:13 | |
| *** ttx has joined #openstack-infra | 09:17 | |
| *** alexchadin has quit IRC | 09:17 | |
| *** urbinaalex17 has quit IRC | 09:17 | |
| *** alexchadin has joined #openstack-infra | 09:21 | |
| *** ykarel_ has joined #openstack-infra | 09:28 | |
| *** tobberydberg has joined #openstack-infra | 09:30 | |
| *** ykarel has quit IRC | 09:31 | |
| aspiers | AJaeger: why are blog posts about SUSE Manager appearing in the http://planet.openstack.org/ feed? | 09:39 |
| aspiers | seems like SUSE Conversations is part of the feed when it shouldn't be | 09:39 |
| aspiers | oh, it's only posts tagged OpenStack | 09:41 |
| aspiers | Hrm, and there's a weak link to SUSE OpenStack cloud. Oh well, I guess we can let that one go | 09:41 |
| *** ccamacho has quit IRC | 09:54 | |
| *** ccamacho has joined #openstack-infra | 09:55 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: enable build query by log_url https://review.openstack.org/594920 | 09:57 |
| *** ykarel_ is now known as ykarel | 09:58 | |
| *** jaosorior_ has quit IRC | 10:05 | |
| *** alexchadin has quit IRC | 10:13 | |
| *** alexchadin has joined #openstack-infra | 10:14 | |
| *** alexchadin has quit IRC | 10:14 | |
| *** alexchadin has joined #openstack-infra | 10:15 | |
| *** alexchadin has quit IRC | 10:15 | |
| *** longkb has quit IRC | 10:17 | |
| *** ykarel is now known as ykarel|lunch | 10:29 | |
| *** ykarel|lunch is now known as ykarel|away | 10:34 | |
| *** jesusaur has quit IRC | 10:38 | |
| AJaeger | dhellmann, mordred, converting infra jobs to use the PTI docs jobs instead of build-sphinx, I run into a few problems since the behavior is different. I propose https://review.openstack.org/594802 and https://review.openstack.org/594803 to fix those. Full set of changes is https://review.openstack.org/#/q/status:open++branch:master+topic:tox-docs . Could you review, please? | 10:39 |
| *** jesusaur has joined #openstack-infra | 10:41 | |
| *** nicolasbock has joined #openstack-infra | 10:44 | |
| *** dave-mccowan has joined #openstack-infra | 10:46 | |
| *** alexchadin has joined #openstack-infra | 10:48 | |
| *** alexchadin has quit IRC | 10:53 | |
| *** apetrich has joined #openstack-infra | 10:58 | |
| *** slagle has quit IRC | 11:01 | |
| *** alexchadin has joined #openstack-infra | 11:04 | |
| *** jaosorior has joined #openstack-infra | 11:15 | |
| *** ccamacho has quit IRC | 11:18 | |
| *** alexchadin has quit IRC | 11:19 | |
| *** ykarel|away has quit IRC | 11:19 | |
| *** jpena is now known as jpena|lunch | 11:20 | |
| *** alexchadin has joined #openstack-infra | 11:22 | |
| *** ccamacho has joined #openstack-infra | 11:25 | |
| AJaeger | config-core, could you review https://review.openstack.org/594159 and https://review.openstack.org/592290 , please? | 11:26 |
| AJaeger | thanks, mordred for reviewing the tox-docs stack! | 11:28 |
| mordred | AJaeger: thanks for making it! | 11:29 |
| AJaeger | ;) | 11:29 |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Update docs build job https://review.openstack.org/594518 | 11:32 |
| *** rh-jelabarre has joined #openstack-infra | 11:33 | |
| *** dhill_ has joined #openstack-infra | 11:33 | |
| openstackgerrit | Merged openstack-infra/zuul-base-jobs master: Update docs build job https://review.openstack.org/594521 | 11:34 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Update tox-docs to follow build-sphinx better https://review.openstack.org/594802 | 11:37 |
| openstackgerrit | Merged openstack-infra/project-config master: Use tox docs for zuul publishing https://review.openstack.org/594513 | 11:39 |
| openstackgerrit | Merged openstack-infra/project-config master: Update publish-openstack-sphinx-docs-direct jobs https://review.openstack.org/594159 | 11:40 |
| openstackgerrit | Merged openstack-infra/nodepool master: Update docs build job https://review.openstack.org/594517 | 11:41 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Update openstack-tox-docs to follow build-sphinx better https://review.openstack.org/594803 | 11:47 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Update docs build job https://review.openstack.org/594522 | 11:48 |
| *** mriedem_away is now known as mriedem | 11:54 | |
| * AJaeger just checked, all publish jobs published fine to zuul-ci | 11:54 | |
| *** mriedem has left #openstack-infra | 11:57 | |
| *** mriedem has joined #openstack-infra | 11:57 | |
| *** mriedem has quit IRC | 12:01 | |
| *** mriedem has joined #openstack-infra | 12:01 | |
| mordred | \o/ | 12:03 |
| mordred | AJaeger, dhellmann: I noticed reviewing os-client-config python3-first patches that the lib-forward-testing project-template is still using legacy-dsvm-tempest-src | 12:10 |
| mordred | lib-forward-testing-py3 uses a new tempest job | 12:10 |
| mordred | just thought I'd mention | 12:11 |
| *** ssbarnea|ruck has quit IRC | 12:13 | |
| *** kgiusti has joined #openstack-infra | 12:19 | |
| *** jpena|lunch is now known as jpena | 12:20 | |
| *** trown|outtypewww is now known as trown | 12:21 | |
| *** rosmaita has joined #openstack-infra | 12:25 | |
| *** agopi has quit IRC | 12:28 | |
| *** rlandy has joined #openstack-infra | 12:32 | |
| *** psachin has quit IRC | 12:32 | |
| mriedem | is it just me, or have some inap node jobs been failing in weird ways? | 12:34 |
| mriedem | like, i'm looking at what appears to be a nova-compute startup auto-killed on startup b/c it timed out | 12:35 |
| mriedem | http://logs.openstack.org/98/587498/3/check/nova-multiattach/609d3a7/logs/devstacklog.txt.gz#_2018-08-21_23_30_34_989 | 12:35 |
| mriedem | shortly before that: | 12:36 |
| mriedem | 2018-08-21 23:30:34.939 | Aug 21 23:29:53 ubuntu-xenial-inap-mtl01-0001455546 nova-consoleauth[28234]: WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 52.74 sec | 12:36 |
| mriedem | i was also noticing a very odd MessagingTimeout on an instances table db update which doesn't happen | 12:36 |
| mriedem | meaning things were so slow that conductor took over a minute to process the db update | 12:37 |
| dhellmann | mordred : yeah, I didn't go back and touch any of the old stuff when I added the new template. EENOUGHONMYPLATE | 12:37 |
| *** jamesmcarthur has joined #openstack-infra | 12:50 | |
| *** agopi has joined #openstack-infra | 12:56 | |
| *** jcoufal has joined #openstack-infra | 13:01 | |
| *** ssbarnea has joined #openstack-infra | 13:03 | |
| *** boden has joined #openstack-infra | 13:05 | |
| *** ssbarnea is now known as ssbarnea_ | 13:07 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for kuryr repositories https://review.openstack.org/594979 | 13:10 |
| mnaser | mriedem: http://logs.openstack.org/98/587498/3/check/nova-multiattach/609d3a7/logs/screen-dstat.txt.gz#_Aug_21_23_28_41_735591 looks like pretty not-ok io wait | 13:10 |
| mnaser | spiking up to 60% iowait | 13:10 |
| mnaser | maybe good to ping mgagne_ ^ | 13:10 |
| *** rh-jelabarre has quit IRC | 13:13 | |
| *** gtmanfred has joined #openstack-infra | 13:13 | |
| *** ssbarnea_ has quit IRC | 13:13 | |
| *** ssbarnea_ has joined #openstack-infra | 13:13 | |
| *** ssbarnea_ has quit IRC | 13:16 | |
| *** ssbarnea|ruck has joined #openstack-infra | 13:16 | |
| *** ssbarnea|ruck has quit IRC | 13:18 | |
| *** ssbarnea|ruck has joined #openstack-infra | 13:18 | |
| *** Bhujay has quit IRC | 13:21 | |
| *** eharney has joined #openstack-infra | 13:21 | |
| *** tpsilva has joined #openstack-infra | 13:25 | |
| *** felipemonteiro has joined #openstack-infra | 13:26 | |
| *** shardy is now known as shardy_mtg | 13:31 | |
| *** jcoufal has quit IRC | 13:33 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for PowerVMStackers repositories https://review.openstack.org/595004 | 13:34 |
| openstackgerrit | Merged openstack-infra/zuul master: Update docs build job https://review.openstack.org/594510 | 13:40 |
| mnaser | infra-root: how does nodepool go about cleaning images? | 13:42 |
| dmsimard | mnaser: it relies on images properties | 13:43 |
| mnaser | dmsimard: so images in our cloud since 2016.. is no bueno? | 13:43 |
| dmsimard | i.e, RAX substitutes the nodepool image properties by their own and images end up leaking | 13:43 |
| mnaser | it's taking up a lot of space in our environment | 13:45 |
| *** jcoufal has joined #openstack-infra | 13:48 | |
| mnaser | can we work together on cleaning them up? | 13:48 |
| *** r-daneel has joined #openstack-infra | 13:48 | |
| AJaeger | config-core, could you help me finish retiring rst2bash, please? https://review.openstack.org/592290 https://review.openstack.org/592292 | 13:49 |
| dmsimard | mnaser: ianw cleaned the ones from rax, there's probably a script around | 13:50 |
| mnaser | i mean i could probably easily lookup and find un-used images | 13:50 |
| mnaser | esp those that are prefixed with template- which are *really* old | 13:51 |
| AJaeger | dhellmann: commented on https://review.openstack.org/#/c/595004/ - please review and update goal-tools scripts | 13:53 |
| *** felipemonteiro has quit IRC | 13:54 | |
| openstackgerrit | Filippo Inzaghi proposed openstack-infra/zuul-jobs master: TEST https://review.openstack.org/595011 | 13:55 |
| dhellmann | AJaeger : could we move the legacy jobs as part of a separate step or something? I really can't manage this goal and deal with all of those changes. That's not what I signed up for. | 13:56 |
| AJaeger | dhellmann: yes, we can | 13:58 |
| *** alexchadin has quit IRC | 13:59 | |
| *** gouthamr has quit IRC | 14:00 | |
| *** dmellado has quit IRC | 14:01 | |
| *** stevebaker has quit IRC | 14:01 | |
| *** shardy_mtg is now known as shardy | 14:01 | |
| *** nicolasbock has quit IRC | 14:04 | |
| *** rh-jelabarre has joined #openstack-infra | 14:06 | |
| *** pbourke has quit IRC | 14:07 | |
| pabelanger | AJaeger: I'll look more for 777 directory | 14:07 |
| pabelanger | AJaeger: Hmm, that is ze01.o.o will debug more after some coffee | 14:08 |
| *** felipemonteiro has joined #openstack-infra | 14:08 | |
| *** pbourke has joined #openstack-infra | 14:09 | |
| *** haleyb has joined #openstack-infra | 14:10 | |
| *** alexchadin has joined #openstack-infra | 14:10 | |
| *** felipemonteiro has quit IRC | 14:10 | |
| *** felipemonteiro has joined #openstack-infra | 14:14 | |
| *** efried is now known as efried_goatin | 14:14 | |
| *** jiapei has joined #openstack-infra | 14:16 | |
| AJaeger | thanks, pabelanger. No urgency on that one... | 14:16 |
| pabelanger | AJaeger: the executor-git dir on ze01 looks correct, checking job-output log now | 14:18 |
| *** felipemonteiro has quit IRC | 14:18 | |
| *** felipemonteiro has joined #openstack-infra | 14:18 | |
| *** pbourke has quit IRC | 14:23 | |
| *** ssbarnea|ruck has quit IRC | 14:23 | |
| *** felipemonteiro has quit IRC | 14:24 | |
| *** pbourke has joined #openstack-infra | 14:25 | |
| *** dabukalam has quit IRC | 14:25 | |
| pabelanger | I'm going to setup an autohold and debug more, the issue looks to be node side from what I can see | 14:25 |
| *** dpawlik has quit IRC | 14:27 | |
| *** felipemonteiro has joined #openstack-infra | 14:27 | |
| *** felipemonteiro has quit IRC | 14:30 | |
| Shrews | mnaser: dmsimard: nodepool keeps the 2 most recent images, based on data stored in zookeeper. image meta properties aren't really used, iirc. | 14:31 |
| *** nicolasbock has joined #openstack-infra | 14:31 | |
| Shrews | mnaser: those beginning with 'template-' are probably from old nodepool | 14:31 |
| pabelanger | oh, yah. if you see template- images, they are likely really old and we leaked them in nodepoolv1 | 14:32 |
| AJaeger | Shrews: So, anything older than a week can be easily deleted? | 14:32 |
| *** eernst has joined #openstack-infra | 14:32 | |
| *** alexchadin has quit IRC | 14:33 | |
| pabelanger | if there is an image older then week and named not template-*, then we likely have a bug in nodepool that needs fixed. | 14:33 |
| *** felipemonteiro has joined #openstack-infra | 14:33 | |
| Shrews | AJaeger: not necessarily. we'd need to verify against image builds | 14:33 |
| pabelanger | but provider could also be paused | 14:33 |
| Shrews | what pabelanger said | 14:33 |
| mnaser | im removing all the extra images then, some since 2017 | 14:35 |
| *** felipemonteiro has quit IRC | 14:35 | |
| pabelanger | yah, we likely didn't clean up the provider when we moved to zk | 14:36 |
| pabelanger | mnaser: IIRC, mordred had an shade script to do it too | 14:36 |
| fungi | mnaser: yeah, i think we went through and cleaned up the old nodepool images when we transitioned, but weren't using vexxhost with nodepool at the time so probably missed those | 14:38 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Create tox_revoke_sudo variable for tox job. https://review.openstack.org/594569 | 14:42 |
| dhellmann | AJaeger : cool, thanks | 14:43 |
| dhellmann | I'll fix up the other issues, of course | 14:44 |
| *** jrist has joined #openstack-infra | 14:45 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: DNM - testing docs https://review.openstack.org/595070 | 14:48 |
| *** gouthamr has joined #openstack-infra | 14:50 | |
| *** jamesmcarthur has quit IRC | 14:51 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for PowerVMStackers repositories https://review.openstack.org/595004 | 14:54 |
| openstackgerrit | Sean McGinnis proposed openstack-infra/devstack-gate master: Remove branch handling for deleted branches https://review.openstack.org/545030 | 14:54 |
| *** mmedvede is now known as mmedvede_ | 14:55 | |
| *** mmedvede_ is now known as mmedvede | 14:55 | |
| *** jamesmcarthur has joined #openstack-infra | 14:56 | |
| *** apetrich has quit IRC | 14:58 | |
| *** ssbarnea|ruck has joined #openstack-infra | 15:02 | |
| *** tonyb has quit IRC | 15:04 | |
| *** jamesmcarthur has quit IRC | 15:06 | |
| *** efried_goatin is now known as efried | 15:06 | |
| *** pcaruana has quit IRC | 15:10 | |
| *** jamesmcarthur has joined #openstack-infra | 15:10 | |
| *** rh-jelabarre has quit IRC | 15:11 | |
| AJaeger | pabelanger, mnaser, could you help me finish retiring rst2bash, please? https://review.openstack.org/592290 and https://review.openstack.org/592292 are the two reviews | 15:12 |
| *** jamesmcarthur has quit IRC | 15:15 | |
| openstackgerrit | Nguyen Hai proposed openstack-infra/project-config master: remove job settings for zun repositories https://review.openstack.org/595221 | 15:22 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/system-config master: Add timezone role https://review.openstack.org/594542 | 15:23 |
| *** r-daneel has quit IRC | 15:23 | |
| *** r-daneel has joined #openstack-infra | 15:24 | |
| openstackgerrit | Merged openstack-infra/project-config master: Retire rst2bash (step 4) https://review.openstack.org/592290 | 15:24 |
| AJaeger | config-core, simple job removal change for openstack-zuul-jobs, please review: https://review.openstack.org/592292 | 15:26 |
| pabelanger | AJaeger: for some reason /home/zuul/src is still 777 on remote node. Trying to figure out why | 15:28 |
| *** sthussey has joined #openstack-infra | 15:30 | |
| *** openstackgerrit has quit IRC | 15:31 | |
| pabelanger | Oh, hmm | 15:32 |
| *** jamesmcarthur has joined #openstack-infra | 15:33 | |
| *** efried has quit IRC | 15:36 | |
| *** efried has joined #openstack-infra | 15:36 | |
| *** nicolasbock has quit IRC | 15:43 | |
| *** openstackgerrit has joined #openstack-infra | 15:45 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: base-test: Create /home/zuul/src directory https://review.openstack.org/595229 | 15:45 |
| *** dmellado has joined #openstack-infra | 15:45 | |
| dhellmann | clarkb , AJaeger : as part of the python3 goal stuff I've been including the infra repos in the set I test the scripts against, assuming that you would want to move the job settings out of project-config. Is that assumption valid? | 15:45 |
| pabelanger | AJaeger: first stab at debugging 777 on remote node ^ | 15:45 |
| pabelanger | AJaeger: in rdoproject /home/zuul/src is 0755, but we don't do use-cache-repos role | 15:46 |
| pabelanger | wonder if somehow that is creating the directory with 777 by default | 15:46 |
| *** nicolasbock has joined #openstack-infra | 15:49 | |
| *** pcaruana has joined #openstack-infra | 15:49 | |
| AJaeger | dhellmann: clarkb is on vacation - best ask him when he's back (Friday?) | 15:52 |
| dhellmann | AJaeger : thanks | 15:52 |
| dhellmann | AJaeger : do you know what happened to the rst2bash project and why it's causing errors in https://review.openstack.org/#/c/595004/ ? | 15:53 |
| corvus | dhellmann: i heard talk about retiring it | 15:53 |
| dhellmann | maybe that just needs a rebase? | 15:53 |
| corvus | dhellmann: https://review.openstack.org/592290 | 15:54 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for PowerVMStackers repositories https://review.openstack.org/595004 | 15:54 |
| *** diablo_rojo has joined #openstack-infra | 16:02 | |
| *** rpioso|afk is now known as rpioso | 16:02 | |
| *** gouthamr has quit IRC | 16:05 | |
| *** jpena is now known as jpena|off | 16:09 | |
| *** efried is now known as efried_rollin | 16:14 | |
| AJaeger | dhellmann: I think a reace... | 16:22 |
| AJaeger | race I mean | 16:22 |
| AJaeger | we just merged the change to retire it and remove it from zuul - while your change was not rebased on top of it and had it in there. So, I rechecked - if that fails, we need to dig into it. | 16:23 |
| *** rh-jelabarre has joined #openstack-infra | 16:28 | |
| *** masayukig has quit IRC | 16:28 | |
| AJaeger | config-core, speaking of rst2bash, last change to cleanup from openstack-zuul-jobs, please review: https://review.openstack.org/592292 | 16:30 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Update query for bug 1770211 https://review.openstack.org/595246 | 16:30 |
| openstack | bug 1770211 in OpenStack Compute (nova) "AttachVolumeMultiAttachTest.test_resize_server_with_multiattached_volume intermittently fails with "Unable to detach from guest transient domain."" [Medium,Confirmed] https://launchpad.net/bugs/1770211 | 16:30 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/project-config master: Add publish-to-pypi job to mistral-tempest-plugin https://review.openstack.org/595248 | 16:32 |
| *** ginopc has quit IRC | 16:33 | |
| *** hashar has quit IRC | 16:34 | |
| *** jiapei has quit IRC | 16:35 | |
| *** jpich has quit IRC | 16:40 | |
| *** eernst has quit IRC | 16:42 | |
| *** gouthamr has joined #openstack-infra | 16:46 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: WIP add system-config-run-base job https://review.openstack.org/595252 | 16:46 |
| corvus | ianw, Shrews, fungi, pabelanger: ^ | 16:47 |
| *** ykarel has joined #openstack-infra | 16:51 | |
| *** trown is now known as trown|lunch | 16:52 | |
| *** jamesmcarthur has quit IRC | 16:52 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: WIP add testinfra to run-base https://review.openstack.org/595258 | 16:58 |
| corvus | pabelanger: ^ does that look roughly correct? | 16:58 |
| *** felipemonteiro has joined #openstack-infra | 16:58 | |
| ianychoi | AJaeger, hello! I wanna ask to you on propose-translation-update job. On last weekend, I found that the failure on propose-translation-update-octavia-dashboard was not shown on status.openstack.org. Would u help how to track such failures on status.o.o site? I wanna share my observation: | 16:58 |
| ianychoi | For example, http://status.openstack.org/openstack-health/#/job/propose-translation-update-octavia-dashboard?duration=P14D does not show failures | 16:59 |
| ianychoi | But when I see actual log file http://logs.openstack.org/periodic/git.openstack.org/openstack/octavia-dashboard/master/propose-translation-update/4e80d84/job-output.txt.gz#_2018-08-18_07_45_58_841602 | 16:59 |
| ianychoi | The job had failure.. | 16:59 |
| Shrews | corvus: to be clear, that doesn't actually run against bridge.o.o, but a new node acting as it, right? | 16:59 |
| corvus | Shrews: correct | 16:59 |
| corvus | Shrews: the bridge playbook is written with 'bridge.openstack.org' as its host matcher, so if i want to use it directly, i have to have a host with that name in the inventory | 17:00 |
| Shrews | corvus: ah, the zuul generated inventory points that to the provisioned node | 17:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: WIP add system-config-run-base job https://review.openstack.org/595252 | 17:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: WIP add testinfra to run-base https://review.openstack.org/595258 | 17:01 |
| corvus | Shrews: yep, cause i named it that in the nodeset for the job | 17:01 |
| Shrews | was trying to wrap my head around the whole "real dns" vs "generated" stuff | 17:01 |
| Shrews | got lost | 17:01 |
| Shrews | where am i? | 17:01 |
| Shrews | :) | 17:01 |
| corvus | in a maze of twisty passages all alike | 17:02 |
| corvus | new patchset just actually runs the job. i have a bunch of stuff commented out to reduce resource usage while we iterate on that. | 17:02 |
| *** gyee has joined #openstack-infra | 17:03 | |
| Shrews | corvus: maybe would could add .fake to the name or something just to make it obvious | 17:03 |
| corvus | Shrews: if we do that, we won't be able to use this file directly: http://git.openstack.org/cgit/openstack-infra/system-config/tree/playbooks/bridge.yaml | 17:03 |
| Shrews | oh yes, well, then let's not do that :) | 17:04 |
| corvus | Shrews: if we run into some reason we can't use that file directly, then yeah, i'm all for it. :) | 17:04 |
| corvus | (probably just call it 'bridge' instead of the fqdn) | 17:04 |
| corvus | i'm guessing root_rsa_key is defined in private hiera | 17:06 |
| corvus | i think we'll need to define that for this job | 17:06 |
| *** e0ne has quit IRC | 17:07 | |
| *** hashar has joined #openstack-infra | 17:09 | |
| ianychoi | AJaeger, Frank shared zuul.o.o instead of status.o.o to see failures.. I should have been more accustomed with zuul.o.o.. never mind and thanks :) | 17:10 |
| corvus | ianychoi: oh yes, sorry, we're still building the zuul dashboard. it's not obvious yet, but you can deep-link like this: http://zuul.openstack.org/builds.html?job_name=propose-translation-update&project=openstack%2Foctavia-dashboard | 17:11 |
| ianychoi | corvus, thanks a lot! | 17:12 |
| *** rh-jelabarre has quit IRC | 17:15 | |
| *** yamahata has quit IRC | 17:18 | |
| *** ykarel has quit IRC | 17:20 | |
| *** ykarel has joined #openstack-infra | 17:21 | |
| *** sambetts is now known as sambetts|afk | 17:23 | |
| *** ykarel_ has joined #openstack-infra | 17:24 | |
| *** ykarel_ has quit IRC | 17:24 | |
| AJaeger | ianychoi: not sure what happened with health dashboard - don't remember who can help with it. But zuul dashboard is best to check for these... | 17:24 |
| *** ykarel_ has joined #openstack-infra | 17:24 | |
| *** electrofelix has quit IRC | 17:25 | |
| *** sambetts|afk has quit IRC | 17:25 | |
| *** xarses has quit IRC | 17:25 | |
| *** Douhet has quit IRC | 17:25 | |
| *** DinaBelova has quit IRC | 17:25 | |
| *** mmedvede has quit IRC | 17:25 | |
| *** rf0lc0 has quit IRC | 17:25 | |
| *** cburgess has quit IRC | 17:25 | |
| *** jpena|off has quit IRC | 17:25 | |
| *** panda|off has quit IRC | 17:25 | |
| *** mnencia has quit IRC | 17:25 | |
| *** mtreinish has quit IRC | 17:25 | |
| *** EmilienM has quit IRC | 17:25 | |
| *** dulek has quit IRC | 17:25 | |
| *** mudpuppy has quit IRC | 17:25 | |
| *** gothicmindfood has quit IRC | 17:25 | |
| *** rakhmerov has quit IRC | 17:25 | |
| *** weshay has quit IRC | 17:25 | |
| *** ericyoung has quit IRC | 17:25 | |
| *** jlvillal has quit IRC | 17:25 | |
| *** hemna has quit IRC | 17:25 | |
| *** dims has quit IRC | 17:25 | |
| *** hrubi has quit IRC | 17:25 | |
| *** Anticimex has quit IRC | 17:25 | |
| *** niedbalski has quit IRC | 17:25 | |
| *** Diabelko has quit IRC | 17:25 | |
| *** corvus has quit IRC | 17:25 | |
| *** Gorian has joined #openstack-infra | 17:25 | |
| *** DinaBelova has joined #openstack-infra | 17:26 | |
| ianychoi | AJaeger, yeah nice experience using zuul dashboard except one thing currently: there are so many translation-update job items but I cannot see page buttons on the web page like "page 2, page 3". But very happy to see zuul dashboard interfaces | 17:26 |
| *** mudpuppy has joined #openstack-infra | 17:26 | |
| *** ykarel has quit IRC | 17:26 | |
| *** jlvillal has joined #openstack-infra | 17:27 | |
| *** kota_ has quit IRC | 17:27 | |
| *** hashar has quit IRC | 17:27 | |
| *** ykarel_ has quit IRC | 17:27 | |
| AJaeger | ianychoi: yeah ;( it would need some more filters like failed only, specific branches | 17:28 |
| *** Guest36339 has joined #openstack-infra | 17:29 | |
| *** gouthamr has quit IRC | 17:29 | |
| *** Guest36339 is now known as jeblair | 17:29 | |
| *** holser_ has quit IRC | 17:30 | |
| AJaeger | jeblair (not corvus?): we said that tag triggered jobs (release pipeline) should be in project-config. What about repos that only have master branch? I see some infra repos that have in master release pipeline. will that work? | 17:30 |
| jeblair | AJaeger: corvus is waiting for nickserv to resync :) | 17:31 |
| AJaeger | Ah ;) | 17:31 |
| *** EmilienM has joined #openstack-infra | 17:31 | |
| jeblair | AJaeger: there should be no problem with an unbranched repo having release jobs in the in-repo config | 17:31 |
| *** cburgess has joined #openstack-infra | 17:32 | |
| AJaeger | jeblair: ah, ok | 17:32 |
| *** dims_ has joined #openstack-infra | 17:36 | |
| *** jpena has joined #openstack-infra | 17:36 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: WIP add system-config-run-base job https://review.openstack.org/595252 | 17:36 |
| *** markvoelker has quit IRC | 17:39 | |
| *** mnencia has joined #openstack-infra | 17:40 | |
| *** jeblair is now known as corvus | 17:41 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove fuxi jobs https://review.openstack.org/595273 | 17:43 |
| *** rodrigods has joined #openstack-infra | 17:44 | |
| *** felipemonteiro has quit IRC | 17:45 | |
| *** mtreinish has joined #openstack-infra | 17:52 | |
| *** mmedvede has joined #openstack-infra | 17:58 | |
| *** eharney has quit IRC | 18:01 | |
| *** eernst has joined #openstack-infra | 18:06 | |
| *** eernst has quit IRC | 18:07 | |
| *** jamesmcarthur has joined #openstack-infra | 18:07 | |
| *** hemna has joined #openstack-infra | 18:08 | |
| *** eernst has joined #openstack-infra | 18:09 | |
| *** mriedem is now known as mriedem_lunch | 18:14 | |
| *** shardy is now known as shardy_afk | 18:15 | |
| *** ChanServ has quit IRC | 18:16 | |
| *** stevebaker has joined #openstack-infra | 18:18 | |
| *** trown|lunch is now known as trown | 18:21 | |
| *** ChanServ has joined #openstack-infra | 18:22 | |
| *** barjavel.freenode.net sets mode: +o ChanServ | 18:22 | |
| *** dmellado has quit IRC | 18:22 | |
| mtreinish | ianychoi: if that's the case then the subunit generation script isn't working properly | 18:22 |
| *** stevebaker has quit IRC | 18:23 | |
| Shrews | corvus: so, what exactly needs done for ntp on bridge? looks like maybe we just need to add the distro specific name for that package in http://git.openstack.org/cgit/openstack-infra/system-config/tree/playbooks/roles/base-server/vars files. What else? | 18:23 |
| Shrews | If it were that simple, I would think mordred would have already done that | 18:24 |
| mtreinish | ianychoi: hmm I don't see a subunit file for the log you linked | 18:24 |
| mtreinish | which would explain why it's not showing up in openstack-health | 18:25 |
| mtreinish | ianychoi, AJaeger: my guess is something changed with the job definition and it's not uploading the testrepository.subunit file for some reason, because the log shows it being generated | 18:30 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Update query for bug 1770211 https://review.openstack.org/595246 | 18:36 |
| openstack | bug 1770211 in OpenStack Compute (nova) "AttachVolumeMultiAttachTest.test_resize_server_with_multiattached_volume intermittently fails with "Unable to detach from guest transient domain."" [Medium,Confirmed] https://launchpad.net/bugs/1770211 | 18:36 |
| AJaeger | mtreinish: mmh, did we break that uploading by accident? ;( | 18:38 |
| *** dtantsur is now known as dtantsur|afk | 18:43 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/system-config master: Add NTP service to base-server https://review.openstack.org/595312 | 18:45 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/system-config master: Add NTP service to base-server https://review.openstack.org/595312 | 18:46 |
| *** tosky has quit IRC | 18:47 | |
| *** jamesmcarthur has quit IRC | 18:50 | |
| corvus | Shrews: that may be it; it doesn't look like our usage of the puppetlabs-ntp module is very sophisticated :) | 18:52 |
| corvus | modules/openstack_project/manifests/server.pp: include '::ntp' | 18:52 |
| Shrews | corvus: yeah. no special config that i saw. 595312 might be all we need | 18:53 |
| Shrews | though i haven't actually tested that | 18:53 |
| *** mriedem_lunch is now known as mriedem | 18:55 | |
| Shrews | corvus: 595312 takes advantage of distro specific tasks to start the service. the other option is to define distro specific variables for the service name and use a single task in the main playbook that uses that var. | 18:56 |
| corvus | Shrews: yeah, i think we've taken the latter approach (variables) when there's a task that applies universally. i don't feel strongly about it, but that might be more consistent if you feel like it. | 18:57 |
| Shrews | corvus: i can change it. | 18:57 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/system-config master: Add NTP service to base-server https://review.openstack.org/595312 | 19:01 |
| Shrews | i like that better, actually | 19:02 |
| fungi | the ntp package in debian/ubuntu starts the daemon automatically and has a default configuration to hit a bunch of machines in the pool.ntp.org set, so yes should "just work" | 19:04 |
| Shrews | how are package names referenced in testinfra? | 19:08 |
| corvus | pabelanger: ^ | 19:09 |
| Shrews | is that distro specific? | 19:09 |
| corvus | https://testinfra.readthedocs.io/en/latest/modules.html#package | 19:09 |
| corvus | that doesn't really help | 19:10 |
| Shrews | no, it does not :( | 19:10 |
| Shrews | i have a feeling it *is* distro specific | 19:10 |
| corvus | Shrews: maybe we'll have to write our own branching logic in the tests, using https://testinfra.readthedocs.io/en/latest/modules.html#testinfra.modules.systeminfo.SystemInfo.distribution | 19:10 |
| Shrews | which complicates testing | 19:10 |
| openstackgerrit | Merged openstack-infra/zuul-website master: Link to Zuul Sphinx extension https://review.openstack.org/591282 | 19:10 |
| Shrews | corvus: probably so | 19:11 |
| pabelanger | Shrews: corvus: Yah, write branching logic for package names | 19:12 |
| Shrews | *vomit* | 19:12 |
| cgoncalves | does Zuul have some sort of cache with jobs defined in all projects? | 19:12 |
| cgoncalves | asking because https://review.openstack.org/#/c/588883/ was merged and yet Zuul complains that job octavia-v2-dsvm-scenario-ubuntu.bionic is defined | 19:13 |
| pabelanger | corvus: will review patches shortly, if that is okay | 19:13 |
| corvus | cgoncalves: yes -- the entire zuul config is in memory and contains jobs from all projects | 19:13 |
| cgoncalves | https://review.openstack.org/#/c/587414/ | 19:13 |
| cgoncalves | corvus, can we flush the cache? :D | 19:13 |
| corvus | cgoncalves: well, it's not a cache per say, it's just live; one sec while i look at those patches | 19:14 |
| corvus | cgoncalves: it's still defined on the stable branches: http://git.openstack.org/cgit/openstack/octavia/tree/zuul.d/jobs.yaml?h=stable/rocky#n54 | 19:17 |
| corvus | cgoncalves: s/stable branches/stable rocky branch/ (just the one branch) | 19:17 |
| corvus | cgoncalves: so if you remove it there, you should be gtg with 587414 | 19:17 |
| corvus | (it would have been helpful if the zuul error message included the branch, sorry about that) | 19:17 |
| rosmaita | fungi: when you have a minute ... glance channel hasn't been logged since 15 Aug, I assume it has something to do with changing the channels to require registered-only users? wondering if you know what's up? | 19:18 |
| fungi | i'll see if i can determine why the bot isn't in that channel any longer | 19:20 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Fix translation subunit upload https://review.openstack.org/595327 | 19:27 |
| AJaeger | mtreinish, ianychoi, eumel8, hope this fixes the health dashboard for failures of translation jobs ^ | 19:27 |
| cgoncalves | corvus, hmm I thought it was not needed as the merged patch and the one open with the zuul error are on master branch | 19:30 |
| AJaeger | cgoncalves: has the other one *only* master branch? | 19:30 |
| *** mriedem has quit IRC | 19:35 | |
| corvus | cgoncalves: as long as there's a job definition in any branch of a project, the job is considered to be defined in that project. and for security reasons, you can't have the same job defined in two separate projects. you can look at it as octavia claiming the name as long as it's defined on any branch in octavia. | 19:36 |
| AJaeger | could I get some reviews on https://review.openstack.org/#/c/592292/ to remove some dead jobs, please? | 19:38 |
| *** mriedem has joined #openstack-infra | 19:38 | |
| *** rosmaita has quit IRC | 19:40 | |
| fungi | infra-root: i've been without broadband access for >2.5 hours at this point, not sure when it's coming back either... if someone has a sec, can they check the meetbot for errors which might explain why it's not joining #openstack-glance for the past week? (see rosmaita's earlier comment) | 19:43 |
| fungi | i realized that wrestling my wireless modem onto the machine which has my openstack ssh keys may not be the best use of my time for now | 19:43 |
| *** pcaruana has quit IRC | 19:47 | |
| Shrews | fungi: logs are on eavesdrop? | 19:47 |
| *** efried_rollin is now known as efried | 19:47 | |
| fungi | yes | 19:49 |
| fungi | probably /var/log/meetbot or something like that | 19:49 |
| Shrews | yeah, i dont see that. | 19:52 |
| Shrews | stuff in /srv/meetbot-openstack | 19:52 |
| *** panda has joined #openstack-infra | 19:55 | |
| cgoncalves | corvus, I see. thanks | 19:56 |
| cgoncalves | johnsom, I'll remove the job definition also from stable/rocky | 19:56 |
| Shrews | ERROR 2018-08-15T02:32:53 supybot Unhandled error message from server: IrcMsg(prefix="barjavel.freenode.net", c | 19:56 |
| Shrews | ommand="477", args=('openstack', '#openstack-glance', 'Cannot join channel (+r) - you need to be identified wit | 19:56 |
| Shrews | h services')) | 19:56 |
| johnsom | cgoncalves yep, sounds good | 19:57 |
| *** e0ne has joined #openstack-infra | 19:57 | |
| fungi | Shrews: odd, why did it think the bot was not identified yet? i wonder if it maybe starts trying to join some channels before the identified state is reflected by the server or something? | 20:01 |
| Shrews | fungi: that may be right. i see a "password accepted" message a second after that one | 20:02 |
| fungi | oof | 20:02 |
| fungi | wonder how to introduce a delay there | 20:02 |
| Shrews | fungi: #openstack-forum and #openstack-trove likely have the same issue | 20:02 |
| fungi | thanks for digging, Shrews | 20:03 |
| Shrews | np! fwiw, logs were hidden in /var/lib | 20:04 |
| pabelanger | corvus: 595252 looks correct from a testing pov. | 20:04 |
| Shrews | also, geesh, large log | 20:04 |
| Shrews | pabelanger: except it cannot install ansible :( | 20:07 |
| pabelanger | Shrews: yah, left comment | 20:07 |
| pabelanger | i think becauae we run playbooks on bridge.o.o as root | 20:07 |
| pabelanger | but in tests we are zuul user | 20:07 |
| pabelanger | so we need to use become: true | 20:07 |
| Shrews | yup | 20:10 |
| *** dpawlik has joined #openstack-infra | 20:12 | |
| *** kota_ has joined #openstack-infra | 20:14 | |
| *** kgiusti has left #openstack-infra | 20:15 | |
| *** mdrabe has quit IRC | 20:22 | |
| *** eernst has quit IRC | 20:26 | |
| *** tonyb has joined #openstack-infra | 20:26 | |
| *** jamesmcarthur has joined #openstack-infra | 20:27 | |
| *** e0ne has quit IRC | 20:27 | |
| *** stevebaker has joined #openstack-infra | 20:31 | |
| *** ericyoung has joined #openstack-infra | 20:33 | |
| *** jamesmcarthur has quit IRC | 20:33 | |
| *** jamesmcarthur has joined #openstack-infra | 20:34 | |
| corvus | it would be really cool if we could add that to the improt_playbook statement | 20:35 |
| *** harlowja has joined #openstack-infra | 20:42 | |
| *** jcoufal has quit IRC | 20:49 | |
| tonyb | Any chance I can get some eyes on https://review.openstack.org/#/c/589007/ it (or something similar) is blocking the u-c bot from running | 20:50 |
| corvus | it looks like if we just add become:true to the bridge playbook that will work. | 20:50 |
| corvus | i tried that on a test node, and got past that, but now there's an issue where the admin group doesn't exist, but apparently we set it in install-ansible | 20:51 |
| corvus | so i wonder how that got created on the real bridge | 20:51 |
| corvus | the users role | 20:52 |
| corvus | so i think we have a bootstrapping problem, since the bridge playbook doesn't do that role | 20:52 |
| openstackgerrit | Merged openstack-infra/storyboard master: Make email public field https://review.openstack.org/589663 | 20:53 |
| corvus | maybe it's okay to include the users role in the bridge playbook? | 20:53 |
| *** trown is now known as trown|outtypewww | 20:54 | |
| fungi | ugh, fiber cut in suffolk, virginia... guess i'm going to be without broadband for a while still | 20:56 |
| *** florianf has quit IRC | 20:57 | |
| *** jamesmcarthur has quit IRC | 20:57 | |
| *** jpena is now known as jpena|off | 20:57 | |
| *** jtomasek has quit IRC | 20:57 | |
| *** felipemonteiro has joined #openstack-infra | 20:58 | |
| *** jamesmcarthur has joined #openstack-infra | 21:03 | |
| *** jamesmcarthur has quit IRC | 21:08 | |
| ianw | mnaser: did you get someone to look at the leaked images? i can take a look today, and dig out some of the stuff i used to cleanup rax if required | 21:11 |
| *** gouthamr has joined #openstack-infra | 21:11 | |
| mnaser | ianw: i kinda just did a manual cleanup myself on our side | 21:11 |
| mnaser | left 2 images which all had dates that made sense | 21:12 |
| mnaser | but i think i deleted maybe over 100 or so | 21:12 |
| mnaser | there is a few extra ones that won't delete.. not sure why | 21:12 |
| mnaser | but thats something on our side | 21:12 |
| ianw | mnaser: ok, np and thanks. i think we've had periods of time where things might have leaked, but there are no known issues in that area now | 21:13 |
| mnaser | ianw: yeah they all seem to be pretty old | 21:14 |
| *** holser_ has joined #openstack-infra | 21:18 | |
| fungi | also as i noted earlier, i think we had vexxhost offline in nodepool at the time we did the big cleanup around v3 upgrade, which explains why we probably missed deleting them there | 21:21 |
| openstackgerrit | Jarek Lukow proposed openstack-infra/zuul master: Add missing parentheses to title invocation https://review.openstack.org/595361 | 21:21 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: WIP: Add system-config-run-base job https://review.openstack.org/595252 | 21:23 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Allow bridge playbook to run under zuul https://review.openstack.org/595363 | 21:23 |
| *** felipemonteiro has quit IRC | 21:23 | |
| corvus | pabelanger, Shrews: ^ i made that as a separate change which should be safe to land now. then i made a bunch of fixes to the new job. | 21:24 |
| openstackgerrit | Merged openstack-infra/system-config master: Support puppet5 for bionic https://review.openstack.org/589007 | 21:26 |
| *** felipemonteiro has joined #openstack-infra | 21:26 | |
| *** dansmith is now known as htimsnad | 21:34 | |
| *** eernst has joined #openstack-infra | 21:42 | |
| eernst | fungi: ping/ | 21:43 |
| eernst | ? | 21:43 |
| eernst | clarkb: mentioned he's 90% ooo and you may be able to help if we had issues this week.... if you can, can you PTAL @ https://github.com/kata-containers/tests/pull/634 ? | 21:43 |
| fungi | eernst: what's up? usually better to just ask whatever you're going to ask | 21:43 |
| fungi | looking! | 21:43 |
| eernst | thx. | 21:44 |
| fungi | eernst: can you link to a pr zuul has reported on? i just want to check that i pick the correct ansible vars out of the inventory | 21:47 |
| eernst | here's one: https://github.com/kata-containers/runtime/pull/617 | 21:48 |
| fungi | thanks! | 21:48 |
| *** boden has quit IRC | 21:49 | |
| *** agopi has quit IRC | 21:52 | |
| fungi | eernst: how is "target_branch" (the branch where the PR is going to be merged) expected to differ from "branch" (the branch that we will be testing)? | 21:52 |
| *** dpawlik has quit IRC | 21:53 | |
| fungi | or maybe restated, if you're testing a different branch than you're merging into, how are you signaling which branch to test? | 21:53 |
| eernst | We are managing 3 branches now. | 21:54 |
| eernst | And merging into three: master, stable-1, stable-2 | 21:54 |
| eernst | (more or less) | 21:54 |
| corvus | may i make a suggestion? perform no git operations at all if you are running under zuul. | 21:54 |
| eernst | Well, let me clarify here. | 21:54 |
| eernst | We manage three branches independently. | 21:54 |
| eernst | but we are merging changes, generally, into stable and master as appropriate | 21:55 |
| *** mriedem has quit IRC | 21:55 | |
| eernst | so, testing and merging would be done for the same branch. | 21:55 |
| eernst | Based on top of stable-1.2 for example in order to merge into stable-1.2 | 21:56 |
| fungi | i guess what's unclear to me is why the separate target_branch and branch variables if they're going to have the same value? | 21:56 |
| *** felipemonteiro has quit IRC | 21:57 | |
| fungi | but also as corvus says, you shouldn't need to explicitly fetch the pr or branch. zuul will push those onto the job node for you | 21:58 |
| fungi | basically zull will give you the repo checked out to the target branch tip with your pr and any of its declared dependencies applied | 21:59 |
| fungi | er, suul | 21:59 |
| fungi | zuul | 21:59 |
| eernst | While I tried to get Salvador in this channel, may be easiest to just clarify this on github pr itself? | 21:59 |
| fungi | i seem to be entirely unable to type on this machine | 21:59 |
| eernst | :) | 21:59 |
| corvus | fungi, eernst: under zuul, most of those git operations aren't going to do anything; some may fail. it might be easier and safer to just wrap all of them with "if not running under zuul". for all of the cases you have described, zuul will automatically have the repo checked out appropriately. | 21:59 |
| fungi | sure, i'll re-ask on the pr | 21:59 |
| eernst | thx fungi | 21:59 |
| *** quite has quit IRC | 22:05 | |
| *** quite has joined #openstack-infra | 22:08 | |
| corvus | well, that got further: http://logs.openstack.org/52/595252/4/check/system-config-run-base/e28db22/job-output.txt.gz#_2018-08-22_22_02_31_999767 | 22:14 |
| *** felipemonteiro has joined #openstack-infra | 22:22 | |
| fungi | #status log manually deleted redundant branch "stable/queen" (previously at e1146cc01bce2c1bd6eecb08d92281297218f884) from openstack/networking-vsphere as requested in http://lists.openstack.org/pipermail/openstack-infra/2018-August/006064.html | 22:22 |
| openstackstatus | fungi: finished logging | 22:22 |
| *** holser_ has quit IRC | 22:26 | |
| *** rcernin has joined #openstack-infra | 22:34 | |
| *** holser_ has joined #openstack-infra | 22:34 | |
| *** felipemonteiro has quit IRC | 22:37 | |
| ianw | mordred: https://imgur.com/a/8NyGSTO here's a graph of run_all.sh runtimes over ~ last 20 hours just because, i dunno, because we can | 22:45 |
| *** ChanServ has quit IRC | 22:49 | |
| openstackgerrit | Kendall Nelson proposed openstack-infra/irc-meetings master: Update FC SIG Meeting https://review.openstack.org/595377 | 22:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add write-inventory role https://review.openstack.org/595378 | 23:03 |
| *** ChanServ has joined #openstack-infra | 23:03 | |
| *** barjavel.freenode.net sets mode: +o ChanServ | 23:03 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: WIP: Add system-config-run-base job https://review.openstack.org/595252 | 23:05 |
| *** eernst has quit IRC | 23:08 | |
| *** jamesdenton has quit IRC | 23:09 | |
| openstackgerrit | Kendall Nelson proposed openstack-infra/irc-meetings master: Update FC SIG Meeting https://review.openstack.org/595377 | 23:09 |
| *** holser_ has quit IRC | 23:10 | |
| *** jamesmcarthur has joined #openstack-infra | 23:13 | |
| *** tpsilva has quit IRC | 23:16 | |
| *** slagle has joined #openstack-infra | 23:17 | |
| *** rpioso is now known as rpioso|afk | 23:18 | |
| *** felipemonteiro has joined #openstack-infra | 23:26 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add write-inventory role https://review.openstack.org/595378 | 23:32 |
| *** jamesmcarthur has quit IRC | 23:35 | |
| *** rlandy has quit IRC | 23:39 | |
| *** sthussey has quit IRC | 23:39 | |
| corvus | ianw, Shrews, pabelanger, fungi: i'm stumped by http://logs.openstack.org/52/595252/5/check/system-config-run-base/c405df3/job-output.txt.gz#_2018-08-22_23_39_44_503305 and about to eod. feel free to poke at it or let me know if you have any ideas. i'll pick it back up tomorrow. | 23:46 |
| fungi | not sure if i can figure it out, but happy to take a look | 23:47 |
| *** gouthamr has quit IRC | 23:48 | |
| ianw | hrm, yeah have to think about what keys and authorized_keys are coming from where ... it's pretty nested! | 23:49 |
| Shrews | corvus: that's odd | 23:50 |
| *** felipemonteiro has quit IRC | 23:55 | |
| *** dtruong has joined #openstack-infra | 23:56 | |
| *** r-daneel has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!