| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: static: enable using a single host with different user or port https://review.opendev.org/659209 | 00:01 |
|---|---|---|
| *** paladox has quit IRC | 00:02 | |
| *** paladox has joined #openstack-infra | 00:02 | |
| *** paladox has quit IRC | 00:02 | |
| *** paladox has joined #openstack-infra | 00:02 | |
| *** jamesmcarthur has joined #openstack-infra | 00:07 | |
| *** paladox has quit IRC | 00:08 | |
| *** paladox has joined #openstack-infra | 00:08 | |
| *** markvoelker has quit IRC | 00:09 | |
| openstackgerrit | Merged openstack/project-config master: Revert "Revert "opendev.org mirrors: switch out DFW, switch in IAD"" https://review.opendev.org/665037 | 00:09 |
| *** paladox has quit IRC | 00:09 | |
| *** weifan has joined #openstack-infra | 00:09 | |
| *** paladox has joined #openstack-infra | 00:10 | |
| *** weifan has quit IRC | 00:15 | |
| *** weifan has joined #openstack-infra | 00:21 | |
| *** goldyfruit has quit IRC | 00:24 | |
| *** goldyfruit has joined #openstack-infra | 00:24 | |
| *** weifan has quit IRC | 00:27 | |
| *** hwoarang has quit IRC | 00:30 | |
| *** liuyulong has quit IRC | 00:31 | |
| *** weifan has joined #openstack-infra | 00:32 | |
| *** weifan has quit IRC | 00:35 | |
| *** lseki has joined #openstack-infra | 00:38 | |
| *** yamamoto has quit IRC | 00:48 | |
| *** jamesmcarthur has quit IRC | 00:49 | |
| *** jamesmcarthur has joined #openstack-infra | 00:50 | |
| *** olaph has joined #openstack-infra | 00:53 | |
| ianw | ok, access starting to happen from ^ | 00:55 |
| *** jamesmcarthur has quit IRC | 00:55 | |
| *** rf0lc0 has joined #openstack-infra | 00:57 | |
| *** mattw4 has quit IRC | 01:05 | |
| *** markvoelker has joined #openstack-infra | 01:05 | |
| ianw | one difference here is that cachefilesd isn't running atm actually | 01:06 |
| ianw | i'll let it run like this for a hour, say, and then enabled cachefiles module/dameon | 01:07 |
| *** dychen has joined #openstack-infra | 01:09 | |
| *** dchen has quit IRC | 01:12 | |
| *** rfarr__ has quit IRC | 01:13 | |
| *** lifeless has joined #openstack-infra | 01:14 | |
| *** _erlon_ has quit IRC | 01:17 | |
| *** jamesmcarthur has joined #openstack-infra | 01:20 | |
| *** weifan has joined #openstack-infra | 01:20 | |
| *** weifan has quit IRC | 01:25 | |
| *** happyhemant has quit IRC | 01:28 | |
| *** dingyichen has joined #openstack-infra | 01:31 | |
| *** dychen has quit IRC | 01:34 | |
| *** smarcet has joined #openstack-infra | 01:36 | |
| *** michael-beaver has joined #openstack-infra | 01:38 | |
| *** markvoelker has quit IRC | 01:39 | |
| *** ykarel|away has joined #openstack-infra | 01:54 | |
| *** ykarel|away is now known as ykarel | 02:04 | |
| *** rlandy|ruck|bbl is now known as rlandy|ruck | 02:06 | |
| *** dayou_ has quit IRC | 02:08 | |
| *** gyee has quit IRC | 02:11 | |
| cloudnull | anyone around able to give https://review.opendev.org/#/c/664932 a review? | 02:11 |
| *** rf0lc0 has quit IRC | 02:13 | |
| *** whoami-rajat has joined #openstack-infra | 02:14 | |
| *** jamesmcarthur has quit IRC | 02:15 | |
| *** jamesmcarthur has joined #openstack-infra | 02:15 | |
| *** jamesmcarthur has quit IRC | 02:18 | |
| *** ykarel is now known as ykarel|afk | 02:19 | |
| *** jamesmcarthur has joined #openstack-infra | 02:20 | |
| ianw | ok, cachefilesd module loaded and enabled (somehow it didn't start withotu commenting out the selinux check ... it does start in CI and when i tried it before; must be something to do with restarting when there's already a cache i'd say) | 02:21 |
| *** dayou_ has joined #openstack-infra | 02:22 | |
| ianw | for reference, the cache was at about 500m before restart; in case we see issues in a similar place | 02:23 |
| *** gregoryo has joined #openstack-infra | 02:25 | |
| *** diablo_rojo has quit IRC | 02:27 | |
| *** rlandy|ruck has quit IRC | 02:32 | |
| *** jamesmcarthur has quit IRC | 02:33 | |
| *** jamesmcarthur has joined #openstack-infra | 02:33 | |
| *** markvoelker has joined #openstack-infra | 02:36 | |
| *** auristor has quit IRC | 02:38 | |
| *** lifeless has quit IRC | 02:38 | |
| cloudnull | thanks ianw! | 02:43 |
| *** lifeless has joined #openstack-infra | 02:44 | |
| *** weifan has joined #openstack-infra | 02:45 | |
| *** weifan has quit IRC | 02:48 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:48 | |
| *** weifan has joined #openstack-infra | 02:48 | |
| *** weifan has quit IRC | 02:49 | |
| openstackgerrit | Merged zuul/zuul master: Update quickstart nodepool node to python3 https://review.opendev.org/658486 | 02:52 |
| *** bhavikdbavishi has quit IRC | 02:53 | |
| *** auristor has joined #openstack-infra | 02:54 | |
| *** jamesmcarthur has quit IRC | 03:01 | |
| *** jamesmcarthur has joined #openstack-infra | 03:02 | |
| *** rascasoft has quit IRC | 03:03 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:03 | |
| *** jamesmcarthur has quit IRC | 03:04 | |
| *** jamesmcarthur has joined #openstack-infra | 03:04 | |
| *** lseki has quit IRC | 03:06 | |
| *** rascasoft has joined #openstack-infra | 03:06 | |
| *** markvoelker has quit IRC | 03:09 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Add more test coverage on using python-path https://review.opendev.org/659812 | 03:31 |
| *** smarcet has quit IRC | 03:32 | |
| *** jamesmcarthur has quit IRC | 03:43 | |
| *** ramishra has joined #openstack-infra | 03:43 | |
| *** jamesmcarthur has joined #openstack-infra | 03:43 | |
| *** michael-beaver has quit IRC | 03:48 | |
| *** ricolin has joined #openstack-infra | 03:50 | |
| *** weifan has joined #openstack-infra | 03:53 | |
| *** dave-mccowan has quit IRC | 03:54 | |
| *** weifan has quit IRC | 03:57 | |
| *** jamesmcarthur has quit IRC | 03:58 | |
| *** ykarel|afk is now known as ykarel | 04:00 | |
| *** ramishra has quit IRC | 04:02 | |
| *** ramishra has joined #openstack-infra | 04:03 | |
| *** yamamoto has joined #openstack-infra | 04:05 | |
| *** kaisers has quit IRC | 04:06 | |
| *** markvoelker has joined #openstack-infra | 04:06 | |
| *** igordc has joined #openstack-infra | 04:07 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] kernel update for mirrors https://review.opendev.org/665057 | 04:13 |
| *** bhavikdbavishi has quit IRC | 04:19 | |
| *** bhavikdbavishi has joined #openstack-infra | 04:20 | |
| *** kaisers has joined #openstack-infra | 04:21 | |
| *** whoami-rajat has quit IRC | 04:23 | |
| *** kjackal has joined #openstack-infra | 04:26 | |
| *** markvoelker has quit IRC | 04:40 | |
| *** udesale has joined #openstack-infra | 04:41 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] kernel update for mirrors https://review.opendev.org/665057 | 04:42 |
| openstackgerrit | Merged openstack/project-config master: Retire independent ansible role https://review.opendev.org/664932 | 04:49 |
| *** weifan has joined #openstack-infra | 04:51 | |
| *** weifan has quit IRC | 04:51 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] kernel update for mirrors https://review.opendev.org/665057 | 04:51 |
| *** janki has joined #openstack-infra | 04:52 | |
| *** pcaruana|afk| has joined #openstack-infra | 05:01 | |
| *** pcaruana|afk| has quit IRC | 05:04 | |
| *** pcaruana has joined #openstack-infra | 05:04 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] kernel update for mirrors https://review.opendev.org/665057 | 05:12 |
| *** cmurphy is now known as cmurphy_afk | 05:14 | |
| *** e0ne has joined #openstack-infra | 05:21 | |
| *** ricolin has quit IRC | 05:36 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] kernel update for mirrors https://review.opendev.org/665057 | 05:36 |
| *** markvoelker has joined #openstack-infra | 05:37 | |
| *** takamatsu has joined #openstack-infra | 05:42 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] kernel update for mirrors https://review.opendev.org/665057 | 05:58 |
| *** kopecmartin|off is now known as kopecmartin | 06:01 | |
| *** spsurya has joined #openstack-infra | 06:08 | |
| *** markvoelker has quit IRC | 06:10 | |
| *** zbr|flow is now known as zbr|ooo | 06:11 | |
| *** zhurong has quit IRC | 06:14 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] kernel update for mirrors https://review.opendev.org/665057 | 06:15 |
| *** dingyichen has quit IRC | 06:17 | |
| *** ykarel_ has joined #openstack-infra | 06:19 | |
| *** zhurong has joined #openstack-infra | 06:20 | |
| *** dchen has joined #openstack-infra | 06:21 | |
| *** ykarel has quit IRC | 06:21 | |
| *** e0ne has quit IRC | 06:24 | |
| *** kjackal has quit IRC | 06:30 | |
| *** iurygregory has joined #openstack-infra | 06:44 | |
| *** pgaxatte has joined #openstack-infra | 06:49 | |
| *** xek_ has joined #openstack-infra | 06:53 | |
| yoctozepto | https://opendev.org/zuul/zuul/search?q=tox-docs gitea search is lame :/ | 06:56 |
| *** ykarel_ is now known as ykarel | 06:57 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 07:00 |
| *** rpittau|afk is now known as rpittau | 07:00 | |
| *** ianychoi has joined #openstack-infra | 07:02 | |
| *** ginopc has joined #openstack-infra | 07:04 | |
| *** markvoelker has joined #openstack-infra | 07:08 | |
| *** dtantsur|afk is now known as dtantsur | 07:10 | |
| *** slaweq has joined #openstack-infra | 07:12 | |
| *** tosky has joined #openstack-infra | 07:13 | |
| *** markvoelker has quit IRC | 07:13 | |
| *** igordc has quit IRC | 07:15 | |
| *** ccamacho has quit IRC | 07:15 | |
| *** jbadiapa has quit IRC | 07:17 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: Update test coverage for openSUSE/-minimal to 15.1 https://review.opendev.org/660137 | 07:20 |
| *** tesseract has joined #openstack-infra | 07:21 | |
| *** trident has quit IRC | 07:31 | |
| *** takamatsu has quit IRC | 07:32 | |
| *** trident has joined #openstack-infra | 07:34 | |
| *** ykarel is now known as ykarel|lunch | 07:34 | |
| *** kjackal has joined #openstack-infra | 07:37 | |
| *** rascasoft has quit IRC | 07:42 | |
| *** rascasoft has joined #openstack-infra | 07:43 | |
| *** jpena|off is now known as jpena | 07:43 | |
| *** pkopec has joined #openstack-infra | 07:51 | |
| *** pkopec has quit IRC | 07:53 | |
| *** pkopec has joined #openstack-infra | 07:53 | |
| *** markvoelker has joined #openstack-infra | 08:10 | |
| *** whoami-rajat has joined #openstack-infra | 08:10 | |
| *** ykarel|lunch is now known as ykarel | 08:14 | |
| *** markvoelker has quit IRC | 08:14 | |
| *** lucasagomes has joined #openstack-infra | 08:14 | |
| *** ralonsoh has joined #openstack-infra | 08:15 | |
| *** takamatsu has joined #openstack-infra | 08:19 | |
| *** gregoryo has quit IRC | 08:23 | |
| *** rcernin has quit IRC | 08:23 | |
| ianw | corvus / clarkb: updated few notes on -> https://etherpad.openstack.org/p/opendev-mirror-afs | 08:24 |
| *** jpich has joined #openstack-infra | 08:24 | |
| ianw | in short, IAD seems to be running fine with 5.1.9 kernel+kafs+cachefilesd ok ... cpu load is low, /var/cache/openafs (yes, using old path) is slowly increasing, requests appear to be moving through | 08:25 |
| ianw | i have started looking at installing the mainline kernels in -> https://review.opendev.org/#/c/665057/ ... if you want to investigate why the rebooted nodes seem to fail to start ntpd feel free :) | 08:25 |
| ianw | and i have also built openafs 1.8.3 bionic packages @ https://launchpad.net/~openstack-ci-core/+archive/ubuntu/openafs ... but i'd say if the kafs thing is working out, this is currently our backup plan | 08:26 |
| *** ccamacho has joined #openstack-infra | 08:27 | |
| *** priteau has joined #openstack-infra | 08:28 | |
| *** jaosorior has joined #openstack-infra | 08:29 | |
| *** ociuhandu has joined #openstack-infra | 08:31 | |
| *** ccamacho has quit IRC | 08:32 | |
| *** ccamacho has joined #openstack-infra | 08:33 | |
| *** ociuhandu has quit IRC | 08:36 | |
| *** yboaron_ has joined #openstack-infra | 08:37 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 08:38 |
| *** imacdonn has quit IRC | 08:38 | |
| *** imacdonn has joined #openstack-infra | 08:39 | |
| *** dpawlik has joined #openstack-infra | 08:40 | |
| *** ociuhandu has joined #openstack-infra | 08:42 | |
| *** tkajinam has quit IRC | 08:51 | |
| *** ykarel is now known as ykarel|afk | 08:53 | |
| *** ykarel|afk has quit IRC | 09:00 | |
| *** gfidente has joined #openstack-infra | 09:08 | |
| *** markvoelker has joined #openstack-infra | 09:10 | |
| *** ykarel|afk has joined #openstack-infra | 09:13 | |
| *** ykarel|afk is now known as ykarel | 09:13 | |
| *** markvoelker has quit IRC | 09:15 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Allow operator to generate auth tokens through the CLI https://review.opendev.org/636197 | 09:20 |
| *** e0ne has joined #openstack-infra | 09:26 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul CLI: allow access via REST https://review.opendev.org/636315 | 09:31 |
| *** derekh has joined #openstack-infra | 09:31 | |
| *** gmann has quit IRC | 09:33 | |
| *** kjackal has quit IRC | 09:40 | |
| *** kjackal has joined #openstack-infra | 09:40 | |
| *** slaweq has quit IRC | 09:45 | |
| *** slaweq has joined #openstack-infra | 09:46 | |
| *** Lucas_Gray has joined #openstack-infra | 09:52 | |
| *** udesale has quit IRC | 09:59 | |
| *** udesale has joined #openstack-infra | 09:59 | |
| *** yamamoto has quit IRC | 10:09 | |
| *** jbadiapa has joined #openstack-infra | 10:11 | |
| *** markvoelker has joined #openstack-infra | 10:11 | |
| *** ykarel is now known as ykarel|afk | 10:14 | |
| *** markvoelker has quit IRC | 10:15 | |
| *** Adri2000 has quit IRC | 10:20 | |
| *** kjackal has quit IRC | 10:28 | |
| *** gfidente has quit IRC | 10:40 | |
| *** sthussey has quit IRC | 10:40 | |
| *** bhavikdbavishi has quit IRC | 10:43 | |
| *** yamamoto has joined #openstack-infra | 10:45 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 10:49 |
| *** ykarel|afk is now known as ykarel | 10:49 | |
| *** whoami-rajat has quit IRC | 10:50 | |
| *** priteau has quit IRC | 10:54 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 10:58 |
| *** jpich has quit IRC | 10:59 | |
| *** yamamoto has quit IRC | 10:59 | |
| *** yamamoto has joined #openstack-infra | 10:59 | |
| *** jpich has joined #openstack-infra | 10:59 | |
| *** jpena is now known as jpena|lunch | 11:01 | |
| *** kjackal has joined #openstack-infra | 11:02 | |
| *** _erlon_ has joined #openstack-infra | 11:05 | |
| *** gmann has joined #openstack-infra | 11:09 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 11:20 |
| *** nicolasbock has joined #openstack-infra | 11:21 | |
| *** yamamoto has quit IRC | 11:24 | |
| *** yamamoto has joined #openstack-infra | 11:28 | |
| *** yamamoto has quit IRC | 11:40 | |
| *** yamamoto has joined #openstack-infra | 11:43 | |
| *** yamamoto has quit IRC | 11:43 | |
| *** yamamoto has joined #openstack-infra | 11:43 | |
| *** jtomasek_ has quit IRC | 11:45 | |
| *** jtomasek has joined #openstack-infra | 11:46 | |
| *** jtomasek has quit IRC | 11:47 | |
| *** jtomasek has joined #openstack-infra | 11:47 | |
| *** yamamoto has quit IRC | 11:48 | |
| *** sshnaidm has quit IRC | 11:52 | |
| *** yamamoto has joined #openstack-infra | 11:53 | |
| *** bobh has joined #openstack-infra | 11:54 | |
| *** rh-jelabarre has joined #openstack-infra | 11:56 | |
| *** sshnaidm has joined #openstack-infra | 11:56 | |
| *** yamamoto has quit IRC | 11:57 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:03 | |
| *** rlandy has joined #openstack-infra | 12:05 | |
| *** rlandy is now known as rlandy|ruck | 12:06 | |
| *** rf0lc0 has joined #openstack-infra | 12:07 | |
| *** bhavikdbavishi has quit IRC | 12:08 | |
| *** gfidente has joined #openstack-infra | 12:10 | |
| *** markvoelker has joined #openstack-infra | 12:13 | |
| *** Adri2000 has joined #openstack-infra | 12:13 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:15 | |
| *** markvoelker has quit IRC | 12:17 | |
| *** spsurya has quit IRC | 12:18 | |
| *** yamamoto has joined #openstack-infra | 12:21 | |
| *** goldyfruit has quit IRC | 12:24 | |
| auristor | ianw: if you are having issues with kafs please reach out to dhowells in the #openafs channel | 12:25 |
| *** janki has quit IRC | 12:26 | |
| *** pkopec has quit IRC | 12:36 | |
| *** pkopec has joined #openstack-infra | 12:37 | |
| *** jpena|lunch is now known as jpena | 12:40 | |
| *** yamamoto has quit IRC | 12:40 | |
| *** aaronsheffield has joined #openstack-infra | 12:56 | |
| *** apetrich has quit IRC | 12:56 | |
| *** apetrich has joined #openstack-infra | 12:57 | |
| *** ekultails has joined #openstack-infra | 12:58 | |
| *** yamamoto has joined #openstack-infra | 12:59 | |
| *** pcaruana has quit IRC | 13:01 | |
| *** chandankumar is now known as raukadah | 13:01 | |
| *** zhurong has quit IRC | 13:02 | |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 13:02 | |
| *** ykarel is now known as ykarel|afk | 13:04 | |
| *** apetrich has quit IRC | 13:06 | |
| *** ekultails has quit IRC | 13:07 | |
| *** ekultails has joined #openstack-infra | 13:08 | |
| *** yamamoto has quit IRC | 13:08 | |
| *** yamamoto has joined #openstack-infra | 13:09 | |
| *** yamamoto has quit IRC | 13:09 | |
| *** mriedem has joined #openstack-infra | 13:10 | |
| *** ykarel|afk has quit IRC | 13:10 | |
| *** jaosorior has quit IRC | 13:11 | |
| *** smarcet has joined #openstack-infra | 13:11 | |
| *** markvoelker has joined #openstack-infra | 13:14 | |
| *** boden has joined #openstack-infra | 13:14 | |
| boden | anyone know why I might be having errors resolving the conditional versions of sphinx in the requirements-check job: 'markers': "python_version=='2.7'" does not match "python_version>='3.4'" | 13:15 |
| boden | http://logs.openstack.org/96/664596/9/check/requirements-check/06e443e/job-output.txt.gz#_2019-06-12_20_47_46_376574 | 13:15 |
| boden | probably something simple, but I can't seem to find it | 13:15 |
| *** markvoelker has quit IRC | 13:18 | |
| *** lseki has joined #openstack-infra | 13:18 | |
| *** roman_g has joined #openstack-infra | 13:19 | |
| *** bobh has quit IRC | 13:21 | |
| *** apetrich has joined #openstack-infra | 13:22 | |
| *** goldyfruit has joined #openstack-infra | 13:22 | |
| *** udesale has quit IRC | 13:22 | |
| *** udesale has joined #openstack-infra | 13:24 | |
| boden | maybe it's the backleveled pbr.. | 13:25 |
| *** bhavikdbavishi has quit IRC | 13:27 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 13:27 |
| *** udesale has quit IRC | 13:29 | |
| frickler | boden: hmm, iiuc you need to match the entry in test-requirements.txt to what you add in doc/reqs.txt | 13:30 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul CLI: allow access via REST https://review.opendev.org/636315 | 13:34 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Add Authorization Rules configuration https://review.opendev.org/639855 | 13:34 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Web: plug the authorization engine https://review.opendev.org/640884 | 13:35 |
| *** yamamoto has joined #openstack-infra | 13:37 | |
| *** dave-mccowan has joined #openstack-infra | 13:41 | |
| boden | frickler hm ok, maybe that's it | 13:41 |
| *** yamamoto has quit IRC | 13:42 | |
| *** jamesmcarthur has joined #openstack-infra | 13:43 | |
| frickler | infra-root: someone complained in #openstack about SSL errors when connecting to eavesdrop.o.o, maybe we should disable port 443 in apache2 when it serves no content. or use that as the next target for the LE deployment? the current virtualhost definition is indeed only for port 80 | 13:44 |
| fungi | ahh, yep, responded there as well but i think we concluded a while back that we should unconfigure that listening port for apache until the service is redeployed with letsencrypt on the new domain | 13:45 |
| *** ricolin has joined #openstack-infra | 13:45 | |
| frickler | infra-root: also please note that due to some surgery I'll be mostly offline for about a week or so starting tomorrow | 13:45 |
| *** goldyfruit has quit IRC | 13:46 | |
| *** goldyfruit has joined #openstack-infra | 13:46 | |
| frickler | fungi: yeah, my response there was written thinking that this was happening in the -infra channel | 13:46 |
| fungi | yoctozepto: yep, we're still also maintaining the hound instance at codesearch.openstack.org and the gitea maintainers are aware search needs a better implementation. https://github.com/go-gitea/gitea/issues/6648 is probably the most relevant issue about it (we're hoping for something along those lines, but if you like hacking in go maybe you want to help them out with it?) | 13:47 |
| *** tosky has quit IRC | 13:48 | |
| *** ykarel|afk has joined #openstack-infra | 13:48 | |
| yoctozepto | fungi: yeah, codesearch is <3 | 13:50 |
| fungi | well, it has its own pains | 13:50 |
| yoctozepto | so far I only noticed it does not remember selected projects when refreshed | 13:51 |
| fungi | also only searches master branch, plus needs an outage to add/remove/rename repositories and then takes quite a while to reindex on start | 13:51 |
| frickler | fungi: yoctozepto: comparing with http://codesearch.openstack.org/?q=tox-docs&i=nope&files=&repos=zuul/zuul it seems that the gitea search is ignoring "hidden" files, i.e. starting with a dot, but I couldn't find an issue for that, do you know whether there is one? | 13:51 |
| yoctozepto | as the search resets them | 13:51 |
| *** smarcet has quit IRC | 13:51 | |
| *** tosky has joined #openstack-infra | 13:52 | |
| *** rossella_s has quit IRC | 13:52 | |
| fungi | frickler: i haven't seen one. odds are that behavior will change in the indexing refactor they're planning, but might be good to file it just so they take that into account for the new design | 13:52 |
| *** apetrich has quit IRC | 13:56 | |
| *** apetrich has joined #openstack-infra | 13:56 | |
| *** gfidente has quit IRC | 13:59 | |
| *** rossella_s has joined #openstack-infra | 13:59 | |
| *** gfidente has joined #openstack-infra | 14:01 | |
| *** ykarel|afk is now known as ykarel | 14:03 | |
| clarkb | boden note those marker messages are benign. It is just saying that the entry for python >= 3.4 does not apply when executed under python 2.7 | 14:05 |
| *** igordc has joined #openstack-infra | 14:05 | |
| *** szaher has quit IRC | 14:05 | |
| frickler | boden: tested locally that amending test-requirements.txt fixes the issue. you also need to use ">=1.6.5" instead of 1.6.2 matching lower-constraints | 14:06 |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 14:10 | |
| *** xek_ has quit IRC | 14:13 | |
| *** markvoelker has joined #openstack-infra | 14:14 | |
| *** sreejithp has joined #openstack-infra | 14:17 | |
| *** szaher has joined #openstack-infra | 14:19 | |
| *** markvoelker has quit IRC | 14:19 | |
| *** gfidente has quit IRC | 14:19 | |
| *** ricolin has quit IRC | 14:21 | |
| *** gfidente has joined #openstack-infra | 14:21 | |
| *** yamamoto has joined #openstack-infra | 14:23 | |
| *** igordc has quit IRC | 14:42 | |
| boden | frickler clarkb thank you for the tips! | 14:42 |
| openstackgerrit | Stephen Finucane proposed openstack/pbr master: Resolve some issue with tox.ini, setup.cfg https://review.opendev.org/660297 | 14:46 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 14:59 |
| *** yboaron_ has quit IRC | 15:00 | |
| *** tosky has quit IRC | 15:01 | |
| *** sthussey has joined #openstack-infra | 15:02 | |
| *** tosky has joined #openstack-infra | 15:04 | |
| corvus | ianw, clarkb: ntpd fails to start because systemd starts it before unbound and it can't resolve hostnames | 15:07 |
| corvus | i think this may happen on our production machines. | 15:08 |
| *** igordc has joined #openstack-infra | 15:23 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 15:24 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 15:31 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 15:32 |
| openstackgerrit | Stephen Finucane proposed openstack/openstack-zuul-jobs master: Add support for building PDFs https://review.opendev.org/664555 | 15:34 |
| *** dpawlik has quit IRC | 15:35 | |
| *** yamamoto has quit IRC | 15:35 | |
| *** gyee has joined #openstack-infra | 15:35 | |
| *** ykarel is now known as ykarel|away | 15:35 | |
| *** pgaxatte has quit IRC | 15:36 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Split plugin tests into two https://review.opendev.org/665161 | 15:41 |
| *** bhavikdbavishi has joined #openstack-infra | 15:45 | |
| corvus | Shrews: do you want to handle restarting nodepool, and i can take care of restarting zuul? | 15:46 |
| Shrews | corvus: sure thing | 15:47 |
| Shrews | clarkb: you restarted builders recently, right? | 15:47 |
| Shrews | looks like nb01 and nb02 restarted on June 10. nb03 on June 03 | 15:50 |
| corvus | restarting zuul now | 15:51 |
| Shrews | i think builders are fine. will do launchers now | 15:51 |
| *** bhavikdbavishi1 has joined #openstack-infra | 15:53 | |
| *** diablo_rojo has joined #openstack-infra | 15:54 | |
| *** bhavikdbavishi has quit IRC | 15:54 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 15:54 | |
| clarkb | Shrews: yup builders are fine | 15:56 |
| clarkb | corvus: is zuul restart keeping the repl change? | 15:57 |
| *** ginopc has quit IRC | 15:57 | |
| corvus | clarkb: yes | 15:57 |
| Shrews | #status log Restarted nodepool launchers on commit 3412764a985b511cdc6b70dc801ffdb357ec02c2 | 15:58 |
| openstackstatus | Shrews: finished logging | 15:58 |
| Shrews | corvus: done | 15:58 |
| corvus | scheduler is back up, executors are catching up | 15:59 |
| *** ofosos has joined #openstack-infra | 16:03 | |
| corvus | executors are up | 16:03 |
| clarkb | unrelated to hte zuul restart the iad kafs mirror still seems to be working | 16:04 |
| *** kjackal has quit IRC | 16:04 | |
| *** sshnaidm is now known as sshnaidm|off | 16:04 | |
| corvus | #status log restarted all of zuul on 7e45f84f056b3fa021aae1eecb0c23d9055656f3 (with repl change cherry-picked onto scheduler) | 16:04 |
| openstackstatus | corvus: finished logging | 16:04 |
| *** iurygregory has quit IRC | 16:05 | |
| *** e0ne has quit IRC | 16:07 | |
| *** jbadiapa has quit IRC | 16:07 | |
| *** ccamacho has quit IRC | 16:08 | |
| corvus | re-enqueue finished | 16:09 |
| corvus | infra-root: we have not cleared our zuul job backlog since this weekend. | 16:09 |
| fungi | yeesh | 16:10 |
| corvus | http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1&fullscreen&panelId=17&from=now-7d&to=now | 16:10 |
| fungi | yeah, seems we *nearly* caught up early utc tuesday but quickly lost ground again | 16:11 |
| fungi | looks like throughput is decent at least | 16:12 |
| fungi | hovering around 700-800jph the entire time | 16:12 |
| *** olaph has quit IRC | 16:12 | |
| corvus | we'll save like 30s per job when we drop the zuul cloner shim :) | 16:12 |
| fungi | yeah | 16:12 |
| fungi | 11 days | 16:12 |
| *** andreykurilin has quit IRC | 16:13 | |
| Shrews | hrm, that grafana graph refuses to load for me | 16:13 |
| corvus | that'll be like another 144 jobs per day we can run :) | 16:13 |
| *** rpittau is now known as rpittau|afk | 16:13 | |
| *** andreykurilin has joined #openstack-infra | 16:14 | |
| clarkb | I'm going through and trying to count the number of certs I need (and names per cert) and am I just misremembering that we had one giant cert with a bunch of altnames? not finding it anymore | 16:14 |
| *** dims has quit IRC | 16:14 | |
| corvus | Shrews: weird... can you go to grafana.openstack.org and load the "zuul status" dashboard? | 16:14 |
| corvus | clarkb: static and files are the places where i would expect that | 16:15 |
| Shrews | corvus: tried that. none of the graphs are loading, even after a FF restart | 16:15 |
| clarkb | oh its static | 16:15 |
| corvus | clarkb: (but i think on files, we have a bunch of certs with SNI) | 16:15 |
| *** markvoelker has joined #openstack-infra | 16:16 | |
| *** lucasagomes has quit IRC | 16:16 | |
| clarkb | I think files may be different cert per vhost | 16:16 |
| clarkb | and then static is bunch of names with SNI | 16:16 |
| corvus | Shrews: wow. erm. well, it's sort of a squiggly line that's flatish over the weekend then goes up and down a bit but mostly stays up. that help? | 16:16 |
| clarkb | Shrews: you may have the old graphite.openstack.org backend cached | 16:17 |
| Shrews | corvus: perfect | 16:17 |
| clarkb | try a hard refresh? | 16:17 |
| clarkb | ok I'll work on putting an etherpad together with listings, then I can go spend some money and hopefully rotate out hiera data a bit later today | 16:18 |
| *** jpich has quit IRC | 16:18 | |
| Shrews | nope. no good. oh well. my life will just be graphless on this machine | 16:18 |
| *** markvoelker has quit IRC | 16:20 | |
| *** jbadiapa has joined #openstack-infra | 16:21 | |
| corvus | Shrews: you'll have a completely different firefox tomorrow, maybe it will work then | 16:22 |
| *** dtantsur is now known as dtantsur|afk | 16:22 | |
| Shrews | corvus: maybe i'll have to wait for the paid version coming out in a few months | 16:23 |
| corvus | "value-add" | 16:26 |
| fungi | Shrews: check to see if you're blocking requests from that page maybe | 16:27 |
| fungi | it needs to be able to fetch from graphite.opendev.org | 16:27 |
| fungi | i have to explicitly allow that in privacy badger, for example | 16:28 |
| *** jamesmcarthur_ has joined #openstack-infra | 16:29 | |
| clarkb | infra-root: https://etherpad.openstack.org/p/2019-june-ssl-cert-refresh how does that look? | 16:31 |
| clarkb | fungi: because the domain is different? | 16:32 |
| fungi | clarkb: i think that may be why, but am not sure | 16:32 |
| *** jamesmcarthur has quit IRC | 16:32 | |
| *** Lucas_Gray has quit IRC | 16:33 | |
| fungi | a page on grafana.openstack.org making requests to fetch content from graphite.opendev.org seems to cause effpb to think it's a "third party tracker" so seems likely the cause | 16:33 |
| corvus | clarkb: is git.zuul-ci.org ok? | 16:33 |
| clarkb | corvus: git.zuul-ci.org:443 Valid Mar 26 2020 287 seems to be | 16:33 |
| corvus | i'm guessing the same for airship? | 16:34 |
| Shrews | fungi: ooh, i use pb too. lemme check | 16:34 |
| clarkb | corvus: ya airship I had to replace last month | 16:34 |
| clarkb | git.airshipit.org:443 Valid May 12 2021 699 | 16:34 |
| *** dims has joined #openstack-infra | 16:35 | |
| corvus | clarkb: lgtm | 16:35 |
| fungi | clarkb: are we sure that's accurate? my browser says 30 November 2019 for git.zuul-ci.org | 16:35 |
| Shrews | fungi: that was it. you are geenyus | 16:35 |
| corvus | fungi: is that after the redirect? | 16:35 |
| fungi | Shrews: not that genius, i beat my head against it for a while before i noticed the little icon in the corner of my browser | 16:35 |
| *** tdasilva has joined #openstack-infra | 16:35 | |
| fungi | corvus: d'oh! yep | 16:36 |
| fungi | that would be it | 16:36 |
| fungi | clearly not genius ;) | 16:36 |
| fungi | i was looking at our opendev.org cert | 16:36 |
| * fungi sighs | 16:36 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 16:36 |
| clarkb | then the other question I was gonna ask, since these will be issued for 2 years should we consider increasing the number of bits in our key? | 16:36 |
| clarkb | fungi: ^ probably question for you | 16:37 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 16:37 |
| fungi | you're generating these on bridge.o.o, right? it has newer openssl with a larger default keysize i think, but checking | 16:37 |
| fungi | ahh, we hardcode rsa:2048 in gencert.sh | 16:38 |
| *** weifan has joined #openstack-infra | 16:38 | |
| clarkb | ya | 16:38 |
| clarkb | seems like 2048 is on the edge of what is recommended now? like its good enough for today but may not be good in 2 years? | 16:38 |
| *** noonedeadpunk has left #openstack-infra | 16:39 | |
| fungi | it looks like we're doing 2048 for our letsencrypt certs too | 16:40 |
| fungi | we could bump it to 3072, sure | 16:40 |
| fungi | though really 2048 is likely good for many years to come, the concern is someone recording traffic now and then decrypting it decades from now | 16:41 |
| clarkb | apparently some devices can only do up to 2048 | 16:41 |
| clarkb | reading more I think 2048 is probably fine | 16:42 |
| clarkb | and may make clients happier | 16:42 |
| *** weifan has quit IRC | 16:42 | |
| clarkb | I'm about to pop out for my daily bike ride. We can think about it a bit more as I won't do any cert generation until a bit later | 16:44 |
| *** pkopec is now known as pkopec|afk | 16:46 | |
| clarkb | But ya plan is to issue 2 year certs from namecheap based on my list on the etherpad | 16:46 |
| fungi | in a 2003 report, rsa labs suggested 2048-bit keys are likely good through at least 2030 | 16:48 |
| fungi | https://www.emc.com/emc-plus/rsa-labs/historical/twirl-and-rsa-key-size.htm | 16:48 |
| fungi | er, need the wayback for that | 16:49 |
| fungi | https://web.archive.org/web/20170417095741/https://www.emc.com/emc-plus/rsa-labs/historical/twirl-and-rsa-key-size.htm | 16:49 |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: Add airship/election https://review.opendev.org/664641 | 16:51 |
| *** weifan has joined #openstack-infra | 16:53 | |
| *** weifan has quit IRC | 16:53 | |
| *** weifan has joined #openstack-infra | 16:53 | |
| *** weifan has quit IRC | 16:54 | |
| *** weifan has joined #openstack-infra | 16:54 | |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard-webclient master: Render user-submitted links with nofollow attribute https://review.opendev.org/648322 | 16:54 |
| *** weifan has quit IRC | 16:55 | |
| *** weifan has joined #openstack-infra | 16:55 | |
| *** jamesmcarthur_ has quit IRC | 16:57 | |
| *** jamesmcarthur has joined #openstack-infra | 16:58 | |
| *** kopecmartin is now known as kopecmartin|off | 17:01 | |
| *** derekh has quit IRC | 17:03 | |
| *** rpioso|afk is now known as rpioso | 17:05 | |
| *** e0ne has joined #openstack-infra | 17:05 | |
| *** ykarel|away has quit IRC | 17:06 | |
| *** mattw4 has joined #openstack-infra | 17:07 | |
| openstackgerrit | Merged opendev/storyboard master: Change the dist name to 'storyboard-api' https://review.opendev.org/661663 | 17:07 |
| *** pcaruana has joined #openstack-infra | 17:09 | |
| *** nicolasbock has quit IRC | 17:10 | |
| *** ociuhandu has quit IRC | 17:15 | |
| *** jpena is now known as jpena|off | 17:19 | |
| *** e0ne has quit IRC | 17:23 | |
| *** panda has quit IRC | 17:24 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Correct quick-start executor volume error https://review.opendev.org/665186 | 17:25 |
| *** ralonsoh has quit IRC | 17:26 | |
| *** panda has joined #openstack-infra | 17:27 | |
| *** pkopec|afk has quit IRC | 17:29 | |
| *** pkopec|afk has joined #openstack-infra | 17:29 | |
| *** yamamoto has joined #openstack-infra | 17:33 | |
| *** whoami-rajat has joined #openstack-infra | 17:38 | |
| *** ramishra has quit IRC | 17:44 | |
| *** signed8bit has quit IRC | 17:52 | |
| *** jtomasek has quit IRC | 17:53 | |
| openstackgerrit | Merged opendev/storyboard master: Retry on DB deadlock when updating story https://review.opendev.org/650614 | 17:55 |
| *** jtomasek has joined #openstack-infra | 17:55 | |
| *** gfidente has quit IRC | 17:56 | |
| *** jamesmcarthur has quit IRC | 17:58 | |
| clarkb | fungi: ok given that I'll stick with the 2048 bit key size | 17:58 |
| *** Goneri has joined #openstack-infra | 18:03 | |
| *** gyee has quit IRC | 18:04 | |
| *** donnyd has joined #openstack-infra | 18:10 | |
| donnyd | Is there a way to contribute resources to infra outside of having a 100 or so public IP's | 18:13 |
| *** ianychoi has quit IRC | 18:13 | |
| clarkb | donnyd: those IPs can be ipv6 addresses, but because we run a multi cloud CI system operating without public IPs that can talk to each other gets difficult and we aren't set up to do that | 18:13 |
| donnyd | Ok | 18:13 |
| donnyd | so IPv6 can be used | 18:14 |
| donnyd | So If I had some resources to contribute, what would be the minimum to be useful? | 18:14 |
| clarkb | yes, we'll need 2 IPv4 addresses in that case (one for the mirror node and one for outbound NAT from the test nodes because github etc cannot ipv6) | 18:14 |
| clarkb | its possible the mirror node might be able to sit behind that one NAT address too but I don't think we have ever tested that | 18:15 |
| mnaser | it probably could just fine imho | 18:15 |
| mnaser | but pure speculation :) | 18:15 |
| donnyd | storage? | 18:15 |
| clarkb | donnyd: I think we need at least 100GB for the mirror node (it caches a bunch of data in the region to avoid hitting the internet so much for it) | 18:16 |
| clarkb | then each test node wants ~80GB disk | 18:16 |
| donnyd | I guess I will get to the point, I built a micro cloud and the project is done on the gear.. so its just sitting here. I have a couple hundred cpus a TB or so of memory and 100TB of storage | 18:16 |
| clarkb | donnyd: the test nodes tend to be 8vcpu, 80GB disk, and 8GB ram. We can likely tweak the cpu number or oversubscribe depending on how it works out, but jobs need a bit of disk to juggle git repos and so on and 8GB memory anymore seems to be a minmum | 18:17 |
| *** markvoelker has joined #openstack-infra | 18:17 | |
| clarkb | as for how many is useful to get started, We try to target that 100 number while understanding that it won't necessraily happen upfront. I think we've started with as little as 10 beofre and ramped up | 18:18 |
| clarkb | (basically we can be flexible and try to work with the clouds to find what works for them and us) | 18:19 |
| *** gyee has joined #openstack-infra | 18:19 | |
| *** tosky has quit IRC | 18:21 | |
| *** markvoelker has quit IRC | 18:22 | |
| *** bhavikdbavishi has quit IRC | 18:23 | |
| *** michael-beaver has joined #openstack-infra | 18:23 | |
| efried | clarkb: back-of-napkin, would adding this amount of hardware result in a noticeable reduction in queue times? | 18:25 |
| efried | (I have absolutely no sense of how much we have right now, or what % increase this represents) | 18:26 |
| donnyd | I can do 100 an not even notice | 18:26 |
| donnyd | and* | 18:26 |
| clarkb | efried: donnyd http://grafana.openstack.org/d/rZtIH5Imz/nodepool?orgId=1 is current status | 18:28 |
| clarkb | we seem to be able to use about 550 full steam ahead right now so that would be about 20% increase. Likely would have an impact | 18:28 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Change default job_dir location https://review.opendev.org/665186 | 18:28 |
| efried | nice | 18:28 |
| clarkb | the building and error numbers there seem a bit high. I wonder if we have a region that has taken a lunch break | 18:28 |
| donnyd | Ok, I will have to swap some things around here, but if someone is willing to work with me.. I am willing to contribute | 18:32 |
| *** pkopec|afk has quit IRC | 18:32 | |
| donnyd | how important is storage performance? | 18:33 |
| donnyd | The front end of my ceph cluster is only 10G, so is 1Gb R/W speeds sufficient? | 18:34 |
| clarkb | mnaser: ^ you probably know best re the ceph performance | 18:35 |
| clarkb | donnyd: I'm certainly willing to try it :) | 18:35 |
| clarkb | also I'm happy to work with you on that. Though I've got to renew certs today/tomorrow and then I'm taking most of next week off. So if we want to move quickly we might need a different volunteer | 18:35 |
| donnyd | I am out next week too, so no biggie | 18:36 |
| donnyd | what about persistent storage? any requirements for keeping data long term? | 18:38 |
| donnyd | like logs and what not. Are they stored on the cloud they are built on? | 18:38 |
| clarkb | donnyd: they are stored remotely so yo udon't have to worry about that. The mirror node we spin up acts as a cache too so isn't canonical source of data either (and we can delete it if necessary) | 18:39 |
| donnyd | ok cool | 18:39 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 18:39 |
| *** weifan has quit IRC | 18:49 | |
| *** yamamoto has quit IRC | 18:52 | |
| clarkb | doing this I've realized that maybe the cert stuff should/could be tracked in git? but for now I'm just going to add a bit extra organization | 18:52 |
| clarkb | then think a bit more baout how a git repo for that would look | 18:53 |
| *** dpawlik has joined #openstack-infra | 18:54 | |
| clarkb | though maybe this is the last time it needs to be orngaized to meh | 18:55 |
| *** jamesmcarthur has joined #openstack-infra | 18:56 | |
| clarkb | I'm taking the hiera lock | 18:56 |
| clarkb | #status log Updated ask.openstack.org's ssl cert as part of regular refresh cycle. | 19:00 |
| openstackstatus | clarkb: finished logging | 19:00 |
| clarkb | Thats my do one first run. If that looks good after lunch I should be able to tear through the other 15 single name cert updates pretty quickly | 19:01 |
| fungi | yeah, just make sure to update the chain cert in addition to the new cert (and key if you generated a new one) | 19:01 |
| fungi | it's tempting to assume the chain certs will all be the same, but that's not guaranteed to be the case | 19:01 |
| clarkb | yup all three got updated | 19:03 |
| clarkb | (they have indeed updated the chain) | 19:03 |
| fungi | never a dull moment where public key infrastructure is concerned | 19:04 |
| *** bnemec has quit IRC | 19:06 | |
| *** markvoelker has joined #openstack-infra | 19:18 | |
| *** raissa has joined #openstack-infra | 19:18 | |
| *** weifan has joined #openstack-infra | 19:21 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Improve retry handling for github driver https://review.opendev.org/665220 | 19:21 |
| *** markvoelker has quit IRC | 19:23 | |
| *** weifan has quit IRC | 19:23 | |
| *** jamesmcarthur has quit IRC | 19:24 | |
| *** e0ne has joined #openstack-infra | 19:25 | |
| clarkb | ask's puppetry doesn't restart apache on cert update. I'm doing that now | 19:26 |
| clarkb | my browser is happy with the new cert there. Given that I will work through the other 15 in a larger batch after lunch | 19:27 |
| johnsom | fyi, the iad rax mirror seems to be unhappy: http://logs.openstack.org/07/665207/2/check/octavia-v2-dsvm-noop-api-stable-stein/00ea536/controller/logs/devstacklog.txt.gz#_2019-06-13_19_02_55_505 | 19:29 |
| johnsom | I just had three jobs fail to stack due to missing packages. | 19:29 |
| clarkb | johnsom: that is because the package is http://mirror.iad.rax.opendev.org/ubuntu/pool/main/g/gcc-7/libgfortran4_7.4.0-1ubuntu1~18.04.1_amd64.deb not http://mirror.iad.rax.opendev.org/ubuntu/pool/main/g/gcc-7/libgfortran4_7.4.0-1ubuntu1~18.04_amd64.deb | 19:30 |
| clarkb | maybe you need an apt-get update or the package deps are broken? | 19:31 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 19:31 |
| johnsom | I get the same on three different packages and the jobs at the other hosts are running. | 19:32 |
| *** takamatsu has quit IRC | 19:32 | |
| johnsom | We didn't change any package deps, etc. | 19:32 |
| clarkb | johnsom: ya its part of the dep chain for libgfortran4 looks like | 19:32 |
| clarkb | so not something you are directly asking for | 19:32 |
| clarkb | I wonder if we are having similar behavior to the openafs node on bionic but with kafs where an index type file is out of date | 19:32 |
| clarkb | fungi: ^ | 19:33 |
| johnsom | lol, we aren't crazy enough to start coding in fortran... lol | 19:34 |
| corvus | i'm crazy enough, but i don't think i'm smart enough. | 19:34 |
| fungi | clarkb: hrm, maybe... | 19:35 |
| clarkb | http://mirror.iad.rax.opendev.org/ubuntu/dists/bionic-updates/main/binary-amd64/Packages points to the right file now | 19:35 |
| clarkb | says Filename: pool/main/g/gcc-7/libgfortran4_7.4.0-1ubuntu1~18.04.1_amd64.deb | 19:36 |
| fungi | did it update? | 19:36 |
| clarkb | Not sure how I would tell | 19:36 |
| clarkb | just observing the current index seems to point to the file that does exist | 19:37 |
| clarkb | which makes me think that ya maybe their job needed an apt-get update | 19:37 |
| clarkb | (though we should keep the old files around long enough that that isn't strictly necessary) | 19:37 |
| corvus | the log had an apt-get update right before that install | 19:37 |
| fungi | so possible at the time the mirror server was serving an out of date reference but has since updated? | 19:38 |
| corvus | Last Update Thu Jun 13 11:38:58 2019 | 19:38 |
| clarkb | fungi: that is what it is beginning to look like | 19:38 |
| corvus | my math says the last volume update was at 18:38 utc | 19:38 |
| corvus | that's still 24 minutes before the apt-get update/install in the job | 19:39 |
| fungi | hrm... | 19:39 |
| corvus | (that's from "vos examine mirror.ubuntu") | 19:39 |
| clarkb | and we are also supposed to keep the old files around for like 3-4 hours iirc so a couple things going on here that we don't expect | 19:39 |
| corvus | maybe cachefilesd doesn't invalidate like openafs | 19:40 |
| corvus | (if so, maybe it was serving an outdated version of the package index) | 19:41 |
| johnsom | If it's any help, just had another job hit the fortran file: http://logs.openstack.org/23/664123/5/check/octavia-v2-dsvm-py2-scenario/5bf6507/controller/logs/devstacklog.txt.gz#_2019-06-13_19_37_28_581 | 19:41 |
| yoctozepto | ah, just came to say ubuntu mirror has problems but you guys know it already, that's proactive :-) | 19:42 |
| clarkb | corvus: reading the manpage for the cachefilesd config file i'm beginning to think it doesn't | 19:43 |
| clarkb | corvus: it seems to cull based on available disk space | 19:43 |
| yoctozepto | kolla hit e.g.: Failed to fetch http://mirror.iad.rax.opendev.org/ubuntu/pool/main/b/binutils/binutils-common_2.30-21ubuntu1~18.04.1_amd64.deb 404 Not Found [IP: 104.130.4.160 80] | 19:43 |
| auristor | clarkb there were some callback handling fixes merged as part of the 5.2 merge window. I don't know if they were backported to 5.1.9. | 19:43 |
| yoctozepto | and it was right after apt-get update | 19:43 |
| fungi | yoctozepto: thanks, same server where we're trying out kafs so likely related | 19:44 |
| corvus | auristor: for kafs? | 19:44 |
| *** rf0lc0 has quit IRC | 19:44 | |
| *** rfolco has joined #openstack-infra | 19:44 | |
| clarkb | ah I suppose it would want to cull in addition to doing invalidation the openafs way so that doesn't necessary rule the expected behavior out. But if there are bugs... | 19:44 |
| fungi | and yeah, that server is booting from a custom build of linux 5.1.9 | 19:45 |
| fungi | (just confirmed) | 19:45 |
| yoctozepto | fungi: no problem, here is a basket of failures: http://logs.openstack.org/48/495148/77/check/kolla-build-ubuntu-source/f13ba68/logs/build/ | 19:45 |
| yoctozepto | in case you are tracking files | 19:45 |
| clarkb | I've got to make lunch now, but I think that bionic-updates packages file is up to date at least. Not sure about the other dists | 19:46 |
| clarkb | we'll likely be fine until the next update from upstream | 19:47 |
| yoctozepto | i run a recheck for us | 19:47 |
| yoctozepto | please ping me when you think you found the cause (always curious) | 19:48 |
| *** whoami-rajat has quit IRC | 19:48 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Improve retry handling for github driver https://review.opendev.org/665220 | 19:48 |
| fungi | i suppose we could try building from linux-next and seeing if the problem persists | 19:49 |
| *** xek has joined #openstack-infra | 19:50 | |
| fungi | or 5.2-rc4 according to kernel.org (assuming the mentioned improvements made it in by sunday) | 19:56 |
| auristor | the kafs callback fixes were in 5.2-rc1 | 19:57 |
| fungi | ahh, cool, thanks! | 19:58 |
| *** auristor has quit IRC | 19:58 | |
| *** eharney has quit IRC | 20:01 | |
| fungi | ianw: what was your build process for the 5.1.9 kernel? or did you grab that deb from somewhere else? | 20:02 |
| * fungi is familiar with the process for doing custom debian kernel packages, but not sure if the process is the same on ubuntu | 20:03 | |
| clarkb | fungi: I believe there is a dnm change up fkr that | 20:04 |
| *** e0ne has quit IRC | 20:04 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Improve retry handling for github driver https://review.opendev.org/665220 | 20:05 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Improve retry handling for github driver https://review.opendev.org/665220 | 20:07 |
| fungi | thanks clarkb, will find | 20:11 |
| *** markvoelker has joined #openstack-infra | 20:19 | |
| *** markvoelker has quit IRC | 20:24 | |
| openstackgerrit | Merged zuul/zuul master: Split plugin tests into two https://review.opendev.org/665161 | 20:31 |
| * fungi did not find, still searching | 20:34 | |
| *** pcaruana has quit IRC | 20:36 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Change default job_dir location https://review.opendev.org/665186 | 20:43 |
| *** auristor has joined #openstack-infra | 20:43 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Web: plug the authorization engine https://review.opendev.org/640884 | 20:44 |
| *** yamamoto has joined #openstack-infra | 20:47 | |
| *** panda has quit IRC | 20:49 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul Web: add /api/user/authorizations endpoint https://review.opendev.org/641099 | 20:49 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 20:50 |
| *** panda has joined #openstack-infra | 20:51 | |
| *** bnemec has joined #openstack-infra | 20:52 | |
| openstackgerrit | Merged opendev/system-config master: linters: ensure openstack/inventory.yaml parses https://review.opendev.org/664751 | 20:54 |
| *** bobh has joined #openstack-infra | 20:56 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Web: plug the authorization engine https://review.opendev.org/640884 | 20:57 |
| *** xek has quit IRC | 20:57 | |
| clarkb | fungi: https://review.opendev.org/665057 | 21:00 |
| *** dpawlik has quit IRC | 21:01 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/project-config master: Retire independent ansible role https://review.opendev.org/665237 | 21:05 |
| *** kaisers1 has joined #openstack-infra | 21:07 | |
| *** kaisers has quit IRC | 21:09 | |
| *** jtomasek has quit IRC | 21:10 | |
| *** raissa has quit IRC | 21:10 | |
| *** takamatsu has joined #openstack-infra | 21:13 | |
| *** mriedem has quit IRC | 21:19 | |
| *** markvoelker has joined #openstack-infra | 21:20 | |
| fungi | huh, no idea why i didn't see that in the list | 21:21 |
| fungi | looks like https://kernel.ubuntu.com/~kernel-ppa/mainline/ does include release candidates for 5.2 | 21:23 |
| fungi | including sunday's rc4 | 21:23 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 21:24 |
| *** markvoelker has quit IRC | 21:24 | |
| ianw | fungi: on IAD in /opt there's a linux directory, i used the inbuilt bindeb-pkg ... i'm still catching up but i can kick off a build of master now | 21:36 |
| *** bobh has quit IRC | 21:36 | |
| clarkb | so much button clicking and file copying | 21:36 |
| ianw | i have to dash for about ~1 hour, but it's building a current master kernel on IAD which we can try, it will take about 15 minutes but debs should come out in /opt/linux on IAD ... bib | 21:37 |
| *** bobh has joined #openstack-infra | 21:42 | |
| *** sreejithp has quit IRC | 21:43 | |
| *** slaweq has quit IRC | 21:43 | |
| *** slaweq has joined #openstack-infra | 21:45 | |
| fungi | ianw: cool--thanks! | 21:45 |
| *** bobh has quit IRC | 21:46 | |
| *** slaweq has quit IRC | 21:49 | |
| *** boden has quit IRC | 21:53 | |
| clarkb | ok taking the hiera lock again to update 13 more ssl certs ( my single name cert list less git.starlingx.io and openstackid.org) | 21:58 |
| *** bnemec has quit IRC | 22:05 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 22:17 |
| *** jamesmcarthur has joined #openstack-infra | 22:18 | |
| *** pfallenop has quit IRC | 22:24 | |
| *** takamatsu has quit IRC | 22:25 | |
| *** dave-mccowan has quit IRC | 22:26 | |
| *** yamamoto has quit IRC | 22:29 | |
| clarkb | ok all of the single domain openstack.org certs have been updated in hiera now | 22:29 |
| clarkb | I'm going to restart apache on files.o.o to pick up the changes there | 22:30 |
| corvus | pabelanger: i may not be around consistently enough in 30+ mins to be responsible for a restart; but feel free to proceed if you're comfortable, or i can help again tomorrow morning | 22:32 |
| pabelanger | corvus: ack, I'll see if ianw is around, if not tomorrow also works | 22:32 |
| clarkb | I'm verifying openstackid.org and git.starlingx.io certs nowish. That requires the dns dance which is why I am doing them separately | 22:35 |
| *** slaweq has joined #openstack-infra | 22:36 | |
| *** tobiash has quit IRC | 22:40 | |
| *** pfallenop has joined #openstack-infra | 22:40 | |
| *** slaweq has quit IRC | 22:41 | |
| openstackgerrit | Jason E. Rist proposed openstack/project-config master: Retiring TripleO-UI and associated projects. https://review.opendev.org/665259 | 22:43 |
| *** yamamoto has joined #openstack-infra | 22:44 | |
| *** aaronsheffield has quit IRC | 22:45 | |
| *** rcernin has joined #openstack-infra | 22:45 | |
| *** weifan has joined #openstack-infra | 22:46 | |
| ianw | ok i'm back :) | 22:46 |
| *** harlowja has quit IRC | 22:47 | |
| *** tesseract has quit IRC | 22:48 | |
| *** yamamoto has quit IRC | 22:50 | |
| *** dave-mccowan has joined #openstack-infra | 22:50 | |
| *** tkajinam has joined #openstack-infra | 22:51 | |
| *** ekultails has quit IRC | 22:51 | |
| *** dtroyer has quit IRC | 22:52 | |
| clarkb | ok git.starlingx.io and openstackid.org ssl certs updated in hiera now too | 22:52 |
| *** dtroyer has joined #openstack-infra | 22:52 | |
| clarkb | I need to restart apache on review.o.o to pick up the new cert. Any reason to not do it really quickly now? | 22:53 |
| clarkb | infra-root ^ | 22:54 |
| *** goldyfruit has quit IRC | 22:54 | |
| fungi | sounds good, and that will also pick up the lingering commentlink change for certain zuul job result types? | 22:59 |
| clarkb | just apache not gerrit | 23:00 |
| clarkb | fungi: how do I update the ssl cert on the wiki? | 23:02 |
| clarkb | is that manual? and if so would you prefer to do it? | 23:02 |
| clarkb | certcheck reports only static (and friends) and wiki.openstack.org as expiring in 9 days now | 23:03 |
| fungi | oh, right, apache restart should be almost entirely invisible | 23:04 |
| clarkb | I'll do static tomorrow (I hvae to figure out the key generation with multiple altnames) and can do wiki as soon as I know what the process is there | 23:04 |
| fungi | sure, i can update it... entirely manual since the production server is still not config-managed | 23:04 |
| clarkb | fungi: ok, hiera and /root/certs/wiki.openstack.org have the data | 23:04 |
| fungi | thanks! | 23:04 |
| clarkb | fungi: also note all three files changed | 23:06 |
| ianw | clarkb / fungi : so ubuntu/pool/main/b/binutils/binutils-common_2.30-21ubuntu1~18.04.1_amd64.deb appears to legitimately not be there, it's ".2" now | 23:08 |
| clarkb | ianw: ya we had similar with a libgfortran file | 23:09 |
| clarkb | ianw: it seemed like the Packages file was not being updated so clients were looking at the old file | 23:09 |
| clarkb | and that old file was pointing to oler packages that did not exist | 23:09 |
| *** yamamoto has joined #openstack-infra | 23:10 | |
| ianw | right, so that would suggest that the "vos release" wasn't as atomic as we'd hope? | 23:10 |
| clarkb | fungi: certcheck shows wiki has been updated too now. Thanks! | 23:10 |
| clarkb | ianw: ya. auristor pointed out that linux 5.2 includes some fixes for the callbacks that help invalidate cache | 23:10 |
| clarkb | ianw: so one idea is there is a bug in that and kafs didn't get notified properly that the file changed | 23:10 |
| fungi | clarkb: "12 June 2021" for https://wiki.openstack.org/ now | 23:10 |
| *** goldyfruit has joined #openstack-infra | 23:11 | |
| ianw | clarkb: right, ok, timeline makes sense now :) sometimes my mornings are like avengers: endgame :) | 23:11 |
| clarkb | fungi: yup. That leaves the one multiname cert as the remaining cert I need to update. I'll do that tomorrow morning | 23:11 |
| clarkb | ianw: today was last day of school before summer break for the kids. I know the feeling :) | 23:12 |
| fungi | sounds good. thanks for tackling all these? | 23:12 |
| fungi | s/?/!!!/ | 23:12 |
| *** _erlon_ has quit IRC | 23:12 | |
| clarkb | the best news is certcheck will be a bit quieter tomorrow in my inbox | 23:13 |
| *** yamamoto has quit IRC | 23:13 | |
| clarkb | also comodo is becoming sectigo so you may notice that some certs are signed by comodo and some are signed by sectigo. This is normal and apparently expected | 23:14 |
| fungi | sectigo. interesting | 23:14 |
| ianw | ok, rebooting iad to 5.2.0-rc4 | 23:16 |
| ianw | i think we should probably switch in dfw with openafs 1.8.3 packages too, to validate that | 23:17 |
| clarkb | #status log developer.openstack.org docs.openstack.org ethercalc.openstack.org etherpad.openstack.org firehose.openstack.org git.openstack.org git.starlingx.io openstackid-dev.openstack.org openstackid.org refstack.openstack.org review.openstack.org storyboard.openstack.org translate.openstack.org wiki.openstack.org zuul.openstack.org ssl certs updated as part of normal refresh cycle. | 23:17 |
| openstackstatus | clarkb: finished logging | 23:17 |
| clarkb | ianw: that seems like a good idea considering kafs seems to have some maybe unhappy behavior (and we hope kernel upate fixes but it may not) | 23:18 |
| *** slaweq has joined #openstack-infra | 23:19 | |
| ianw | ok, iad back and up serving with 5.2.0-rc4 ; cachefilesd is running | 23:19 |
| *** dave-mccowan has quit IRC | 23:20 | |
| ianw | i think we have to validate it with that ... right now it's rax internal for the afs network access, but when it's globally distributed mirrors we know we need all the caching we can get | 23:20 |
| clarkb | ya particularly when crossing the atlantic | 23:21 |
| *** markvoelker has joined #openstack-infra | 23:22 | |
| ianw | the unbound/ntp thing is interesting. might be good to put a reboot: into our base job CI to validate it | 23:23 |
| *** jamesmcarthur has quit IRC | 23:24 | |
| *** markvoelker has quit IRC | 23:26 | |
| openstackgerrit | Merged zuul/zuul master: Improve retry handling for github driver https://review.opendev.org/665220 | 23:31 |
| *** dave-mccowan has joined #openstack-infra | 23:40 | |
| *** eernst has joined #openstack-infra | 23:43 | |
| *** eernst has quit IRC | 23:47 | |
| *** sthussey has quit IRC | 23:48 | |
| *** eernst has joined #openstack-infra | 23:49 | |
| corvus | ianw, clarkb, fungi: oh i did some more looking into the ntp thing but i forgot to report | 23:49 |
| corvus | i'm starting to think maybe we should drop ntpd in bionic and just rely on systemd-timesyncd | 23:50 |
| ianw | i'm sure that's considered the way of the future | 23:51 |
| corvus | there's a debian bug report from like 2010 about howe the init system ordering is wrong for ntpd; and even after a lot of discussion, they never resolved the bug before changing the init system | 23:51 |
| corvus | this doesn't give me a lot of confidence that it'll get fixed upstream | 23:51 |
| corvus | i guess systemd lets us write shadow configurations, so we could probably put the correct dependencies in there? | 23:52 |
| *** eernst has quit IRC | 23:52 | |
| *** eernst has joined #openstack-infra | 23:52 | |
| corvus | but, if our production systems haven't been running ntpd because of this bug (it seems that way), and we haven't noticed (probably because they're running on hypervisors that keep their clocks pretty stable?), then it's probably not that important, and perhaps just letting systemd-timesyncd do its thing is enough? | 23:53 |
| *** slaweq has quit IRC | 23:56 | |
| *** eernst has quit IRC | 23:57 | |
| ianw | i think so, or we could just hack around it with a manual restart of ntpd in a late service file ... but how many tools on the system do we need to do the same thing | 23:57 |
| *** eernst has joined #openstack-infra | 23:59 | |
| ianw | i can stack a change before the kernel update and see how we go | 23:59 |
| *** weifan has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!