| corvus | yep. it's better for everyone but us :) | 00:00 |
|---|---|---|
| fungi | heh | 00:00 |
| corvus | it will get better for us someday, i'm sure. | 00:00 |
| fungi | during the storyboard meeting it was identified that a similar rename api and redirecting feature would be a good idea to add there as well in the future | 00:01 |
| corvus | oh yeah, good idea | 00:01 |
| clarkb | fungi: thats a relatively new concern with storyboard supporting names in the api | 00:01 |
| fungi | yep | 00:01 |
| fungi | well, and by extension in the webclient | 00:02 |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack-infra/project-config master: Retire openstack-ansible unused roles https://review.openstack.org/650422 | 00:02 |
| fungi | quick zuul question for the script for friday... is it only job objects which will have pre-run, run and post-run keys referring to playbook filenames or lists of filenames? | 00:04 |
| fungi | and are there any other keys which could also refer to local playbooks? | 00:05 |
| clarkb | I beleive those are the only keys that refer to the zuul playbooks | 00:05 |
| fungi | and they only appear in job definitions? | 00:05 |
| *** ijw has quit IRC | 00:06 | |
| clarkb | ya | 00:06 |
| fungi | okay, cool | 00:06 |
| fungi | and zuul roles will only ever be in a top-level "roles" directory? or is there additional logic required to identify them? | 00:07 |
| *** lseki has quit IRC | 00:07 | |
| clarkb | I think that is the rule zuul relies on to find the roles | 00:08 |
| clarkb | if they are elsewhere then zuul can't use them | 00:08 |
| fungi | okay, so roles relative to the directories in which playbooks reside aren't a thing? | 00:09 |
| clarkb | no beacuse zuul has to set up a rolepath and I believe it does that relative to the top level dir of the repo under test | 00:09 |
| clarkb | but maybe ansible magic works relative to the playbooks? | 00:09 |
| fungi | perfect. i gathered ansible had extra logic to search for roles, which would have gotten complicated quickly | 00:10 |
| fungi | hopefully zuul is quashing that behavior | 00:10 |
| *** _erlon_ has quit IRC | 00:11 | |
| corvus | well... | 00:11 |
| corvus | technically pre-run, run, and post-run can happen in project-pipeline variants and project-templates. it's rare. | 00:12 |
| fungi | those keys will be at the same level of yaml structure as jobs though? i can just add them to the match | 00:13 |
| fungi | if node in ['job', 'project-pipeline', 'project-templates']: ... | 00:13 |
| *** ijw has joined #openstack-infra | 00:13 | |
| corvus | ah... i think it'd be different | 00:14 |
| *** ijw has quit IRC | 00:14 | |
| fungi | well, i'll start with just jobs then and we can improve on it if desired | 00:14 |
| corvus | because it's [{job: {run: foo}}] vs [{project: {check: {jobs: {jobname: {run: foo}}}}}] | 00:15 |
| corvus | (and same for project-template) | 00:15 |
| corvus | i would bet a nickel that there are no incidences of that in our system | 00:15 |
| fungi | eee, yeah that gets deeper into the nesting then | 00:15 |
| fungi | happy to owe you a nickel on that one | 00:16 |
| corvus | fungi: so if you want to prioritize that at the end after you run out of things to do and don't actually get to it before the migration, i think that's entirely reasonable. :) | 00:17 |
| fungi | yep, that is now the plan ;) | 00:17 |
| corvus | fungi: regarding roles -- ansible will also find roles next to playbooks | 00:17 |
| fungi | so i should look at all yaml/yml files in the same directory as a playbook? | 00:18 |
| corvus | (eg system-config/playbooks/roles) | 00:18 |
| fungi | oh, roles subdir | 00:18 |
| corvus | yeah that | 00:18 |
| openstackgerrit | Merged openstack-infra/nodepool master: Add support for yappi and objgraph output https://review.openstack.org/653541 | 00:18 |
| fungi | will figure out how to incorporate that | 00:18 |
| fungi | is .yml a supported prefix for zuul (not ansible) configs? | 00:20 |
| fungi | i want to say we had a transition forcing them all to .yaml? | 00:20 |
| fungi | https://zuul-ci.org/docs/zuul/user/config.html#configuration-items indicates only .yaml | 00:22 |
| fungi | and .zuul.d/ isn't a thing, right? docs don't mention it anyway | 00:22 |
| corvus | fungi: .yaml only -- .zuul.d should be a thing | 00:23 |
| corvus | yeah, both of those items are mentioned in https://zuul-ci.org/docs/zuul/user/config.html#configuration-loading | 00:23 |
| corvus | i guess the second section should add a "(or .zuul.d)" to clear that up | 00:23 |
| fungi | oh, thanks, i looked at the wrong section | 00:23 |
| *** eernst has joined #openstack-infra | 00:24 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Clarify .zuul.d directory https://review.openstack.org/653644 | 00:25 |
| corvus | fungi: every misunderstanding is a doc bug :) | 00:25 |
| fungi | fair | 00:25 |
| fungi | and thanks! reviewing | 00:25 |
| *** eernst has quit IRC | 00:28 | |
| *** ijw has joined #openstack-infra | 00:30 | |
| *** ijw has quit IRC | 00:34 | |
| *** dangtrinhnt_ has quit IRC | 00:36 | |
| *** dangtrinhnt has joined #openstack-infra | 00:36 | |
| fungi | hrm... !encrypted/pkcs1-oaep freaks out yaml.safe_load() | 00:36 |
| *** diablo_rojo has quit IRC | 00:36 | |
| fungi | is there an easier solution than subclassing yaml.SafeLoader? | 00:38 |
| *** diablo_rojo has joined #openstack-infra | 00:39 | |
| corvus | i think so, lemme dig | 00:43 |
| *** jamesmcarthur has joined #openstack-infra | 00:44 | |
| corvus | fungi: i believe that merely implementing this class will do the trick: https://opendev.org/openstack-infra/zuul/src/branch/master/zuul/configloader.py#L392-L423 | 00:44 |
| corvus | fungi: (remove most of the actual code) | 00:45 |
| fungi | so, like, an empty __init__(self) and drop the other methods? | 00:45 |
| corvus | yeah -- maybe keep from_yaml | 00:46 |
| corvus | not sure exactly how much is required | 00:46 |
| fungi | i'll know soon enough. thanks! | 00:46 |
| fungi | any other custom tags you can think of zuul might grok? | 00:46 |
| corvus | but the main thing is that pyyaml has some python magic to use that class if it exists (subclassing yaml.YAMLObject and the yaml_* class attributes are the key things there) | 00:46 |
| corvus | fungi: that's the only one | 00:46 |
| fungi | awesome | 00:46 |
| *** zhurong has joined #openstack-infra | 00:47 | |
| fungi | i included from_yaml() too and that seems to have done the trick, thanks! | 00:49 |
| *** TheJulia_ has joined #openstack-infra | 00:54 | |
| *** sthussey_ has joined #openstack-infra | 00:54 | |
| *** kmalloc_ has joined #openstack-infra | 00:54 | |
| *** zaro_ has joined #openstack-infra | 00:54 | |
| *** dmsimard5 has joined #openstack-infra | 00:54 | |
| *** srwilkers_ has joined #openstack-infra | 00:54 | |
| *** coreycb_ has joined #openstack-infra | 00:54 | |
| *** sreejithp has joined #openstack-infra | 00:55 | |
| *** csatari_ has joined #openstack-infra | 00:55 | |
| *** jamesmcarthur has quit IRC | 00:56 | |
| *** tinwood_ has joined #openstack-infra | 00:56 | |
| *** kjackal has quit IRC | 00:59 | |
| *** kjackal has joined #openstack-infra | 00:59 | |
| *** mriedem has joined #openstack-infra | 01:00 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: WIP: Support Ansible 2.8 https://review.openstack.org/631933 | 01:01 |
| *** sthussey has quit IRC | 01:02 | |
| *** sshnaidm has quit IRC | 01:02 | |
| *** harlowja has quit IRC | 01:02 | |
| *** kmalloc has quit IRC | 01:02 | |
| *** TheJulia has quit IRC | 01:02 | |
| *** HenryG has quit IRC | 01:02 | |
| *** coreycb has quit IRC | 01:02 | |
| *** srwilkers has quit IRC | 01:02 | |
| *** csatari has quit IRC | 01:02 | |
| *** zaro has quit IRC | 01:02 | |
| *** tinwood has quit IRC | 01:02 | |
| *** adam_g has quit IRC | 01:02 | |
| *** melwitt has quit IRC | 01:02 | |
| *** dmsimard has quit IRC | 01:02 | |
| *** kmalloc_ is now known as kmalloc | 01:02 | |
| *** srwilkers_ is now known as srwilkers | 01:02 | |
| *** sthussey_ is now known as sthussey | 01:02 | |
| *** zaro_ is now known as zaro | 01:02 | |
| *** TheJulia_ is now known as TheJulia | 01:02 | |
| *** coreycb_ is now known as coreycb | 01:02 | |
| *** csatari_ is now known as csatari | 01:02 | |
| *** dmsimard5 is now known as dmsimars | 01:03 | |
| *** dmsimars is now known as dmsimard | 01:03 | |
| *** gregoryo has joined #openstack-infra | 01:04 | |
| *** irclogbot_1 has quit IRC | 01:05 | |
| *** bgmccollum has quit IRC | 01:05 | |
| *** ricolin has joined #openstack-infra | 01:05 | |
| *** irclogbot_2 has joined #openstack-infra | 01:05 | |
| *** bgmccollum has joined #openstack-infra | 01:07 | |
| *** sshnaidm has joined #openstack-infra | 01:08 | |
| *** harlowja has joined #openstack-infra | 01:08 | |
| *** michael-beaver has quit IRC | 01:10 | |
| *** gyee has quit IRC | 01:20 | |
| *** michael-beaver has joined #openstack-infra | 01:20 | |
| *** whoami-rajat has joined #openstack-infra | 01:24 | |
| *** melwitt has joined #openstack-infra | 01:25 | |
| *** jamesmcarthur has joined #openstack-infra | 01:27 | |
| *** jamesmcarthur has quit IRC | 01:27 | |
| *** jamesmcarthur has joined #openstack-infra | 01:28 | |
| *** rlandy|ruck has quit IRC | 01:38 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: WIP: Support Ansible 2.8 https://review.openstack.org/631933 | 01:41 |
| *** bobh has joined #openstack-infra | 01:50 | |
| *** jamesmcarthur has quit IRC | 01:51 | |
| *** bobh has quit IRC | 01:51 | |
| *** jamesmcarthur has joined #openstack-infra | 01:54 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] rhel8 continuing work https://review.openstack.org/653646 | 01:55 |
| *** mriedem has quit IRC | 02:00 | |
| *** ykarel has joined #openstack-infra | 02:02 | |
| *** jamesmcarthur has quit IRC | 02:08 | |
| *** jamesmcarthur has joined #openstack-infra | 02:11 | |
| *** jamesmcarthur has quit IRC | 02:14 | |
| *** jamesmcarthur has joined #openstack-infra | 02:14 | |
| *** tkajinam has quit IRC | 02:15 | |
| *** jamesmcarthur has quit IRC | 02:17 | |
| *** rascasoft has quit IRC | 02:17 | |
| *** ykarel has quit IRC | 02:18 | |
| *** ykarel has joined #openstack-infra | 02:19 | |
| *** tkajinam has joined #openstack-infra | 02:19 | |
| *** rascasoft has joined #openstack-infra | 02:20 | |
| *** jamesmcarthur has joined #openstack-infra | 02:22 | |
| *** ykarel has quit IRC | 02:30 | |
| *** tkajinam has quit IRC | 02:31 | |
| *** nicolasbock has quit IRC | 02:36 | |
| *** diablo_rojo has quit IRC | 02:43 | |
| *** sreejithp has quit IRC | 02:47 | |
| *** sthussey has quit IRC | 02:56 | |
| *** jamesmcarthur has quit IRC | 03:00 | |
| *** jamesmcarthur has joined #openstack-infra | 03:01 | |
| *** psachin has joined #openstack-infra | 03:07 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:10 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 03:13 | |
| *** bhavikdbavishi has quit IRC | 03:15 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:15 | |
| *** jamesmcarthur has quit IRC | 03:20 | |
| *** jamesmcarthur has joined #openstack-infra | 03:21 | |
| timburke | swift | 03:22 |
| timburke | heh, sorry -- meant to search my buffer | 03:22 |
| *** michael-beaver has quit IRC | 03:40 | |
| *** tkajinam has joined #openstack-infra | 03:45 | |
| *** ykarel has joined #openstack-infra | 03:48 | |
| *** ykarel_ has joined #openstack-infra | 03:50 | |
| *** ykarel has quit IRC | 03:52 | |
| *** lpetrut has joined #openstack-infra | 03:54 | |
| *** jamesmcarthur has quit IRC | 04:07 | |
| *** imacdonn has quit IRC | 04:07 | |
| *** imacdonn has joined #openstack-infra | 04:08 | |
| *** lpetrut has quit IRC | 04:17 | |
| *** jamesmcarthur has joined #openstack-infra | 04:38 | |
| *** ramishra has joined #openstack-infra | 04:43 | |
| *** jamesmcarthur has quit IRC | 04:44 | |
| *** slaweq has joined #openstack-infra | 04:45 | |
| *** slaweq has quit IRC | 04:49 | |
| *** yboaron_ has joined #openstack-infra | 04:55 | |
| openstackgerrit | Riju Khatri proposed openstack-infra/storyboard-webclient master: Fix navigating back from the dashboard https://review.openstack.org/653656 | 04:56 |
| openstackgerrit | Riju Khatri proposed openstack-infra/storyboard-webclient master: Fix navigating back from the dashboard https://review.openstack.org/653656 | 04:58 |
| *** cloudnull has quit IRC | 05:02 | |
| *** d34dh0r53 has quit IRC | 05:02 | |
| *** eglute has quit IRC | 05:02 | |
| *** kjackal has quit IRC | 05:03 | |
| *** dpawlik has joined #openstack-infra | 05:04 | |
| *** jamesmcarthur has joined #openstack-infra | 05:07 | |
| *** jamesmcarthur has quit IRC | 05:17 | |
| rcernin | anyone from horizon can review https://github.com/zerodayz/openstack/blob/master/changes/Change_652427.md#check_message-function I was checking check_message vs handle handling the tabs.py and I think the check_message could be removed. It doesnt handle anything and when I tried to raise exception within check_message it just made horizon fail completely. When using the handle everything was one. | 05:23 |
| rcernin | fine* | 05:23 |
| rcernin | wrong channel sorry | 05:24 |
| *** lmiccini has joined #openstack-infra | 05:47 | |
| *** kopecmartin|off is now known as kopecmartin | 05:47 | |
| *** jamesmcarthur has joined #openstack-infra | 05:48 | |
| *** jamesmcarthur has quit IRC | 05:54 | |
| *** slaweq has joined #openstack-infra | 05:56 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: docs: fix a typo in executor zone documentation https://review.openstack.org/653663 | 06:07 |
| *** rascasoft has quit IRC | 06:13 | |
| *** lpetrut has joined #openstack-infra | 06:13 | |
| *** lpetrut has quit IRC | 06:14 | |
| *** lpetrut has joined #openstack-infra | 06:15 | |
| *** rascasoft has joined #openstack-infra | 06:15 | |
| *** e0ne has joined #openstack-infra | 06:17 | |
| *** e0ne has quit IRC | 06:18 | |
| *** lpetrut has quit IRC | 06:30 | |
| *** iurygregory has joined #openstack-infra | 06:38 | |
| *** pgaxatte has joined #openstack-infra | 06:43 | |
| *** jamesmcarthur has joined #openstack-infra | 06:50 | |
| *** jamesmcarthur has quit IRC | 06:55 | |
| *** kjackal has joined #openstack-infra | 06:57 | |
| *** ginopc has joined #openstack-infra | 07:02 | |
| *** ianw is now known as ianw_pto | 07:15 | |
| *** slaweq has quit IRC | 07:16 | |
| *** rpittau|afk is now known as rpittau | 07:17 | |
| *** iurygregory is now known as iurygregory_wfh | 07:17 | |
| *** roman_g has quit IRC | 07:19 | |
| *** lpetrut has joined #openstack-infra | 07:20 | |
| *** yboaron_ has quit IRC | 07:27 | |
| *** yboaron_ has joined #openstack-infra | 07:28 | |
| *** slaweq has joined #openstack-infra | 07:28 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: static: add host-checking toggle https://review.openstack.org/653679 | 07:29 |
| *** pcaruana has joined #openstack-infra | 07:32 | |
| *** shardy has joined #openstack-infra | 07:42 | |
| *** tosky has joined #openstack-infra | 07:42 | |
| *** ykarel_ is now known as ykarel | 07:44 | |
| *** ykarel is now known as ykarel|lunch | 07:45 | |
| *** jamesmcarthur has joined #openstack-infra | 07:51 | |
| *** jpich has joined #openstack-infra | 07:56 | |
| *** jamesmcarthur has quit IRC | 07:56 | |
| *** gregoryo has quit IRC | 08:02 | |
| *** roman_g has joined #openstack-infra | 08:03 | |
| *** ralonsoh has joined #openstack-infra | 08:16 | |
| zbr | pabelanger: clarkb : any chance of getting that bindep change merged? https://review.openstack.org/#/c/639951/ -- it has being months... | 08:25 |
| *** dtantsur|afk is now known as dtantsur | 08:25 | |
| *** dpawlik has quit IRC | 08:27 | |
| *** dpawlik has joined #openstack-infra | 08:27 | |
| *** lucasagomes has joined #openstack-infra | 08:29 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Also use selinuxenabled to check selinux status https://review.openstack.org/643367 | 08:32 |
| *** tkajinam has quit IRC | 08:32 | |
| openstackgerrit | Merged openstack/diskimage-builder master: debian-minimal buster support https://review.openstack.org/649496 | 08:32 |
| *** e0ne has joined #openstack-infra | 08:33 | |
| *** e0ne has quit IRC | 08:34 | |
| *** ykarel|lunch is now known as ykarel | 08:37 | |
| *** e0ne has joined #openstack-infra | 08:45 | |
| *** e0ne has quit IRC | 08:47 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: webtrigger: add web route and rpclistener https://review.openstack.org/554839 | 08:52 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add build button to trigger job https://review.openstack.org/635716 | 08:52 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: webtrigger: add support for parameterized trigger https://review.openstack.org/644484 | 08:52 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add build modal with a parameter form https://review.openstack.org/644485 | 08:52 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add support for checkbox and list parameters https://review.openstack.org/648661 | 08:52 |
| *** jamesmcarthur has joined #openstack-infra | 08:52 | |
| *** yboaron_ has quit IRC | 08:53 | |
| *** jamesmcarthur has quit IRC | 08:57 | |
| *** e0ne has joined #openstack-infra | 08:59 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: webtrigger: add web route and rpclistener https://review.openstack.org/554839 | 09:06 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add build button to trigger job https://review.openstack.org/635716 | 09:06 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: webtrigger: add support for parameterized trigger https://review.openstack.org/644484 | 09:06 |
| *** lpetrut has quit IRC | 09:10 | |
| *** markvoelker has joined #openstack-infra | 09:11 | |
| *** lyarwood has joined #openstack-infra | 09:24 | |
| *** jamesmcarthur has joined #openstack-infra | 09:26 | |
| *** jamesmcarthur has quit IRC | 09:30 | |
| *** electrofelix has joined #openstack-infra | 09:31 | |
| *** ramishra has quit IRC | 09:35 | |
| *** ramishra has joined #openstack-infra | 09:40 | |
| *** markvoelker has quit IRC | 09:41 | |
| *** ramishra has quit IRC | 09:51 | |
| *** ramishra has joined #openstack-infra | 09:51 | |
| *** lpetrut has joined #openstack-infra | 09:54 | |
| *** e0ne has quit IRC | 09:54 | |
| *** yboaron_ has joined #openstack-infra | 09:56 | |
| *** iurygregory_wfh has quit IRC | 10:00 | |
| *** e0ne has joined #openstack-infra | 10:00 | |
| *** markmcd has joined #openstack-infra | 10:04 | |
| sshnaidm | is there any example of periodic job configuration? | 10:18 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.openstack.org/576907 | 10:26 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Allow operator to generate auth tokens through the CLI https://review.openstack.org/636197 | 10:26 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Zuul CLI: allow access via REST https://review.openstack.org/636315 | 10:26 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Add Authorization Rules configuration https://review.openstack.org/639855 | 10:26 |
| *** jamesmcarthur has joined #openstack-infra | 10:26 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Web: plug the authorization engine https://review.openstack.org/640884 | 10:27 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: Zuul Web: add /api/user/authorizations endpoint https://review.openstack.org/641099 | 10:27 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul-jobs master: Support ssh-enabled windows hosts in add-build-sshkey https://review.openstack.org/653712 | 10:27 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: authentication config: add optional token_expiry https://review.openstack.org/642408 | 10:27 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [DNM] admin REST API: docker-compose PoC, frontend https://review.openstack.org/643536 | 10:27 |
| *** jamesmcarthur has quit IRC | 10:31 | |
| *** ttx has quit IRC | 10:31 | |
| *** ttx has joined #openstack-infra | 10:31 | |
| *** nicolasbock has joined #openstack-infra | 10:35 | |
| *** pcaruana has quit IRC | 10:36 | |
| *** markvoelker has joined #openstack-infra | 10:38 | |
| *** zigo has joined #openstack-infra | 10:51 | |
| *** pcaruana has joined #openstack-infra | 10:58 | |
| aspiers | corvus, mordred, fungi: Have you seen https://blog.acolyer.org/2019/04/18/keeping-master-green-at-scale/ ? (Well I guess that's a rhetorical question since it probably got released earlier while you were asleep ...) Anyway, the SubmitQueue sounds quite a bit like Zuul IIUC | 11:05 |
| aspiers | I've only skim-read it, but the speculation tree prioritises commits whose builds are more likely to succeed (no idea how it guesses this) and also high priority changes (e.g. security patches) - sounds interesting | 11:10 |
| *** markvoelker has quit IRC | 11:11 | |
| *** slaweq is now known as slaweq|lunch | 11:15 | |
| *** eharney has quit IRC | 11:25 | |
| *** panda is now known as panda|lunch | 11:26 | |
| *** jamesmcarthur has joined #openstack-infra | 11:27 | |
| *** shardy has quit IRC | 11:31 | |
| *** jamesmcarthur has quit IRC | 11:32 | |
| *** slaweq|lunch is now known as slaweq | 11:33 | |
| *** rh-jelabarre has joined #openstack-infra | 11:49 | |
| *** jcoufal has joined #openstack-infra | 11:50 | |
| *** ramishra has quit IRC | 11:52 | |
| *** bobh has joined #openstack-infra | 11:59 | |
| *** markvoelker has joined #openstack-infra | 12:08 | |
| *** Lucas_Gray has joined #openstack-infra | 12:11 | |
| *** ramishra has joined #openstack-infra | 12:11 | |
| *** jamesmcarthur has joined #openstack-infra | 12:15 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Fix Fedora aarch64 image location https://review.openstack.org/651283 | 12:22 |
| *** bobh has quit IRC | 12:23 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 12:25 | |
| *** bhavikdbavishi has quit IRC | 12:25 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 12:26 | |
| mordred | aspiers: yeah - sounds like zuul - except they've decided to give up and use a monorepo | 12:33 |
| *** rlandy|ruck has joined #openstack-infra | 12:34 | |
| openstackgerrit | boden proposed openstack-infra/project-config master: update vmware-nsx related jobs https://review.openstack.org/652680 | 12:37 |
| *** markvoelker has quit IRC | 12:41 | |
| *** eharney has joined #openstack-infra | 12:49 | |
| aspiers | mordred: monorepos sound like terrible ideas to me, the very antithesis of modular design... but hey, apparently it works for google so what do I know... | 12:51 |
| mordred | aspiers: well - I like to make jokes in presentations about google giving up on solving the hard problems | 12:51 |
| aspiers | X-D | 12:51 |
| openstackgerrit | sean mooney proposed openstack/gertty master: replace MultiQueue with PQueue https://review.openstack.org/653275 | 12:52 |
| openstackgerrit | sean mooney proposed openstack/gertty master: use timers for periodic tasks https://review.openstack.org/653276 | 12:52 |
| openstackgerrit | sean mooney proposed openstack/gertty master: use threadpool for syncing. https://review.openstack.org/653277 | 12:52 |
| *** yamamoto has joined #openstack-infra | 12:52 | |
| *** Lucas_Gray has quit IRC | 12:53 | |
| *** Lucas_Gray has joined #openstack-infra | 12:55 | |
| *** aaronsheffield has joined #openstack-infra | 12:55 | |
| *** Lucas_Gray has quit IRC | 12:57 | |
| *** jcoufal has quit IRC | 13:04 | |
| *** snapiri has quit IRC | 13:08 | |
| *** pcaruana has quit IRC | 13:10 | |
| *** jamesmcarthur has quit IRC | 13:13 | |
| *** yamamoto has quit IRC | 13:13 | |
| *** kgiusti has joined #openstack-infra | 13:15 | |
| *** jcoufal has joined #openstack-infra | 13:15 | |
| *** Goneri has joined #openstack-infra | 13:17 | |
| *** dpawlik has quit IRC | 13:22 | |
| *** mriedem has joined #openstack-infra | 13:24 | |
| *** rossella_s has quit IRC | 13:31 | |
| *** kaiokmo has joined #openstack-infra | 13:32 | |
| *** pcaruana has joined #openstack-infra | 13:32 | |
| *** markvoelker has joined #openstack-infra | 13:38 | |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack-infra/project-config master: [WIP] Retire openstack-ansible unused roles https://review.openstack.org/650422 | 13:40 |
| *** markvoelker has quit IRC | 13:42 | |
| *** lseki has joined #openstack-infra | 13:43 | |
| *** Emine has joined #openstack-infra | 13:50 | |
| *** yamamoto has joined #openstack-infra | 13:52 | |
| *** yamamoto has quit IRC | 13:55 | |
| *** yamamoto has joined #openstack-infra | 13:55 | |
| *** Emine has quit IRC | 13:58 | |
| *** Emine has joined #openstack-infra | 13:58 | |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack-infra/project-config master: Retire openstack-ansible unused roles https://review.openstack.org/650422 | 13:59 |
| *** jamesmcarthur has joined #openstack-infra | 13:59 | |
| *** yboaron_ has quit IRC | 14:01 | |
| *** sreejithp has joined #openstack-infra | 14:01 | |
| *** yboaron_ has joined #openstack-infra | 14:01 | |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 14:02 | |
| *** rossella_s has joined #openstack-infra | 14:05 | |
| *** sreejithp has quit IRC | 14:06 | |
| *** sreejithp has joined #openstack-infra | 14:08 | |
| *** lpetrut has quit IRC | 14:10 | |
| *** tinwood_ is now known as tinwood | 14:14 | |
| *** jaosorior has quit IRC | 14:15 | |
| guilhermesp | morning all! So, one of our cores just pointed out something that I might missed here https://review.openstack.org/#/c/650477/3/zuul.d/projects.yaml. I just added noop jobs for those roles that are going to be retired and were present in projects.yaml. The others I haven't added. Should I? | 14:20 |
| *** slittle1 has joined #openstack-infra | 14:25 | |
| *** markvoelker has joined #openstack-infra | 14:26 | |
| corvus | guilhermesp: there's no need to add noop jobs to a project unless there are no other jobs; so as long as the other projects aren't being retired and do have jobs running on them in check and gate, that's fine. | 14:27 |
| *** yamamoto has quit IRC | 14:28 | |
| *** samueldmq has joined #openstack-infra | 14:29 | |
| *** ricolin has quit IRC | 14:32 | |
| *** sthussey has joined #openstack-infra | 14:37 | |
| *** pgaxatte has quit IRC | 14:40 | |
| *** yolanda_ has quit IRC | 14:40 | |
| guilhermesp | thanks corvus | 14:42 |
| *** rkukura_ has joined #openstack-infra | 14:42 | |
| *** rkukura has quit IRC | 14:46 | |
| *** rkukura_ is now known as rkukura | 14:46 | |
| *** sreejithp has quit IRC | 14:47 | |
| AJaeger | guilhermesp: looking at https://review.openstack.org/#/c/653636/ and https://review.openstack.org/#/c/653635/ - you have no noop-jobs template in project-config. You need to add it. | 14:47 |
| AJaeger | So, you plan to retire those 2 repos - and you cannot merge those two changes without a noop job in project-config. | 14:48 |
| *** bdodd has quit IRC | 14:49 | |
| guilhermesp | yeah that's what I was thinking. I will -w those patches that doesn't have noop-jobs in project-config and submit a PR to add these messing roles with noop jobs | 14:49 |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack-infra/project-config master: Retire openstack-ansible unused roles https://review.openstack.org/653767 | 14:58 |
| guilhermesp | I appreciate reviews ^ :) | 14:58 |
| AJaeger | guilhermesp: LGTM, +2 | 14:59 |
| *** _erlon_ has joined #openstack-infra | 15:02 | |
| *** yamamoto has joined #openstack-infra | 15:03 | |
| *** kaiokmo has quit IRC | 15:05 | |
| *** kaiokmo has joined #openstack-infra | 15:06 | |
| guilhermesp | would be good to add it as depends-on here https://review.openstack.org/#/c/653636/ AJaeger ? | 15:08 |
| *** e0ne has quit IRC | 15:08 | |
| *** e0ne has joined #openstack-infra | 15:09 | |
| AJaeger | guilhermesp: would be nice - but not really needed. Fix the failure and try again instead ;) | 15:10 |
| *** yamamoto has quit IRC | 15:11 | |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack-infra/project-config master: Retire openstack-ansible unused roles https://review.openstack.org/653767 | 15:14 |
| guilhermesp | done :) | 15:14 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Use swift to back intermediate docker registry https://review.openstack.org/653613 | 15:14 |
| corvus | clarkb: ^ that should be gtg now | 15:14 |
| *** e0ne has quit IRC | 15:16 | |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 15:18 | |
| clarkb | aspiers: mordred reading that their partitions are our separate queues. And they repeat what I've tried to impress upon openstack: failures have a large impact. When it comes to the predictive stuff it is interesting that developer name is apparently a big predictor | 15:24 |
| corvus | that's a road i'll happily not go down | 15:25 |
| aspiers | haha | 15:25 |
| aspiers | interesting, I missed that in my skim-read | 15:25 |
| fungi | amusing indeed | 15:25 |
| aspiers | Still, the probabilistic approach is an interesting idea | 15:25 |
| mordred | yeah - that seems like a policy that might work inside of a company, but would be kind of contrary to our "our tools treat everyone the same" | 15:26 |
| corvus | zuul pipeline managers are pluggable; we have 3 :) | 15:26 |
| mordred | corvus: :) | 15:26 |
| *** sshnaidm has quit IRC | 15:27 | |
| corvus | which makes me think... we could have zuul reorder changes based on the review-priority label in gerrit | 15:28 |
| clarkb | worth noting their implementation does not currently order based on priority | 15:28 |
| corvus | yeah; we frequently do though | 15:28 |
| clarkb | but that could be a useful feature for us too | 15:28 |
| aspiers | I'm guessing they have no plans to open-source their solution | 15:30 |
| aspiers | but it's google so could happen I guess | 15:30 |
| clarkb | *uber | 15:31 |
| clarkb | but I think end result is about the same for chances of open sourcing :) | 15:31 |
| *** pcaruana has quit IRC | 15:31 | |
| aspiers | google own uber, I thought? | 15:31 |
| aspiers | or is that alphabet | 15:31 |
| clarkb | nope. They fight each other in court though | 15:31 |
| aspiers | Weird. I guess I must have been thinking of Waze | 15:32 |
| corvus | google was an early investor in uber | 15:33 |
| aspiers | ah ok | 15:33 |
| *** sshnaidm has joined #openstack-infra | 15:38 | |
| *** gyee has joined #openstack-infra | 15:38 | |
| *** bobh has joined #openstack-infra | 15:41 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: WIP: Sketch of a registry test which uses swift https://review.openstack.org/653797 | 15:44 |
| corvus | clarkb, mordred: ^ that's a rough sketch of what it might look like to use devstack to run a swift to back the registry in a zuul test job | 15:45 |
| corvus | it's probably also working up a sketch of the container aio container approach | 15:45 |
| corvus | though it just occured to me -- do we need keystone to make this a valid test? i'm guessing https://hub.docker.com/r/bouncestorage/swift-aio doesn't use it | 15:46 |
| corvus | (the devstack version does use keystone) | 15:46 |
| clarkb | corvus: swift without keystone uses basicauth which I think doesn't need all your tenant type info | 15:47 |
| clarkb | considering that we will be using keystoned swift in production we probably want to test swift with keystone | 15:47 |
| corvus | clarkb: yeah, so that would be a production difference. agreed | 15:47 |
| corvus | (otherwise we have to put a bunch of conditionals in the production jinja template) | 15:47 |
| corvus | tdasilva: do you know if someone has made a swift container image with keystone? :) | 15:48 |
| tdasilva | corvus: in fact 1space does run bouncestorage/swift-aio with keystone...one sec...i'll point you to the repo and let you checkout the Dockerfiles... | 15:49 |
| AJaeger | config-core, anybody around for a quick review of 653767 to add some noop jobs, please? | 15:49 |
| corvus | AJaeger: +3 | 15:50 |
| *** lucasagomes has quit IRC | 15:50 | |
| AJaeger | thanks, corvus | 15:51 |
| tdasilva | corvus: checkout https://github.com/swiftstack/1space/tree/master/containers | 15:51 |
| corvus | tdasilva: thanks! | 15:51 |
| tdasilva | you just need to run a separate keystone container and then modify proxy-server.conf to point to that keystone | 15:52 |
| tdasilva | like here: https://github.com/swiftstack/1space/blob/master/containers/swift-s3-sync/proxy-server.conf | 15:52 |
| *** mattw4 has joined #openstack-infra | 15:52 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Support Ansible 2.8 https://review.openstack.org/631933 | 15:54 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Remove support for ansible 2.5 https://review.openstack.org/650431 | 15:54 |
| *** eernst has joined #openstack-infra | 15:54 | |
| *** eernst has quit IRC | 15:56 | |
| *** eernst has joined #openstack-infra | 15:56 | |
| *** eernst has quit IRC | 15:57 | |
| *** eernst has joined #openstack-infra | 15:58 | |
| *** armstrong has joined #openstack-infra | 15:58 | |
| *** eernst has quit IRC | 16:00 | |
| *** mattw4 has quit IRC | 16:02 | |
| clarkb | infra-root https://review.openstack.org/#/c/653499/ is an easy update to our specification of a gpg key id to make puppet-4 happy on the nodepool builders (its just a warning as is, but cleaning those up as much as possible makes it easier to find the real issues when puppet has a sad) | 16:02 |
| openstackgerrit | Merged openstack-infra/project-config master: Retire openstack-ansible unused roles https://review.openstack.org/653767 | 16:02 |
| corvus | clarkb: +1 i have confirmed that number is bigger than the old number. | 16:03 |
| clarkb | ha | 16:03 |
| clarkb | in reality its not a concern because we vendor the public key itself | 16:04 |
| clarkb | but I like not seeing things that look like errors | 16:04 |
| *** ginopc has quit IRC | 16:06 | |
| clarkb | infra-root I'll plan to reduce dns ttls near the end of my day today | 16:06 |
| clarkb | per https://etherpad.openstack.org/p/the-great-opendev-git-migration | 16:06 |
| *** jpich has quit IRC | 16:06 | |
| clarkb | For tomorrow morning if anyone is around a little earlier than 1500UTC it would be good to get the ansible cron on bridge disabled early | 16:06 |
| clarkb | and I think the last thing we are waiting on are fungi's generated changes | 16:07 |
| fungi | sounds good. i'm still plugging away at that script yes | 16:07 |
| clarkb | that and people reviewing things like corvus' playbook and the results of fungi's changes | 16:07 |
| fungi | and doing some baseline testing across the full set of repos/branches as i go | 16:07 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: WIP - Pagure driver - https://pagure.io/pagure/ https://review.openstack.org/604404 | 16:07 |
| clarkb | https://review.openstack.org/653507 and https://review.openstack.org/653614 are redy for reviewing now if you have a moment | 16:07 |
| fungi | so far i've got it cloning each bare repo to a tempdir, checking out each branch, parsing any zuul configs it finds, using those to locate job playbooks and also to find sibling roles in those playbook directories or in any top-level roles directory | 16:08 |
| fungi | that alone takes ~6 minutes over the full body of repos | 16:08 |
| fungi | oh, also it loads the renames.yaml into a dict structure for lookups | 16:09 |
| fungi | working now on the code which churns through those files line by line looking for matches | 16:09 |
| *** panda|lunch is now known as panda | 16:11 | |
| *** ykarel is now known as ykarel|away | 16:15 | |
| *** bobh has quit IRC | 16:16 | |
| *** eernst has joined #openstack-infra | 16:19 | |
| fungi | interestingly, python's re.compile() seems not to break a sweat with a regular expression many thousands of characters in length | 16:21 |
| fungi | that will make this a bit simpler to implement | 16:22 |
| clarkb | hopefully it is ok with evaluating them too :) | 16:22 |
| *** ramishra has quit IRC | 16:22 | |
| *** ykarel|away has quit IRC | 16:22 | |
| fungi | i'll know one way or the other shortly | 16:22 |
| *** eernst has quit IRC | 16:23 | |
| *** bdodd has joined #openstack-infra | 16:26 | |
| * fungi sighs at random binary files in some folks' roles directories | 16:27 | |
| corvus | fungi: cover your eyes! do not look! | 16:27 |
| fungi | yesterday i found projects with zero-byte zuul configs. that's awesome | 16:27 |
| corvus | la la la la la | 16:28 |
| fungi | it's interesting what you find when you turn over *all* the rocks | 16:28 |
| *** bobh has joined #openstack-infra | 16:29 | |
| fungi | is it safe to limit edits for files in "roles" trees to just those ending in .yaml or .yml? that may be the easy way out | 16:29 |
| corvus | fungi: that would get all the ansible tasks. it could miss templated config files, etc, but we're getting into decreasing likelihood of needing those kinds of changes i think. | 16:30 |
| fungi | not wanting to futz with the mime magic module just so i can skip jpegs people have put in their roles (what?!?) | 16:31 |
| fungi | i can include other extensions... .j2? | 16:31 |
| *** rcernin has quit IRC | 16:32 | |
| corvus | fungi: that'd be good... that should handle that case | 16:33 |
| *** kgiusti has left #openstack-infra | 16:33 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Constraint networkx to <2.3 for Python 2 https://review.openstack.org/652063 | 16:33 |
| fungi | ginormous compiled regex matches great, and still very fast! | 16:37 |
| clarkb | nice | 16:37 |
| fungi | /opt/fungi-testing/git/openstack/mixmatch.git | 16:42 |
| fungi | [master] playbooks/legacy/mixmatch-dsvm-functional/run.yaml (openstack/mixmatch): export DEVSTACK_LOCAL_CONFIG="enable_plugin mixmatch https://git.openstack.org/openstack/mixmatch" | 16:42 |
| *** kopecmartin is now known as kopecmartin|off | 16:42 | |
| corvus | that looks like an appropriate line to rewrite | 16:42 |
| clarkb | ++ though I think the url there is unused in a zuul context | 16:43 |
| clarkb | (it is required argment to enable_plugin though) | 16:43 |
| fungi | yup, just have it spitting out matches right now so i can be sure it's doing what i want before i start writing them out to tempfiles | 16:43 |
| fungi | but that's a playbook it found via a reference in local zuul configuration, so that much is working too | 16:44 |
| *** rcernin has joined #openstack-infra | 16:45 | |
| fungi | oh, neat, just found a project which has zuul configuration referencing a nonexistent playbook | 16:48 |
| fungi | i think, still digging | 16:48 |
| *** e0ne has joined #openstack-infra | 16:48 | |
| *** igordc has joined #openstack-infra | 16:49 | |
| fungi | openstack/congress-tempest-plugin master branch has configuration referencing a playbooks/legacy/congress-legacy-pe-replicated-base/run.yaml | 16:49 |
| *** igordc has quit IRC | 16:49 | |
| fungi | https://opendev.org/openstack/congress-tempest-plugin/src/branch/master/.zuul.yaml#L69 | 16:50 |
| *** dtantsur is now known as dtantsur|afk | 16:50 | |
| *** armax has quit IRC | 16:50 | |
| clarkb | https://opendev.org/openstack/congress-tempest-plugin/src/branch/master/playbooks/legacy/congress-pe-replicated-base/run.yaml is probably what they mean but ya thats broken | 16:51 |
| *** ykarel has joined #openstack-infra | 16:51 | |
| *** e0ne has quit IRC | 16:52 | |
| fungi | i mean, i should be checking these files exist before i try to open them, so... os.path.exists() to the rescue | 16:52 |
| *** e0ne has joined #openstack-infra | 16:52 | |
| fungi | but still amusing all the corner cases i'm turning up | 16:52 |
| clarkb | and I think we'll just have to accept that we can't hand curate all that broken stuff and if we miss things beacuse their configs were broken already well tough luck | 16:52 |
| mordred | speaking of - corvus, we don't validate that playbooks referenced in jobs exist because it woudl take another merger round trip, right? | 16:52 |
| fungi | clarkb: yep, i'm already *well* into the acceptance phase | 16:53 |
| clarkb | I've added a bit more specifics/detail to the etherpad plan doc | 16:55 |
| clarkb | commands for db backups and reindexing and specific list of dns records to change ttls for | 16:55 |
| *** tosky has quit IRC | 16:56 | |
| *** bobh has quit IRC | 16:57 | |
| *** e0ne has quit IRC | 16:58 | |
| clarkb | fungi: don't let this nerd snipe you but for your weekend entertainment https://github.com/historicalsource/zork source for zork and many other classic games on that github account | 17:01 |
| fungi | oh, i have a ton of the original files and just run them under frotz | 17:02 |
| fungi | though source code, nice! | 17:02 |
| *** pcaruana has joined #openstack-infra | 17:02 | |
| fungi | okay, at this stage the script is reading all the identified files successfully and checking them against the massive list of project renames, and also creating tempfiles to write the modified versions out to (though not actually writing those yet). total runtime across the full set of repos/branches is now up to 6m42.185s (real) on review.o.o | 17:02 |
| *** jchhatbar has joined #openstack-infra | 17:03 | |
| fungi | as long as the git commit and push operations aren't super slow, this will likely complete within a reasonable timeframe | 17:03 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/openstack-zuul-jobs master: Drop legacy-grenade-dsvm-neutron-multinode-live-migration https://review.openstack.org/640209 | 17:06 |
| *** diablo_rojo has joined #openstack-infra | 17:07 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/openstack-zuul-jobs master: Drop legacy-grenade-dsvm-neutron-multinode-live-migration https://review.openstack.org/640209 | 17:07 |
| clarkb | fungi: ya in the original dialect of lisp apparently | 17:08 |
| *** jchhatbar has quit IRC | 17:08 | |
| mordred | clarkb: I had no idea zork was in lisp | 17:10 |
| clarkb | mordred: the readme says "It is written in ZIL (Zork Implementation Language), a refactoring of MDL (Muddle), itself a dialect of LISP created by MIT students and staff" | 17:11 |
| clarkb | mordred: I've got a note on the etherpad that says "Update hostname on review.openstack.org (our puppet stuff seems to assume $::fqdn for things)" | 17:15 |
| clarkb | mordred: you are the other person that poked around the puppetry with me. Thoughts on whether or not we should do that? | 17:16 |
| *** jamesmcarthur_ has joined #openstack-infra | 17:16 | |
| clarkb | I'm now a bit conflicted because I don't want nova api and host to say different things | 17:16 |
| mordred | clarkb: I don't think we need to. I do think we probably should, you know - just to make things consistent | 17:16 |
| clarkb | so thinking maybe we don't change that and instead update puppetry as needed? | 17:16 |
| mordred | ah - that's a good point | 17:16 |
| mordred | yeah | 17:16 |
| mordred | I thnik updating the puppetry is fine (Which I believe we did in that patch) | 17:16 |
| mordred | and then next time we replace the gerrit server we can boot it as review.opendev.org | 17:17 |
| clarkb | ya | 17:17 |
| mordred | reviewXX that is | 17:17 |
| mordred | but you know | 17:17 |
| clarkb | whcih leads to another question should we do review.opendev.org CNAME review.openstack.org instead of review.openstack.org CNAME review.opendev.org ? None of this is super critical before tomorrow and can be switched around in the future | 17:18 |
| *** jamesmcarthur has quit IRC | 17:19 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Add a opendev migration repo rename scripts https://review.openstack.org/653138 | 17:20 |
| fungi | added the current state of the patching script ^ if people want to start reviewing what i have there so far | 17:20 |
| fungi | hopefully everything i haven't done yet has a todo comment as well | 17:20 |
| fungi | gonna break for lunch before i get back to it | 17:21 |
| *** kgiusti has joined #openstack-infra | 17:21 | |
| corvus | mordred: re merger roundtrip for playbooks: yes | 17:25 |
| corvus | i'm going to be laggy today as i do summit prep stuff | 17:26 |
| *** psachin has quit IRC | 17:27 | |
| clarkb | fungi: only gotten through opendev-migrate so far and it looks good. Did leave one question though | 17:29 |
| *** yboaron_ has quit IRC | 17:29 | |
| *** yboaron_ has joined #openstack-infra | 17:29 | |
| corvus | fungi: your use of 'deli' as a variable name is making me hungry | 17:32 |
| *** mattw4 has joined #openstack-infra | 17:41 | |
| *** ralonsoh has quit IRC | 17:43 | |
| *** rpittau is now known as rpittau|afk | 17:49 | |
| clarkb | I see the large regex and approve of this hack :) | 17:50 |
| clarkb | fungi: the file editing script looks good as well and todo list covers the things that I think need to be done | 17:58 |
| *** mattw4 has quit IRC | 18:03 | |
| *** electrofelix has quit IRC | 18:07 | |
| *** rkukura has quit IRC | 18:08 | |
| *** jamesmcarthur_ has quit IRC | 18:09 | |
| openstackgerrit | Kendall Nelson proposed openstack-infra/storyboard master: Link development.rst to contributing.rst https://review.openstack.org/645960 | 18:11 |
| *** rkukura has joined #openstack-infra | 18:17 | |
| clarkb | I added verification that https://git.openstack.org/openstack-infra/project-config redirects work after we update DNS to the etherpad steps. This is the project_config_repo url value we use in puppet so we'll need that to succeed to puppet runs to work | 18:22 |
| clarkb | If you can think of any other intermediate verification steps please add them to the document | 18:22 |
| openstackgerrit | Jason Lee proposed openstack-infra/storyboard master: WIP: Corrected pep8 errors and assorted typos https://review.openstack.org/649666 | 18:28 |
| *** jamesmcarthur has joined #openstack-infra | 18:28 | |
| cmurphy | could i ask for a couple of specs site cleanups: keystone-specs/specs/keystone/ongoing/python3.html and keystone-specs/specs/keystone/ongoing/role-check-from-middleware.html were moved to other directories but they're still stuck on http://specs.openstack.org/openstack/keystone-specs/specs/keystone/ongoing/ | 18:29 |
| *** sshnaidm is now known as sshnaidm|off | 18:32 | |
| corvus | cmurphy: sure thing | 18:37 |
| corvus | cmurphy: done | 18:38 |
| *** armstrong has quit IRC | 18:38 | |
| cmurphy | thanks corvus | 18:38 |
| corvus | those look like the only two out-of-date files in that dir | 18:38 |
| *** ykarel is now known as ykarel|away | 18:44 | |
| *** jcoufal has quit IRC | 18:47 | |
| *** diablo_rojo has quit IRC | 18:49 | |
| openstackgerrit | Jason Lee proposed openstack-infra/storyboard master: WIP: Corrected pep8 errors (again) and assorted typos https://review.openstack.org/649666 | 18:56 |
| *** diablo_rojo has joined #openstack-infra | 18:58 | |
| *** eernst has joined #openstack-infra | 19:02 | |
| *** diablo_rojo_ has joined #openstack-infra | 19:03 | |
| *** diablo_rojo has quit IRC | 19:04 | |
| *** diablo_rojo_ is now known as diablo_rojo | 19:04 | |
| *** eernst has quit IRC | 19:07 | |
| smcginnis | fungi or others: We had that weird ensure-twine failure on the solum release earlier this week. Wondering if we can just reenqueue that to see if it was a random failure or if we need to get out the shovels and pick axes. | 19:13 |
| smcginnis | http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005255.html | 19:14 |
| *** e0ne has joined #openstack-infra | 19:14 | |
| clarkb | smcginnis: I would be surprised if an error like that is a one off. Is the python3 interpreter there running in a virtualenv? if so it seems new pip won't touch the --user env | 19:15 |
| mnaser | infra-root: regarding opendev, 653360, 653446, 653447, 653450 and 653449 have landed which were needed to make sure we avoid renaming certain projects. | 19:15 |
| mnaser | so that should be in place for tomorrow. | 19:15 |
| clarkb | mnaser: thanks. fungi's scripting should pick all the at up | 19:15 |
| mnaser | sounds good, they're in the gate now | 19:16 |
| smcginnis | clarkb: It shouldn't be running in a virtualenv. None of our jobs set things up that way, and this job has been working for others before and after that failure. | 19:16 |
| smcginnis | clarkb: There was a hunch that maybe an in-repo job could have set up a virtualenv, but I don't even know if it's possible that would have impacted this role execution. | 19:16 |
| corvus | mnaser, clarkb: oh, er, well, we should keep an eye on them to make sure they actually merge; once they do fungi's script will pick them up :) | 19:16 |
| openstackgerrit | Jason Lee proposed openstack-infra/storyboard master: WIP: Corrected pep8 errors (hat trick) and assorted typos https://review.openstack.org/649666 | 19:17 |
| mnaser | corvus: cool, I'll ping again when it's landed for whoever needs to verify | 19:17 |
| smcginnis | clarkb: We figured if we reenqueu and it happens again, it would be a clear sign someone needs to dig through the sequence of events and figure out what's unique about this situation. | 19:17 |
| clarkb | corvus: oh I read landed to mean merged | 19:18 |
| mnaser | yes, i should have been a bit clear, sorry :> | 19:18 |
| *** e0ne has quit IRC | 19:18 | |
| clarkb | smcginnis: I have confirmed you get that error when doing pip install --user from within a venv | 19:19 |
| smcginnis | Yeah | 19:19 |
| *** rh-jelabarre has quit IRC | 19:20 | |
| clarkb | twine python is just `python3` as implied by the logging | 19:20 |
| clarkb | so I wonder if ubuntu broke packaging such that python3 looks like a virtualenv | 19:21 |
| clarkb | smcginnis: I'll queue up a rerun in a sec and we can see how persistent it is | 19:21 |
| clarkb | smcginnis: fwiw I like to do a minimal amount of debugging to understand the problem before retrying :) | 19:22 |
| clarkb | smcginnis: `zuul enqueue-ref --tenant openstack --trigger gerrit --pipeline release --project openstack/solum --ref refs/tags/5.4.1 --newrev 19a7574237f44807b16c37e0983223ff57340ba3` look correct to you? | 19:23 |
| *** rh-jelabarre has joined #openstack-infra | 19:23 | |
| smcginnis | clarkb: Me too, but fungi, tonyb[m] and maybe others looked at it and weren't sure. | 19:24 |
| smcginnis | Command looks correct to me. | 19:24 |
| clarkb | smcginnis: done | 19:25 |
| *** hwoarang has quit IRC | 19:26 | |
| smcginnis | Thanks clarkb | 19:26 |
| *** hwoarang has joined #openstack-infra | 19:27 | |
| *** lyarwood has quit IRC | 19:30 | |
| mnaser | fungi, corvus: the changes have all landed | 19:34 |
| *** mujahidali has joined #openstack-infra | 19:41 | |
| mujahidali | clarkb: I want to build an ubuntu image for ppc64le using disk-imagebuilder, Is there any element for it or any other config? | 19:43 |
| clarkb | mujahidali: I am not aware of anyone doing that today. Vorrtex in #openstack-dib is working on it for centos and octavia images | 19:43 |
| clarkb | mujahidali: might be able to figure out common bits between the distro together in that channel | 19:44 |
| mujahidali | clarkb: thanks :) | 19:44 |
| *** diablo_rojo has quit IRC | 19:46 | |
| *** ykarel|away has quit IRC | 19:54 | |
| *** eharney has quit IRC | 19:55 | |
| *** jamesmcarthur has quit IRC | 19:58 | |
| smcginnis | clarkb: That rerun appears to have been fine. | 20:05 |
| smcginnis | *shrug* | 20:05 |
| smcginnis | Guess we'll just have to watch for something like it happening again. | 20:05 |
| *** michael-beaver has joined #openstack-infra | 20:05 | |
| *** armax has joined #openstack-infra | 20:19 | |
| *** tjgresha has joined #openstack-infra | 20:22 | |
| *** jamesmcarthur has joined #openstack-infra | 20:24 | |
| *** diablo_rojo has joined #openstack-infra | 20:24 | |
| *** kaiokmo has quit IRC | 20:24 | |
| *** tjgresha has quit IRC | 20:26 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: WIP: Sketch of a registry test which uses swift https://review.openstack.org/653797 | 20:27 |
| *** kgiusti has left #openstack-infra | 20:32 | |
| fungi | glad i pushed up my work in progress before lunch, because the whole island has been without broadband for 2.5 hours now and counting (but i'm continuing work via a wireless modem) | 20:33 |
| *** kjackal has quit IRC | 20:34 | |
| fungi | mnaser: clarkb: corvus: anything which doesn't merge in time *can* be added to the ethercalc and it takes precedence as of my recent updates to that script (in part for that reason) | 20:35 |
| *** jamesmcarthur has quit IRC | 20:35 | |
| *** jcoufal has joined #openstack-infra | 20:38 | |
| *** pcaruana has quit IRC | 20:47 | |
| clarkb | fungi: roger | 20:47 |
| *** jamesmcarthur has joined #openstack-infra | 20:48 | |
| clarkb | hopefully your internet woes aren't affecting your hackery too much | 20:48 |
| *** hwoarang has quit IRC | 20:50 | |
| *** ijw has joined #openstack-infra | 20:50 | |
| *** hwoarang has joined #openstack-infra | 20:51 | |
| fungi | nah, i was actually writing the script in my homedir on review.o.o for ease of performance/functional testing | 20:54 |
| *** efried has quit IRC | 20:55 | |
| *** ijw has quit IRC | 20:55 | |
| *** efried has joined #openstack-infra | 20:56 | |
| mnaser | is ovh-bhs1 seeing problems? | 20:57 |
| mnaser | http://logs.openstack.org/92/649892/13/check/openstack-ansible-deploy-aio_distro_metal-centos-7/0e486a2/job-output.txt.gz#_2019-04-18_20_46_49_936562 | 20:57 |
| mnaser | I had another failure earlier too (don't have the logs) where it timed out trying to hit files.pythonhosted.org too | 20:58 |
| *** jamesmcarthur has quit IRC | 20:59 | |
| clarkb | http://status.ovh.com/ doesn't show any recent network issues | 20:59 |
| clarkb | though the little bubble is red (not sure if that means much if there aren't incidents) | 21:00 |
| clarkb | mnaser: unfortunately network issues are typically opaque to us | 21:01 |
| mnaser | bummer | 21:01 |
| clarkb | zuul attempts to retry jobs that fail due to network failures if it can detect that situation | 21:01 |
| mnaser | our centos jobs are like 2h10m long or so | 21:01 |
| clarkb | (it relies on ansible exit codes) | 21:01 |
| mnaser | so it throws us off for quite sometime unfortunately :< | 21:01 |
| *** jamesmcarthur has joined #openstack-infra | 21:02 | |
| *** ijw has joined #openstack-infra | 21:02 | |
| *** Goneri has quit IRC | 21:03 | |
| clarkb | I wonder if that code has regressed with the support for newer ansibles | 21:05 |
| *** mujahidali has quit IRC | 21:08 | |
| *** ijw has quit IRC | 21:08 | |
| *** yamamoto has joined #openstack-infra | 21:09 | |
| corvus | clarkb: i don't follow? | 21:11 |
| clarkb | corvus: mnaser's example had ssh failures but the job doesn't seem to have been retried? or maybe it is and I'm not able to see that | 21:11 |
| mnaser | I see it stuck in post_failure now | 21:11 |
| mnaser | but a lot of our jobs are just queued .. so I dunno. | 21:12 |
| clarkb | http://logs.openstack.org/92/649892/13/check/openstack-ansible-deploy-aio_distro_metal-centos-7/0e486a2/job-output.txt.gz#_2019-04-18_20_51_18_192663 | 21:12 |
| clarkb | oh wait that is the rsync failure which is different than the main ssh connection | 21:12 |
| clarkb | so maybe not a regression, we probably just can't detect that specific rsync case | 21:12 |
| *** ijw has joined #openstack-infra | 21:13 | |
| *** yamamoto has quit IRC | 21:14 | |
| corvus | yeah, it doesn't show up as 'unreachable' in the play recap | 21:15 |
| corvus | neither does this: http://logs.openstack.org/92/649892/13/check/openstack-ansible-deploy-aio_distro_metal-centos-7/0e486a2/job-output.txt.gz#_2019-04-18_20_46_49_991742 | 21:16 |
| mnaser | is this something that OSA is doing wrong? | 21:22 |
| clarkb | mnaser: no it is a limitation of zuul's ability to detect network disruptions | 21:22 |
| corvus | not that i can see; i think it's just inconsintency internal to ansible about when it detects a host is unreachable | 21:22 |
| mnaser | ok, just wanted to make sure, thanks | 21:23 |
| corvus | clarkb: well, in that zuul is limited to ansible's ability, yes :) | 21:23 |
| clarkb | corvus: right | 21:23 |
| *** jamesmcarthur has quit IRC | 21:29 | |
| *** jamesmcarthur has joined #openstack-infra | 21:30 | |
| *** diablo_rojo has quit IRC | 21:36 | |
| corvus | clarkb: it looks like nodepool is preferring the ipv6 address in vexxhost-sjc1 now, which is triggering the docker bug there... do you know why that's preferred but we use ipv4 in rax? | 21:37 |
| corvus | mordred, Shrews: ^ this may be an openstacksdk q | 21:38 |
| clarkb | corvus: there is a nodepool setting to prefer ipv4 iirc we have to set it on ovh because the ovh api returns v6 info but the instances don't ahev that data in the config drive | 21:38 |
| clarkb | I didn't think we run rax that way but perhaps we do? | 21:38 |
| mordred | corvus: I don't think we configure ipv6 in rax nodepool nodes | 21:39 |
| clarkb | rax is special too because they don't do RAs you have to staticly configure it via glean | 21:39 |
| corvus | rax -- http://logs.openstack.org/13/653613/3/check/system-config-run-docker-registry/16d158e/zuul-info/inventory.yaml | 21:39 |
| clarkb | that static ipv6 config should work on debuntu on rax with glean but not on centos or fedora | 21:39 |
| corvus | vexxhost -- http://logs.openstack.org/13/653613/3/gate/opendev-buildset-registry/04d3d61/zuul-info/inventory.yaml | 21:39 |
| mordred | oh - so we do have it | 21:39 |
| mordred | weird | 21:39 |
| clarkb | thinking about it more we may set it to ipv4 on rax because of the centos/fedora issue | 21:40 |
| mordred | oh yeah | 21:40 |
| clarkb | I don't recall if that is a provider setting or a cloud config setting | 21:41 |
| clarkb | but that is what I would check first | 21:41 |
| corvus | i don't see it in nodepool, i'll check clouds.yaml | 21:41 |
| corvus | force_ipv4: true | 21:41 |
| corvus | rax in clouds.yaml ^ | 21:41 |
| corvus | clarkb: do you have a link to your docker bug handy? | 21:42 |
| clarkb | corvus: ya one sec | 21:42 |
| clarkb | https://github.com/moby/moby/issues/39033 there is a link to my pr there to | 21:42 |
| fungi | it's a "moby" bug i guess? | 21:43 |
| *** jamesmcarthur has quit IRC | 21:43 | |
| clarkb | fungi: moby is the community version of docker | 21:43 |
| clarkb | the PR is actually against docker/distribution which is a library that bundles up image pushing and pulling stuff | 21:43 |
| fungi | could then be part of the problem is your bug is in the open core bits? | 21:44 |
| clarkb | possibly, though the open core bits had the issue filing hand hold template | 21:44 |
| clarkb | hrm I wonder | 21:45 |
| clarkb | mordred: corvus any idea what the default port is for docker? | 21:45 |
| clarkb | er docker registry? if we drop the :portvalue we can drop the []s too maybe | 21:46 |
| clarkb | then possibly this would work? | 21:46 |
| corvus | i don't know if it's 5000 or 443 or 80 | 21:46 |
| corvus | i will experiment | 21:46 |
| clarkb | rereading my regex PR made me realize that we might be able to sneak through the ipv6 address on that | 21:47 |
| clarkb | hrm no I think not | 21:48 |
| corvus | it appears not to be 5000 | 21:48 |
| corvus | it's 80 or 443 | 21:48 |
| clarkb | they don't allow : separators either | 21:48 |
| clarkb | so my PR is insufficient to fix it I think | 21:48 |
| clarkb | yay for custom url parsing | 21:49 |
| mordred | clarkb: writing everything from scratch is the new hotness. using libraries is what old people do | 21:52 |
| corvus | okay, well this change has caused our chances of hitting that error to increase significantly | 21:52 |
| corvus | should we set vexxhost to force_ipv4 to true? | 21:52 |
| corvus | (obviously that should be a short-term thing until we find a better fix eg skopeo) | 21:53 |
| mordred | corvus: it makes me sad, but yes, I think so short term | 21:53 |
| clarkb | considering we already do it in two other clouds I'm not aopposed | 21:53 |
| clarkb | corvus: re skopeo they have refused my bug and wish to maintain bug compat with dicker | 21:54 |
| clarkb | er docker | 21:54 |
| corvus | clarkb: can you point me at that? | 21:54 |
| clarkb | https://github.com/containers/skopeo/issues/631 | 21:54 |
| *** rlandy|ruck is now known as rlandy | 21:57 | |
| mordred | clarkb, corvus: I have two ideas for workarounds | 22:02 |
| mordred | both assume a baseline of clarkb's patch to write entries to /etc/hosts on remote nodes | 22:03 |
| mordred | a) modify zuul to, instead of bind-mounting host /etc/hosts into the bubblewrap container, make a copy of the /etc/hosts file and bind-mount the copy in r/w - so that it's safe to edit /etc/hosts in the bubblewrap execution context | 22:04 |
| mordred | or | 22:04 |
| mordred | b) for the operatios on the executor that need to interact with the buildset registry, ssh-tunnel to the buildset registry with a port-forward, and then use 127.0.0.1:$forwarded_port as the registry location in the config on the executor | 22:05 |
| corvus | the rejection from skopeo combined with my knowledge of docker's responsiveness to issues makes me think it's time to push more on workarounds :) | 22:05 |
| clarkb | it does make me wonder why write a different tool if you are going to maintain the bugset of the other tool | 22:05 |
| mordred | editing /etc/hosts on the remote nodes is straightfoward and non-hacky | 22:05 |
| corvus | clarkb: right? | 22:05 |
| clarkb | I guess the answer is that the only accepted bug they are fixing is the requirement for the daemon | 22:05 |
| clarkb | everything else is a feature | 22:05 |
| mordred | so it's really only the executor/buildset-registry interaction that has to be solved | 22:06 |
| fungi | or serves as a convenient excuse to reject bug fixes | 22:06 |
| clarkb | mordred: I like the ssh proxy more than writeable /etc/hosts | 22:06 |
| clarkb | I worry that writeable /etc/hosts could be abused in ways we don't actually want | 22:06 |
| openstackgerrit | Jason Lee proposed openstack-infra/storyboard master: WIP: Included Blueprint Reader and Loader https://review.openstack.org/649666 | 22:06 |
| clarkb | (like connecting to locahost or other test nodes or whatever) | 22:06 |
| mordred | clarkb: funny, I think I like writable hosts better myself - and we don't prevent connecting to localhost or other test nodes in any way ... | 22:07 |
| corvus | mordred: we may need to add "127.0.0.1" as a valid subject name to the cert we generate for the buildset registry | 22:07 |
| mordred | other than per-build ssh keys | 22:07 |
| mordred | corvus: oh - that's a good point | 22:07 |
| mordred | clarkb: but - I don't like writable hosts more STRONGLY - so I'd be fine with tunnels and adding 127.0.0.1 as a valid subject name | 22:08 |
| corvus | mordred: i think that should be straightforward and should work -- we just generate them with the current ip address now; but it should be fine to include it as a SAN | 22:08 |
| mordred | cool | 22:08 |
| clarkb | mordred: we prevent connecting to localhost and without being able to udpate /etc/hosts to point back at yourself I think that is the case? | 22:08 |
| mordred | clarkb: nah - ssh key is still going to block trying to trick ansible by modifying /etc/hosts | 22:09 |
| mordred | because the localhost block is about blocking things that would cause ansible to default to using the local connection plugin instead of ssh | 22:09 |
| mordred | which are strings localhost, 127.0.0.1 as the hostname | 22:10 |
| clarkb | got it and is that generally true for people not using our base jobs ? though I guess we should really encourage per job ssh key swapout so maybe the bug there is not using a base job that does that | 22:10 |
| mordred | yeah. maybe so - that's a good point | 22:10 |
| mordred | so maybe trying ssh tunnel is the better thing to try | 22:10 |
| corvus | it's easier to try :) | 22:11 |
| mordred | and we can keep writable-etc-hosts in our back pocket if something about port forwarding turns out to have unintended complexity sads | 22:11 |
| mordred | corvus: ++ | 22:11 |
| mordred | I feel like "unintended complexity sads" should be a band | 22:11 |
| corvus | clarkb: did you intentionally make the name one which can not appear in dns? | 22:11 |
| corvus | (it has an underscore) | 22:12 |
| clarkb | corvus: sort of. I wanted something unlikely to clash elsewhere | 22:12 |
| corvus | that makes it very unlikely to clash :) | 22:13 |
| clarkb | ya probably doesnt have to be that strong of a prevention if we dont like it | 22:13 |
| corvus | clarkb: however, it may not match the docker domain regex :) | 22:13 |
| clarkb | O think they do allow _ | 22:13 |
| clarkb | er I | 22:13 |
| corvus | domainComponentRegexp = match(`(?:[a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9-]*[a-zA-Z0-9])`) | 22:14 |
| corvus | it's that, right? | 22:14 |
| clarkb | ah yup. For some reason I thought separatorRegexp = match(`(?:[._]|__|[-]*)`) was in that domain part before | 22:15 |
| clarkb | but I think that regex is only in the stuff after the domain | 22:15 |
| clarkb | so ya should not use _ | 22:15 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Add SANs to buildset registry cert https://review.openstack.org/653869 | 22:16 |
| corvus | clarkb: ^ that's intended to be squashed into your revert revert | 22:16 |
| mnaser | infra-root: I think ovh-bhs1 is maybe having some serious network problems | 22:18 |
| mnaser | http://logs.openstack.org/92/649892/13/check/openstack-ansible-deploy-aio_proxy-ubuntu-bionic/f1063e4/job-output.txt.gz | 22:18 |
| mnaser | "No viable v4 or v6 route found to git.openstack.org. The build node is assumed to be invalid." | 22:18 |
| mnaser | "git.openstack.org: Temporary failure in name resolution" | 22:18 |
| mnaser | that job failed with retry_limit so I assume it isn't limited to 1 or 2 machines.. | 22:18 |
| clarkb | mnaser: that one should be retried (it is why we have that check in there) | 22:18 |
| clarkb | mnaser: 3 attempts | 22:18 |
| mnaser | retry_limit = it's tried 3 times already though, no? | 22:19 |
| clarkb | yes | 22:19 |
| mnaser | 649892,13 in gate, so I guess it failed 3 times | 22:19 |
| mnaser | either it failed in pre on 3 different clouds .. or maybe it got 3 ovh nodes that broke | 22:19 |
| corvus | i see 2 runs in bhs1: http://logs.openstack.org/92/649892/13/check/openstack-ansible-deploy-aio_proxy-ubuntu-bionic/ | 22:20 |
| mnaser | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22No%20viable%20v4%20or%20v6%20route%20found%20to%20git.openstack.org.%5C%22 | 22:21 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for nova unit test bug 1825435 https://review.openstack.org/653871 | 22:21 |
| mnaser | it looks like its hitting a few times (if you add node_provider) | 22:21 |
| openstack | bug 1825435 in OpenStack Compute (nova) "TestRPC unit tests intermittently fail with "'>' not supported between instances of 'NoneType' and 'datetime.datetime'"" [Undecided,Confirmed] https://launchpad.net/bugs/1825435 | 22:21 |
| clarkb | looking at the ansible facts the ipv4 interface looks correctly configured. | 22:21 |
| clarkb | (yes the mask of 255.255.255.255 is expected and correct) | 22:21 |
| clarkb | I don't think we are configuring the host incorrectly as a result | 22:22 |
| *** diablo_rojo has joined #openstack-infra | 22:23 | |
| clarkb | this connection comes to you via bhs1 | 22:24 |
| mnaser | http://logs.openstack.org/73/651173/1/check/tempest-full/a257167/controller/logs/ | 22:24 |
| mnaser | here's another one that failed | 22:24 |
| mnaser | but this one collected some logs | 22:24 |
| clarkb | and dig git.openstack.org @1.1.1.1 works most of the time but I managed to get it to hang | 22:24 |
| clarkb | which does seem to point to some cloud side networking trouble talking to that dns provider | 22:25 |
| clarkb | yup it timed out eventually neat | 22:25 |
| corvus | perhaps we should go ahead and switch that to opendev.org? | 22:27 |
| clarkb | maybe? I think the problem is in connecting to cloudflare not cloudflare connecting to rax dns | 22:27 |
| clarkb | my traceroute to 1.1.1.1 from bhs1 shows 10/8 addrs O_O | 22:27 |
| mnaser | thats kinda normal if their internal network uses 10.x addresses to route things | 22:28 |
| mnaser | same reason you see bunch of * * * in trace routes usually | 22:28 |
| mnaser | http://logs.openstack.org/73/651173/1/check/tempest-full/a257167/controller/logs/unbound_log.txt.gz | 22:30 |
| mnaser | that might be useful? | 22:30 |
| corvus | i'm going to restart haproxy on the opendev lb to test out v6 before i push up a change | 22:31 |
| mordred | corvus: ++ | 22:31 |
| clarkb | mnaser: reading that log we don't appear to remove the ipv6 addrs from our unbound config which we should (but unbound seems to handle that properly) and there are tcp errors to 1.1.1.1 | 22:34 |
| clarkb | and I see that consistently with udp and tcp from my bhs1 instance digging against 1.1.1.1 too | 22:35 |
| mnaser | is there any reason why we don't make unbound run as an actually full recursive dns server? | 22:35 |
| clarkb | I think we wanted to be better internet citizens and not botnet a bunch of cacheable lookups against root servers? | 22:35 |
| mnaser | that seems like a reasonable thing :) | 22:38 |
| mnaser | I guess if we somehow had subnets of all of our VMs | 22:38 |
| clarkb | considering the other rsync failures I'm wary of blaming dns here | 22:38 |
| mnaser | we could do a regional dns thing | 22:38 |
| mnaser | but yeah | 22:38 |
| clarkb | it is just a reproduceable aspect of the problem | 22:38 |
| mnaser | it seems more network based. | 22:38 |
| *** ijw has quit IRC | 22:38 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Bind to v4 and v6 in haproxy https://review.openstack.org/653874 | 22:39 |
| *** yamamoto has joined #openstack-infra | 22:39 | |
| *** _erlon_ has quit IRC | 22:40 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zone-opendev.org master: Add IPv6 addr to opendev.org https://review.openstack.org/653875 | 22:41 |
| *** yamamoto has quit IRC | 22:41 | |
| corvus | clarkb, fungi, mordred: ^ those are the only 2 changes needed to make opendev.org answer on ipv6 (thanks to the magic mnaser made happen behind the scenes!) the first change is already manually in place; you should be able to connect to the ipv6 addr manually to test if you want | 22:41 |
| clarkb | corvus: I ssh'd in and confirmed the output of ifconfig matches the dns update | 22:42 |
| clarkb | and the other change lgtm so +2 on both | 22:42 |
| corvus | clarkb: while i'm thinking about it -- friendly reminder on ttls :) | 22:43 |
| clarkb | corvus: oh thank you | 22:43 |
| clarkb | I listed the domains I'll be reducing the TTLs for on the etherpad | 22:44 |
| clarkb | please add any if I missed them | 22:44 |
| mnaser | curl is weird with ipv6 so you need to disable globbing (-g), but this works for me: curl -g 'http://[2604:e100:3:0:f816:3eff:fe6b:ad62]/' | 22:44 |
| corvus | clarkb: lgtm | 22:46 |
| *** jcoufal has quit IRC | 22:48 | |
| *** yamamoto has joined #openstack-infra | 22:49 | |
| clarkb | ok half of them were already 5 minute ttls. updated the others | 22:51 |
| mnaser | another 2 jobs just failed collect logs too im pretty sure :X | 22:51 |
| mnaser | they just went back to queued again, but zuul logs might expose more info | 22:51 |
| mnaser | http://zuul.openstack.org/stream/c05feefcc6e34e8fb8ecadfb8a037295?logfile=console.log | 22:52 |
| mnaser | this one is also about to fail | 22:52 |
| mnaser | Failed to connect to the host via ssh: ssh: connect to host 158.69.66.89 port 22: Connection timed ou | 22:52 |
| corvus | mnaser: you want to pause the provider? | 22:52 |
| clarkb | mnaser: from home to bhs1 I get some packet loss in chicago according to mtr | 22:52 |
| openstackgerrit | Merged openstack-infra/zone-opendev.org master: Add IPv6 addr to opendev.org https://review.openstack.org/653875 | 22:52 |
| mnaser | I can push up a patch, by pause I assume max-servers: 0 unless there's an actual new `pause` pattern? | 22:53 |
| mordred | corvus: wow. :::80 is the right syntax? | 22:53 |
| clarkb | mnaser: thats it | 22:53 |
| mnaser | mordred: yes, haproxy is terrible when it comes to that | 22:53 |
| clarkb | mordred: ya listne on :: port 80 is :80 | 22:53 |
| mnaser | it doesn't use [] around IPv6 addresses | 22:53 |
| mordred | yeah - it's just ... wow | 22:53 |
| mnaser | I wonder how their parsing code works :) | 22:53 |
| *** tkajinam has joined #openstack-infra | 22:54 | |
| corvus | better than docker | 22:55 |
| mnaser | clarkb: were you seeing packet loss to both regions? | 22:55 |
| clarkb | mnaser: no I'm just using my little bhs1 VM to check in either direction | 22:56 |
| clarkb | so only bhs1 | 22:56 |
| mnaser | http://grafana.openstack.org/d/BhcSH5Iiz/nodepool-ovh?orgId=1 | 22:56 |
| mnaser | hmm | 22:56 |
| mnaser | those big dips.. are those just Grafana things or? | 22:56 |
| clarkb | that could be network crashing all the jobs then rebuilding after | 22:57 |
| mnaser | a ton of error node launch attempts too | 22:57 |
| mnaser | for gra1 | 22:57 |
| clarkb | I think those are "normal" because their quota syncage isn't fast | 22:58 |
| mnaser | ah ok | 22:58 |
| clarkb | so we'll be under quota and it thinks we are over until nova catches up and its a perpetual race | 22:58 |
| corvus | i added "Move zuul keys (on zuul01.openstack.org):" to the etherpad | 22:59 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Update gerrit connection in Zuul https://review.openstack.org/653877 | 23:00 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: nodepool: pause ovh https://review.openstack.org/653878 | 23:00 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Revert "nodepool: pause ovh" https://review.openstack.org/653879 | 23:00 |
| corvus | i also added 653877 to the etherpad | 23:00 |
| *** aaronsheffield has quit IRC | 23:01 | |
| mnaser | pushed those two up, im gonna add comments to the pause patch so we can point the ovh team to it | 23:02 |
| * mnaser isn't sure who to contact and will defer to whoever knows | 23:02 | |
| clarkb | mnaser: amorin has been our contact recently | 23:02 |
| mnaser | cool, ive added him to the review to perhaps ping him by email too | 23:03 |
| *** whoami-rajat has quit IRC | 23:03 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for nova unit test bug 1825435 https://review.openstack.org/653871 | 23:04 |
| openstack | bug 1825435 in OpenStack Compute (nova) "TestRPC unit tests intermittently fail with "'>' not supported between instances of 'NoneType' and 'datetime.datetime'"" [Undecided,Confirmed] https://launchpad.net/bugs/1825435 | 23:04 |
| *** yamamoto has quit IRC | 23:05 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Create a script to handle zuul key file moves https://review.openstack.org/653881 | 23:07 |
| mordred | corvus: ^^ oh - nice | 23:07 |
| corvus | fungi: ^ we should really put that in the rename playbook, but it's close to time and i don't want to futz with it; mind if we tag that onto your script? maybe you can just squash that in when you're back | 23:07 |
| mordred | corvus: and yes, we should really put that in the rename playbook | 23:08 |
| corvus | yeah, i don't think it'd be hard | 23:08 |
| mordred | and yes, putting it in to that now seems like the wrong time | 23:08 |
| corvus | if it were this morning, i'd just do that :) | 23:08 |
| corvus | but it's the end of the day before | 23:08 |
| mordred | yup | 23:08 |
| *** yamamoto has joined #openstack-infra | 23:10 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/nodepool master: Fix AWS provider ignoring region in providers https://review.openstack.org/649423 | 23:17 |
| clarkb | I'm going to pop out for a bit but back before calling it a day to check if there is anything else to review (hopefully updates from fungi) | 23:18 |
| clarkb | DNS should be ready to go | 23:19 |
| *** eernst has joined #openstack-infra | 23:19 | |
| *** igordc has joined #openstack-infra | 23:20 | |
| fungi | yeah, still incoming. broadband outage definitely has set me back, sorry | 23:23 |
| fungi | getting close now | 23:23 |
| *** yamamoto has quit IRC | 23:25 | |
| openstackgerrit | Merged openstack-infra/project-config master: nodepool: pause ovh https://review.openstack.org/653878 | 23:28 |
| *** lseki has quit IRC | 23:45 | |
| *** sthussey has quit IRC | 23:47 | |
| *** mattw4 has joined #openstack-infra | 23:49 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!