| tobias-urdin | clarkb: all good, it was uploaded | 00:00 |
|---|---|---|
| *** whoami-rajat has quit IRC | 00:00 | |
| clarkb | tobias-urdin: ok enqueing the others | 00:02 |
| *** hwoarang has quit IRC | 00:03 | |
| *** smarcet has joined #openstack-infra | 00:04 | |
| clarkb | tobias-urdin: do manila and mistral have the same sha? | 00:04 |
| clarkb | mitral should be 072f02e66005ee30c8e820d0244c3460f7d78ec7 | 00:05 |
| tobias-urdin | yeah, was a human copy paste error right there | 00:05 |
| clarkb | ok continuing with the value above then | 00:06 |
| *** hwoarang has joined #openstack-infra | 00:06 | |
| tobias-urdin | thanks, my bad on that one | 00:06 |
| *** markvoelker has joined #openstack-infra | 00:09 | |
| fungi | conference call with students concluded | 00:09 |
| fungi | tobias-urdin: you may be interested in https://review.openstack.org/637076 | 00:10 |
| clarkb | tobias-urdin: no problem. All of them should be enqueued. I ended up doing them one by one to quickyl sanity check them all after I caught that sha1 dup | 00:10 |
| fungi | (trying to keep things moving along for them so they don't get distracted) | 00:10 |
| clarkb | fungi: corvus the github event queue length remains at 0 | 00:11 |
| clarkb | not sure how long it normally takes to grow | 00:11 |
| *** ssusteve has quit IRC | 00:11 | |
| corvus | okay, the idea of templating stuff into k8s yaml via ansible and jinja has hit a snag -- i've got a multiline string, and from_yaml seems to be putting it all on one line | 00:12 |
| fungi | clarkb: seems per anecdotes to pick up around 08:00-09:00 utc | 00:12 |
| fungi | i guess ansible has a lot of emea contributors | 00:12 |
| tobias-urdin | fungi: yeah got a message about that earlier iirc, a teacher whos students were going to do a reimplementation of some puppet module and wondered who could take ownership of it afterwards | 00:13 |
| tobias-urdin | not sure what to do with that tbh | 00:13 |
| fungi | tobias-urdin: hogepodge is leading that effort | 00:13 |
| fungi | diablo_rojo and i are helping guide them as well | 00:14 |
| corvus | these are the files at issue: http://git.openstack.org/cgit/openstack-infra/system-config/tree/kubernetes/gitea/gitea-playbook.yaml#n12 and http://git.openstack.org/cgit/openstack-infra/system-config/tree/kubernetes/gitea/k8s/secret.yaml | 00:14 |
| fungi | tobias-urdin: i mostly point it out because there will be reviews at some stage | 00:14 |
| tobias-urdin | okok, reading commit message i understand it as working around licenses, i know we are using another non-apache licensed module as well | 00:14 |
| fungi | yep | 00:15 |
| fungi | this is providing an apache alternative to a gpl puppet module since puppet-openstack needs to have non-copyleft dependencies | 00:15 |
| tobias-urdin | cool, nice effort then, was wondering why one otherwise would reimplement it | 00:16 |
| fungi | i know there are more but it was one that ndsu could throw some capstone interns at | 00:16 |
| fungi | and it's icy up in north dakota this time of year, so great opportunity for them to stay indoors and become openstack contributors ;) | 00:17 |
| tobias-urdin | hehe | 00:17 |
| tobias-urdin | off topic; out of curiosity is all the gitea stuff about getting zuul support or is there some greater plan that i have missed? | 00:17 |
| fungi | tobias-urdin: it's the opendev.org replacement for git.openstack.org and friends | 00:18 |
| corvus | tobias-urdin: http://specs.openstack.org/openstack-infra/infra-specs/specs/opendev-gerrit.html new code browsing system | 00:18 |
| clarkb | corvus: that must be the from_yaml filter doing that? | 00:18 |
| corvus | clarkb: yeah, i can't quite figure out what's what :/ | 00:18 |
| corvus | i've reproduced in a test playbook | 00:18 |
| clarkb | corvus: can we change the secrets.yaml to do key : |\n "{{ var_name }}" | 00:19 |
| clarkb | maybe it will get the hint if we are bit more explicit in the quoting? | 00:19 |
| corvus | that doesn't seem to work | 00:19 |
| hogepodge | tobias-urdin: hi :-) | 00:21 |
| tobias-urdin | ack, that is really a very github-like interface, almost a replica. then i know we won't move to pull requests :D | 00:21 |
| tobias-urdin | hogepodge: o/ | 00:21 |
| *** jamesmcarthur has quit IRC | 00:22 | |
| corvus | clarkb: oh, hrm, maybe i can just drop "from_yaml" | 00:22 |
| corvus | clarkb: https://docs.ansible.com/ansible/latest/modules/k8s_module.html see 'resource_definition | 00:22 |
| hogepodge | tobias-urdin: yeah, that was me who contacted you earlier (former puppet-openstack ptl many years ago) | 00:22 |
| clarkb | corvus: oh interesting | 00:22 |
| corvus | clarkb: says it takes string, list, or dict | 00:22 |
| clarkb | corvus: ya that should do it then | 00:22 |
| tobias-urdin | hogepodge: cool, my bad i closed that chat tab and forgot about it until it was mentioned here :( | 00:23 |
| hogepodge | I'm going to have to get them started on the project, setting up the git type and provider, and also testing, then patches to puppet-* to replace puppet-vcsrepo calls. The most important thing to me would be just a general sense of support, reviews, and patience. My hope is puppet-openstack team is the best owner since it would be the biggest consumer (and once it's done should be easy to maintain) | 00:24 |
| hogepodge | this has to fit into a semester, and now that they have their initial patch up I'm going to start expecting code patches this week | 00:24 |
| fungi | tobias-urdin: at some point that and the corresponding governance patch (which doesn't exist yet) will need a ptl to +1 if the puppet openstack team agrees to own the resulting module, but safe to let the project-config reviewers weigh in and help them get it into shape first | 00:25 |
| *** jamesmcarthur has joined #openstack-infra | 00:26 | |
| fungi | also making sure the resulting module meets the needs of the puppet openstack modules which are currently relying on the vcsrepo module is, like, the #1 priority for it so feedback from the team will be necessary there as it really gets underway | 00:27 |
| corvus | clarkb: hrm. removing from_yaml makes it look right when passed to the "debug" module, but it looks like k8s is somehow still turning the newlines into whitespace | 00:29 |
| tobias-urdin | hogepodge: fungi ack, sound good. shouldn't be much maintenance one it's in a more polished state | 00:30 |
| clarkb | corvus: is it possible the definition alias for resource_definition does additional processing? | 00:30 |
| tobias-urdin | not sure but dont think vcsrepo is used to a great extend, i can only think about 2-3 placed in openstack related modules where it's used, maybe in some testing as well | 00:31 |
| clarkb | unlikely but may be worth double checking | 00:31 |
| corvus | clarkb: hrm. reading the source now | 00:31 |
| *** jamesmcarthur has quit IRC | 00:31 | |
| tobias-urdin | we can get back to governance about that, i dont think anybody would mind maintaining it but i guess we would vote about that, i need to grab some sleep now | 00:32 |
| tobias-urdin | let me know if i can help with anything | 00:33 |
| * tobias-urdin goes to bed | 00:33 | |
| clarkb | I guess the other option we have is to just stick the full definition in the playbook and not try to deserialize it in | 00:33 |
| corvus | yeah... why don't i try that to see if it works | 00:33 |
| corvus | the output was yellow... | 00:34 |
| corvus | hey that looks better! | 00:35 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Fix gitea k8s files https://review.openstack.org/637080 | 00:39 |
| corvus | clarkb: thanks! ^ | 00:39 |
| *** gyee has quit IRC | 00:41 | |
| *** markvoelker has quit IRC | 00:42 | |
| corvus | i'm going to go ahead and approve that since it's just corrective, and it's needed to normalize the git repo with "production" | 00:43 |
| corvus | i think tomorrow we should be able to verify everything is stable and land https://review.openstack.org/635486 | 00:45 |
| clarkb | yay | 00:45 |
| clarkb | I totally got distracted by zuul and nodepool and gitea today and didn't end up pbxing but I should be able to boot a new instance tomorrow | 00:47 |
| clarkb | Also I reviewed cmurphy's next set of puppet-4 changes. I think the next one should be mostly safe as it only applies to -dev servers | 00:47 |
| *** jamesmcarthur has joined #openstack-infra | 00:47 | |
| hogepodge | thanks tobias-urdin | 00:49 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 00:52 |
| *** jamesmcarthur has quit IRC | 00:58 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix undefined attrs in registry push/pull roles https://review.openstack.org/637072 | 01:08 |
| *** wolverineav has quit IRC | 01:08 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 01:09 |
| *** wolverineav has joined #openstack-infra | 01:10 | |
| *** sthussey has quit IRC | 01:14 | |
| *** jamesmcarthur has joined #openstack-infra | 01:15 | |
| *** wolverineav has quit IRC | 01:15 | |
| *** jamesmcarthur has quit IRC | 01:20 | |
| *** smarcet has joined #openstack-infra | 01:32 | |
| *** wolverineav has joined #openstack-infra | 01:32 | |
| *** jamesmcarthur has joined #openstack-infra | 01:34 | |
| *** zul has quit IRC | 01:37 | |
| *** markvoelker has joined #openstack-infra | 01:39 | |
| *** hwoarang has quit IRC | 01:43 | |
| *** hwoarang has joined #openstack-infra | 01:44 | |
| *** jamesmcarthur has quit IRC | 01:46 | |
| *** wolverineav has quit IRC | 01:48 | |
| *** wolverineav has joined #openstack-infra | 01:50 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/puppet-openstackid master: Fix on mysql ssl certs https://review.openstack.org/637084 | 01:51 |
| *** hwoarang has quit IRC | 01:54 | |
| *** wolverineav has quit IRC | 01:55 | |
| *** hwoarang has joined #openstack-infra | 01:55 | |
| *** hongbin has joined #openstack-infra | 01:56 | |
| openstackgerrit | Merged openstack-infra/system-config master: Fix gitea k8s files https://review.openstack.org/637080 | 01:57 |
| *** wolverineav has joined #openstack-infra | 02:02 | |
| *** jamesmcarthur has joined #openstack-infra | 02:06 | |
| *** jamesmcarthur has quit IRC | 02:11 | |
| *** mriedem has quit IRC | 02:11 | |
| *** markvoelker has quit IRC | 02:12 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/puppet-openstackid master: Fix on mysql ssl certs https://review.openstack.org/637084 | 02:26 |
| *** jamesmcarthur has joined #openstack-infra | 02:27 | |
| *** wolverineav has quit IRC | 02:32 | |
| *** jamesmcarthur has quit IRC | 02:32 | |
| *** wolverineav has joined #openstack-infra | 02:33 | |
| *** wolverineav has quit IRC | 02:37 | |
| *** jamesmcarthur has joined #openstack-infra | 02:48 | |
| *** hongbin has quit IRC | 02:48 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 02:52 |
| *** jamesmcarthur has quit IRC | 02:53 | |
| *** psachin has joined #openstack-infra | 02:56 | |
| *** jamesmcarthur has joined #openstack-infra | 03:00 | |
| *** smarcet has quit IRC | 03:02 | |
| *** smarcet has joined #openstack-infra | 03:02 | |
| *** smarcet has quit IRC | 03:04 | |
| *** markvoelker has joined #openstack-infra | 03:09 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 03:11 |
| *** apetrich has quit IRC | 03:15 | |
| *** ramishra has joined #openstack-infra | 03:16 | |
| *** jamesmcarthur has quit IRC | 03:18 | |
| *** jamesmcarthur has joined #openstack-infra | 03:19 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] testing... https://review.openstack.org/636759 | 03:35 |
| *** markvoelker has quit IRC | 03:43 | |
| *** diablo_rojo has quit IRC | 03:48 | |
| *** janki has joined #openstack-infra | 03:58 | |
| *** yamamoto has joined #openstack-infra | 04:01 | |
| *** dave-mccowan has quit IRC | 04:03 | |
| *** ykarel has joined #openstack-infra | 04:04 | |
| *** wolverineav has joined #openstack-infra | 04:36 | |
| *** markvoelker has joined #openstack-infra | 04:40 | |
| openstackgerrit | Rico Lin proposed openstack-infra/project-config master: Add an openstack/auto-scaling-sig repository https://review.openstack.org/637125 | 04:46 |
| *** jamesmcarthur has quit IRC | 04:49 | |
| *** jamesmcarthur has joined #openstack-infra | 04:50 | |
| *** jamesmcarthur has quit IRC | 04:54 | |
| *** ramishra has quit IRC | 04:57 | |
| *** ramishra has joined #openstack-infra | 05:02 | |
| *** wolverineav has quit IRC | 05:04 | |
| *** wolverineav has joined #openstack-infra | 05:05 | |
| *** jamesmcarthur has joined #openstack-infra | 05:10 | |
| *** markvoelker has quit IRC | 05:12 | |
| *** jamesmcarthur has quit IRC | 05:15 | |
| *** pabelanger has quit IRC | 05:15 | |
| *** otherwiseguy has quit IRC | 05:15 | |
| *** otherwiseguy has joined #openstack-infra | 05:15 | |
| *** janki has quit IRC | 05:18 | |
| *** janki has joined #openstack-infra | 05:18 | |
| *** hwoarang has quit IRC | 05:21 | |
| *** hwoarang has joined #openstack-infra | 05:26 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] letsencrypt prototype implementation https://review.openstack.org/636759 | 05:28 |
| *** ykarel has quit IRC | 05:30 | |
| *** jamesmcarthur has joined #openstack-infra | 05:31 | |
| openstackgerrit | Rico Lin proposed openstack-infra/project-config master: Add an openstack/auto-scaling-sig repository https://review.openstack.org/637125 | 05:34 |
| *** ricolin has joined #openstack-infra | 05:35 | |
| *** jamesmcarthur has quit IRC | 05:35 | |
| *** psachin has quit IRC | 05:38 | |
| *** wolverineav has quit IRC | 05:40 | |
| *** lathiat_ has joined #openstack-infra | 05:42 | |
| *** lathiat has quit IRC | 05:42 | |
| *** ykarel has joined #openstack-infra | 05:44 | |
| *** jamesmcarthur has joined #openstack-infra | 05:51 | |
| openstackgerrit | Rico Lin proposed openstack-infra/project-config master: Add an openstack/auto-scaling-sig repository https://review.openstack.org/637125 | 05:52 |
| *** wolverineav has joined #openstack-infra | 05:53 | |
| *** wolverineav has quit IRC | 05:54 | |
| *** jamesmcarthur has quit IRC | 05:56 | |
| openstackgerrit | Rico Lin proposed openstack-infra/project-config master: Add an openstack/auto-scaling-sig repository https://review.openstack.org/637125 | 05:57 |
| *** auristor has quit IRC | 06:01 | |
| *** ricolin has quit IRC | 06:03 | |
| *** rlandy|bbl has quit IRC | 06:04 | |
| *** eernst has joined #openstack-infra | 06:08 | |
| *** whoami-rajat has joined #openstack-infra | 06:08 | |
| *** markvoelker has joined #openstack-infra | 06:09 | |
| *** eandersson has quit IRC | 06:12 | |
| *** jamesmcarthur has joined #openstack-infra | 06:12 | |
| *** eandersson has joined #openstack-infra | 06:13 | |
| *** eandersson has quit IRC | 06:13 | |
| *** eandersson has joined #openstack-infra | 06:14 | |
| *** jamesmcarthur has quit IRC | 06:16 | |
| *** e0ne has joined #openstack-infra | 06:18 | |
| *** kjackal has joined #openstack-infra | 06:23 | |
| *** lujinluo has quit IRC | 06:27 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/infra-specs master: letsencrypt spec https://review.openstack.org/587283 | 06:28 |
| *** tkajinam_ has joined #openstack-infra | 06:33 | |
| *** jamesmcarthur has joined #openstack-infra | 06:33 | |
| ianw | clarkb / fungi / corvus / jhesketh : ^ is ready for review. can be cross-referenced with the prototype @ https://review.openstack.org/636759 | 06:35 |
| *** tkajinam has quit IRC | 06:35 | |
| jhesketh | ianw: cool, will take a look shortly | 06:36 |
| *** jamesmcarthur has quit IRC | 06:38 | |
| *** dpawlik has joined #openstack-infra | 06:40 | |
| *** markvoelker has quit IRC | 06:43 | |
| *** e0ne has quit IRC | 06:43 | |
| *** e0ne has joined #openstack-infra | 06:46 | |
| *** ramishra has quit IRC | 06:47 | |
| *** ramishra has joined #openstack-infra | 06:47 | |
| *** e0ne has quit IRC | 06:48 | |
| *** psachin has joined #openstack-infra | 06:49 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Re-use the github PR object when fetching reviews https://review.openstack.org/636705 | 06:49 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add comment about extra issues request https://review.openstack.org/636706 | 06:50 |
| *** wolverineav has joined #openstack-infra | 06:53 | |
| *** jamesmcarthur has joined #openstack-infra | 06:54 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Add dockerized test setup https://review.openstack.org/636424 | 06:54 |
| *** quiquell|off is now known as quiquell|rover | 06:54 | |
| *** wolverineav has quit IRC | 06:58 | |
| *** jamesmcarthur has quit IRC | 06:59 | |
| *** jtomasek has joined #openstack-infra | 07:00 | |
| *** e0ne has joined #openstack-infra | 07:05 | |
| *** e0ne has quit IRC | 07:07 | |
| *** pgaxatte has joined #openstack-infra | 07:09 | |
| *** eernst has quit IRC | 07:12 | |
| *** tkajinam_ has quit IRC | 07:14 | |
| *** kopecmartin|off is now known as kopecmartin | 07:15 | |
| *** jamesmcarthur has joined #openstack-infra | 07:15 | |
| *** ramishra has quit IRC | 07:16 | |
| *** jamesmcarthur has quit IRC | 07:20 | |
| *** gfidente has joined #openstack-infra | 07:23 | |
| *** kjackal has quit IRC | 07:23 | |
| *** ramishra has joined #openstack-infra | 07:25 | |
| *** ricolin has joined #openstack-infra | 07:27 | |
| *** tkajinam has joined #openstack-infra | 07:28 | |
| *** chkumar|pto is now known as chkumar|ruck | 07:29 | |
| *** tkajinam_ has joined #openstack-infra | 07:29 | |
| *** apetrich has joined #openstack-infra | 07:30 | |
| *** tkajinam has quit IRC | 07:32 | |
| *** aojea has joined #openstack-infra | 07:34 | |
| *** quiquell|rover is now known as quique|rover|brb | 07:36 | |
| *** jamesmcarthur has joined #openstack-infra | 07:37 | |
| *** markvoelker has joined #openstack-infra | 07:40 | |
| *** jamesmcarthur has quit IRC | 07:42 | |
| *** cgoncalves has quit IRC | 07:48 | |
| *** cgoncalves has joined #openstack-infra | 07:49 | |
| *** jamesmcarthur has joined #openstack-infra | 07:58 | |
| *** wolverineav has joined #openstack-infra | 07:58 | |
| *** ykarel is now known as ykarel|lunch | 07:59 | |
| *** memoussati has joined #openstack-infra | 07:59 | |
| *** jamesmcarthur has quit IRC | 08:03 | |
| *** rpittau has joined #openstack-infra | 08:04 | |
| *** kjackal has joined #openstack-infra | 08:04 | |
| *** manjeets has quit IRC | 08:09 | |
| *** manjeets has joined #openstack-infra | 08:11 | |
| *** markvoelker has quit IRC | 08:12 | |
| *** quique|rover|brb is now known as quiquell|rover | 08:17 | |
| *** jamesmcarthur has joined #openstack-infra | 08:19 | |
| *** tkajinam_ has quit IRC | 08:20 | |
| *** jamesmcarthur has quit IRC | 08:24 | |
| *** yikun has quit IRC | 08:25 | |
| *** memoussati has quit IRC | 08:25 | |
| *** memoussati has joined #openstack-infra | 08:29 | |
| *** iurygregory has joined #openstack-infra | 08:36 | |
| zbr | infra-root: i need bit of review/merge help with few maintenance fixes on bindep https://review.openstack.org/#/c/605613/ | 08:38 |
| *** tosky has joined #openstack-infra | 08:38 | |
| *** jamesmcarthur has joined #openstack-infra | 08:41 | |
| *** jamesmcarthur has quit IRC | 08:45 | |
| *** ykarel|lunch is now known as ykarel | 08:48 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: gerrit: add support for report only connection https://review.openstack.org/568216 | 08:48 |
| *** rascasoft has joined #openstack-infra | 08:50 | |
| *** jpena|off is now known as jpena | 08:57 | |
| *** jamesmcarthur has joined #openstack-infra | 09:02 | |
| *** yamamoto has quit IRC | 09:04 | |
| *** yamamoto has joined #openstack-infra | 09:05 | |
| *** xek has joined #openstack-infra | 09:05 | |
| *** jamesmcarthur has quit IRC | 09:07 | |
| *** markvoelker has joined #openstack-infra | 09:10 | |
| *** ccamacho has joined #openstack-infra | 09:10 | |
| *** shardy has joined #openstack-infra | 09:11 | |
| *** yamamoto has quit IRC | 09:18 | |
| *** raissa has joined #openstack-infra | 09:18 | |
| *** priteau has joined #openstack-infra | 09:20 | |
| *** memoussati has quit IRC | 09:21 | |
| *** jamesmcarthur has joined #openstack-infra | 09:23 | |
| *** derekh has joined #openstack-infra | 09:28 | |
| *** jamesmcarthur has quit IRC | 09:28 | |
| *** yamamoto has joined #openstack-infra | 09:37 | |
| *** luizbag has joined #openstack-infra | 09:39 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Manage ansible installations https://review.openstack.org/631930 | 09:40 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Symlink ansible plugins https://review.openstack.org/636022 | 09:40 |
| *** markvoelker has quit IRC | 09:42 | |
| *** jamesmcarthur has joined #openstack-infra | 09:45 | |
| *** raissa has quit IRC | 09:46 | |
| *** raissa has joined #openstack-infra | 09:46 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Manage ansible installations https://review.openstack.org/631930 | 09:46 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Symlink ansible plugins https://review.openstack.org/636022 | 09:46 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Support boolean values in get_default https://review.openstack.org/637158 | 09:46 |
| *** raissa has quit IRC | 09:46 | |
| *** ccamacho has quit IRC | 09:48 | |
| *** raissa has joined #openstack-infra | 09:48 | |
| *** raissa has quit IRC | 09:48 | |
| *** ccamacho has joined #openstack-infra | 09:48 | |
| *** jamesmcarthur has quit IRC | 09:49 | |
| *** memoussati has joined #openstack-infra | 09:54 | |
| *** ricolin has quit IRC | 09:57 | |
| *** jamesmcarthur has joined #openstack-infra | 10:06 | |
| openstackgerrit | Quique Llorente proposed openstack/diskimage-builder master: Keep git after ironic-agent post https://review.openstack.org/637162 | 10:08 |
| *** dtantsur|afk is now known as dtantsur | 10:09 | |
| openstackgerrit | Quique Llorente proposed openstack/diskimage-builder master: Keep git after ironic-agent post https://review.openstack.org/637162 | 10:11 |
| *** jamesmcarthur has quit IRC | 10:11 | |
| frickler | tobias-urdin: fungi: I'm wondering whether the trailing \n that seems to be contained in the secret may cause an issue with the forge password | 10:20 |
| *** ociuhandu has joined #openstack-infra | 10:24 | |
| *** yamamoto has quit IRC | 10:26 | |
| frickler | tobias-urdin: I repeated your test from yesterday and when I add the \n to the password, I get a 400 | 10:26 |
| frickler | tobias-urdin: so either you need to rstrip() it or we need to create the secret without it | 10:27 |
| *** jamesmcarthur has joined #openstack-infra | 10:28 | |
| *** jamesmcarthur has quit IRC | 10:32 | |
| *** kjackal has quit IRC | 10:32 | |
| *** priteau has quit IRC | 10:33 | |
| odyssey4me | Hi folks - are there any reviewers around for stackalytics? I've got a patch up to add another email address for me: https://review.openstack.org/636178 | 10:38 |
| *** markvoelker has joined #openstack-infra | 10:39 | |
| cmurphy | odyssey4me: i don't think the stackalytics site uses that source code any more, I tried to fix my email address not long ago but I'm still duplicated on the site | 10:39 |
| *** memoussati has quit IRC | 10:44 | |
| AJaeger | frickler: https://review.openstack.org/637067 fixed this, so all should be good - isn't it? | 10:45 |
| AJaeger | odyssey4me: the infra team is not reviewing stackalytics, https://review.openstack.org/#/admin/groups/183,members gives the list of other reviewers | 10:46 |
| AJaeger | infra-root, didn't you want to remove yourself from that acl? ^ | 10:46 |
| AJaeger | odyssey4me: Ilja just merged a single change - might want to ask him? See https://review.openstack.org/636866 | 10:48 |
| *** jamesmcarthur has joined #openstack-infra | 10:49 | |
| odyssey4me | AJaeger it'd be helpful if any of the cores were on IRC... anyway, I've added them all to the review in the hopes that someone picks it up | 10:50 |
| AJaeger | odyssey4me: all - including infra-team which is not reviewing? ;) | 10:51 |
| *** ramishra has quit IRC | 10:52 | |
| *** jamesmcarthur has quit IRC | 10:54 | |
| *** yamamoto has joined #openstack-infra | 11:01 | |
| *** kjackal has joined #openstack-infra | 11:02 | |
| *** smarcet has joined #openstack-infra | 11:07 | |
| *** ramishra has joined #openstack-infra | 11:08 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/puppet-openstackid master: Fix on mysql ssl certs https://review.openstack.org/637084 | 11:09 |
| *** smarcet has quit IRC | 11:10 | |
| *** jamesmcarthur has joined #openstack-infra | 11:10 | |
| mgoddard | hi, looks like the readthedocs webhook jobs are all failing: http://zuul.openstack.org/builds?job_name=trigger-readthedocs-webhook | 11:11 |
| *** yamamoto has quit IRC | 11:11 | |
| *** wolverineav has quit IRC | 11:11 | |
| AJaeger | mgoddard, yeah, they broke it ;( Let me find a link for you... | 11:13 |
| *** markvoelker has quit IRC | 11:13 | |
| frickler | AJaeger: ah, I hadn't seen that patch yet, but good to see that they came to the same conclusion as myself ;) | 11:13 |
| *** memoussati has joined #openstack-infra | 11:13 | |
| *** jamesmcarthur has quit IRC | 11:15 | |
| AJaeger | mgoddard: see http://lists.openstack.org/pipermail/openstack-infra/2018-December/006247.html | 11:18 |
| mgoddard | Thanks AJaeger, I'll update manually for now in the hope they fix that issue | 11:21 |
| AJaeger | mgoddard: AFAIK they run regularly, so should update your pages eventually... | 11:21 |
| *** zul has joined #openstack-infra | 11:29 | |
| *** yamamoto has joined #openstack-infra | 11:30 | |
| *** jamesmcarthur has joined #openstack-infra | 11:31 | |
| *** rfolco|off is now known as rfolco | 11:33 | |
| *** yamamoto has quit IRC | 11:35 | |
| *** jamesmcarthur has quit IRC | 11:36 | |
| *** yamamoto has joined #openstack-infra | 11:50 | |
| *** jamesmcarthur has joined #openstack-infra | 11:53 | |
| *** roman_g has joined #openstack-infra | 11:53 | |
| *** memoussati has quit IRC | 11:54 | |
| *** yamamoto has quit IRC | 11:56 | |
| *** jamesmcarthur has quit IRC | 11:57 | |
| *** electrofelix has joined #openstack-infra | 11:57 | |
| *** markvoelker has joined #openstack-infra | 12:10 | |
| *** rh-jelabarre has joined #openstack-infra | 12:13 | |
| *** jamesmcarthur has joined #openstack-infra | 12:14 | |
| *** janki has quit IRC | 12:16 | |
| *** jamesmcarthur has quit IRC | 12:19 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Manage ansible installations https://review.openstack.org/631930 | 12:19 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Symlink ansible plugins https://review.openstack.org/636022 | 12:19 |
| *** tosky has quit IRC | 12:30 | |
| *** jpena is now known as jpena|lunch | 12:32 | |
| *** lucasagomes is now known as lucas-hungry | 12:33 | |
| *** lucas-hungry is now known as lucasagomes | 12:33 | |
| *** yamamoto has joined #openstack-infra | 12:33 | |
| *** yamamoto has quit IRC | 12:35 | |
| *** yamamoto has joined #openstack-infra | 12:35 | |
| *** jamesmcarthur has joined #openstack-infra | 12:35 | |
| *** tosky has joined #openstack-infra | 12:36 | |
| *** yamamoto has quit IRC | 12:40 | |
| *** lpetrut has joined #openstack-infra | 12:40 | |
| *** smarcet has joined #openstack-infra | 12:40 | |
| *** jamesmcarthur has quit IRC | 12:40 | |
| *** yamamoto has joined #openstack-infra | 12:40 | |
| *** markvoelker has quit IRC | 12:42 | |
| *** EmilienM is now known as EvilienM | 12:43 | |
| tobias-urdin | frickler: we solved it yesterday, it worked after secret was updated | 12:43 |
| *** e0ne has joined #openstack-infra | 12:47 | |
| *** yamamoto has quit IRC | 12:49 | |
| *** zhangfei has joined #openstack-infra | 12:52 | |
| *** memoussati has joined #openstack-infra | 12:52 | |
| *** ccamacho has quit IRC | 12:55 | |
| *** ccamacho has joined #openstack-infra | 12:55 | |
| *** smarcet has quit IRC | 12:56 | |
| *** jamesmcarthur has joined #openstack-infra | 12:57 | |
| *** whoami-rajat has quit IRC | 12:57 | |
| frickler | tobias-urdin: yes, I only read half of the backlog and then started to do my own testing, sorry for the noise ;) | 12:59 |
| *** jamesmcarthur has quit IRC | 13:01 | |
| *** trown|outtypewww is now known as trown | 13:07 | |
| *** yamamoto has joined #openstack-infra | 13:08 | |
| *** ramishra has quit IRC | 13:12 | |
| *** yamamoto has quit IRC | 13:13 | |
| *** electrofelix has quit IRC | 13:14 | |
| *** rh-jelabarre has quit IRC | 13:17 | |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: List changed files for all commits between refs https://review.openstack.org/631797 | 13:17 |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Retrieve full list of jobs with details per tenant via API https://review.openstack.org/635714 | 13:17 |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Add new merger job to get role definitions from a repository https://review.openstack.org/637181 | 13:17 |
| *** jamesmcarthur has joined #openstack-infra | 13:18 | |
| *** dave-mccowan has joined #openstack-infra | 13:18 | |
| zbr | what do i need to do to make git-review work from inside zuul jobs? I got a "Description: Host key verification failed" which makes me believe that either the host does not have the key or the user being used should not be zuul. --- that's for cherry picking only, not push needed. | 13:23 |
| *** jamesmcarthur has quit IRC | 13:24 | |
| *** tosky has quit IRC | 13:25 | |
| *** tosky has joined #openstack-infra | 13:26 | |
| *** ekultails has joined #openstack-infra | 13:28 | |
| *** ekultails has quit IRC | 13:28 | |
| *** e0ne has quit IRC | 13:28 | |
| *** ekultails has joined #openstack-infra | 13:28 | |
| *** e0ne has joined #openstack-infra | 13:30 | |
| *** yamamoto has joined #openstack-infra | 13:33 | |
| *** Sadarex has joined #openstack-infra | 13:33 | |
| *** mriedem has joined #openstack-infra | 13:34 | |
| *** Sadarex has left #openstack-infra | 13:34 | |
| *** smarcet has joined #openstack-infra | 13:34 | |
| *** psachin has quit IRC | 13:36 | |
| *** efried is now known as fried_rice | 13:37 | |
| *** yamamoto has quit IRC | 13:37 | |
| *** yamamoto has joined #openstack-infra | 13:39 | |
| *** yamamoto has quit IRC | 13:39 | |
| *** markvoelker has joined #openstack-infra | 13:39 | |
| *** jamesmcarthur has joined #openstack-infra | 13:40 | |
| *** whoami-rajat has joined #openstack-infra | 13:41 | |
| *** jpena|lunch is now known as jpena | 13:41 | |
| *** rlandy has joined #openstack-infra | 13:42 | |
| *** yamamoto has joined #openstack-infra | 13:43 | |
| *** yamamoto has quit IRC | 13:44 | |
| *** jamesmcarthur has quit IRC | 13:44 | |
| *** yamamoto has joined #openstack-infra | 13:44 | |
| *** jamesmcarthur has joined #openstack-infra | 13:45 | |
| *** dpawlik has quit IRC | 13:49 | |
| *** jamesmcarthur has quit IRC | 13:50 | |
| *** dpawlik has joined #openstack-infra | 13:50 | |
| *** agopi_ is now known as agopi | 13:58 | |
| *** mandre is now known as mandre_away | 14:02 | |
| *** jamesmcarthur has joined #openstack-infra | 14:06 | |
| *** dpawlik has quit IRC | 14:06 | |
| *** ekultails has quit IRC | 14:08 | |
| *** dpawlik has joined #openstack-infra | 14:08 | |
| *** eharney has joined #openstack-infra | 14:10 | |
| *** jamesmcarthur has quit IRC | 14:10 | |
| *** markvoelker has quit IRC | 14:12 | |
| *** gfidente has quit IRC | 14:22 | |
| *** ekultails has joined #openstack-infra | 14:24 | |
| *** gfidente has joined #openstack-infra | 14:26 | |
| *** memoussati has quit IRC | 14:26 | |
| *** pabelanger has joined #openstack-infra | 14:26 | |
| *** jamesmcarthur has joined #openstack-infra | 14:27 | |
| *** roman_g has quit IRC | 14:27 | |
| *** ykarel is now known as ykarel|away | 14:29 | |
| *** jamesmcarthur has quit IRC | 14:31 | |
| *** rh-jelabarre has joined #openstack-infra | 14:32 | |
| fungi | AJaeger: i mentioned it to clarkb and some other members of the infra-core group in gerrit and it seemed like there was probably agreement but i was going to leave it to clarkb to make the final call, as ptl | 14:32 |
| fungi | frickler: also not sure if you saw in the scrollback, but i provided an openssl command line pipe you can use to decrypt a secret locally on the scheduler, for future reference | 14:33 |
| *** ykarel|away has quit IRC | 14:33 | |
| *** e0ne has quit IRC | 14:33 | |
| fungi | maybe we should add that to https://docs.openstack.org/infra/system-config/zuul.html#sysadmin | 14:34 |
| frickler | fungi: yes, that's the point where I stopped reading backlog and starting my own testing, repeating your later efforts ;) | 14:34 |
| frickler | fungi: and +1 to adding it to the doc | 14:35 |
| fungi | oh, no worries. nice to have independent confirmation of our findings at least! | 14:35 |
| fungi | if any infra-puppet-core is available for a second review on https://review.openstack.org/637084 that's currently blocking the openstackid-dev database move testing | 14:37 |
| *** lpetrut has quit IRC | 14:39 | |
| *** ociuhandu_ has quit IRC | 14:41 | |
| *** ociuhandu_ has joined #openstack-infra | 14:42 | |
| fungi | zbr: do you have a link to an example of how you're calling git review from a job? if the gerrit fetch remote is ssh:// protocol then you'd need to set up credentials, zuul doesn't have any magic which does that for you | 14:45 |
| *** memoussati has joined #openstack-infra | 14:45 | |
| fungi | but also, review refs are deterministic, so you could in theory fetch them directly | 14:45 |
| zbr | fungi: yes, http://logs.openstack.org/44/636144/8/check/tripleo-build-containers-fedora-28/0c2820c/job-output.txt.gz#_2019-02-15_12_41_23_325582 | 14:45 |
| fungi | but also also, having a job fetch refs from gerrit on its own sounds like you're doing it wrong, zuul should be capable of setting up local checkouts for you | 14:46 |
| zbr | fungi: is a special case, where on a specific node and job, we cherrypick a special CR | 14:46 |
| openstackgerrit | Ngong Ivoline-Clarisse proposed openstack-infra/storyboard master: Fixed hyperlinks to worklist and boards documents https://review.openstack.org/637193 | 14:46 |
| fungi | and depends-on isn't sufficient for that? | 14:47 |
| zbr | something that is likely never to merge, but we need it for testing that execution path. is on a repo which already may have its own changeset managed by zuul. | 14:47 |
| zbr | we just one to cherry-pick our "magic" on top of it | 14:47 |
| fungi | interesting. what is the change you're cherry-picking? why not just stuff the diff into a file checked into your test fixtures or something? | 14:48 |
| zbr | i can send you the link to the change if you are curious. but the idea is that apparently the worked is not able to talk to gerrit. | 14:48 |
| zbr | because this is a wip change. ... a moving target. | 14:49 |
| zbr | fungi: but now that you asked, check https://review.openstack.org/#/c/636144/10/playbooks/tripleo-buildcontainers/run.yaml | 14:49 |
| fungi | and yeah, ssh://zuul@review.openstack.org:29418/ would require you have a valid ssh key for the zuul user in gerrit on review.openstack.org | 14:49 |
| zbr | fungi: cannot we use agents? | 14:49 |
| fungi | i have no idea what you mean by that | 14:50 |
| fungi | might as well s/agents/magic/ in that question | 14:50 |
| fungi | you could switch that to some other user account and embed a its ssh private key in an encrypted zuul secret in your job definition | 14:51 |
| fungi | but allowing jobs to perform actions in gerrit as the zuul user would be... exceedingly dangerous | 14:51 |
| *** quiquell|rover is now known as quiquell|off | 14:51 | |
| zbr | fungi: my question is:does zuul worker has access to any key that allows access to gerrit? | 14:52 |
| *** dpawlik has quit IRC | 14:52 | |
| zbr | i would be surprised to be the first to try this..... | 14:52 |
| fungi | it does not, no | 14:52 |
| fungi | you can access gerrit anonymously over https however, no ssh key required | 14:52 |
| *** sthussey has joined #openstack-infra | 14:52 | |
| zbr | fungi: thanks. in this case I will avoid using git-review and use the bash approach which works over https | 14:53 |
| fungi | and no account needed | 14:53 |
| zbr | fungi: git-review was nice/cool/simple, but it works only with ssh. | 14:53 |
| fungi | you _can_ configure git-review to use https instead of ssh, but it's also extreme overkill just to fetch a git ref | 14:53 |
| zbr | in the future I will fix git-review to allow it to perform this without ssh | 14:53 |
| *** dpawlik has joined #openstack-infra | 14:53 | |
| fungi | it already can | 14:53 |
| fungi | configure it to use https if that's what you desire | 14:54 |
| *** auristor has joined #openstack-infra | 15:00 | |
| *** ykarel|away has joined #openstack-infra | 15:04 | |
| *** memoussati has quit IRC | 15:04 | |
| *** dpawlik has quit IRC | 15:05 | |
| *** e0ne has joined #openstack-infra | 15:07 | |
| *** jamesmcarthur has joined #openstack-infra | 15:08 | |
| *** markvoelker has joined #openstack-infra | 15:10 | |
| *** takamatsu has joined #openstack-infra | 15:10 | |
| *** jamesmcarthur has quit IRC | 15:13 | |
| *** bnemec is now known as bnemec-pto | 15:16 | |
| *** dpawlik has joined #openstack-infra | 15:17 | |
| *** dpawlik has quit IRC | 15:22 | |
| *** cmurphy is now known as cmorpheus | 15:22 | |
| *** smarcet has quit IRC | 15:23 | |
| *** jamesmcarthur has joined #openstack-infra | 15:29 | |
| *** jistr is now known as jistr|mtg | 15:32 | |
| *** kmalloc is now known as needscoffee | 15:32 | |
| *** jamesmcarthur has quit IRC | 15:34 | |
| *** chkumar|ruck is now known as chandankumar | 15:38 | |
| *** markvoelker has quit IRC | 15:42 | |
| *** memoussati has joined #openstack-infra | 15:45 | |
| *** smarcet has joined #openstack-infra | 15:45 | |
| openstackgerrit | Ngong Ivoline-Clarisse proposed openstack-infra/storyboard master: Reverted execute permissions on files https://review.openstack.org/637209 | 15:47 |
| *** gfidente has quit IRC | 15:47 | |
| *** gfidente has joined #openstack-infra | 15:48 | |
| *** zhangfei has quit IRC | 15:51 | |
| *** memoussati has quit IRC | 15:51 | |
| *** jamesmcarthur has joined #openstack-infra | 15:51 | |
| *** jamesmcarthur has quit IRC | 15:53 | |
| *** jamesmcarthur has joined #openstack-infra | 15:53 | |
| *** memoussati has joined #openstack-infra | 16:04 | |
| *** jistr|mtg is now known as jistr | 16:04 | |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: Change openstack-dev to openstack-discuss https://review.openstack.org/625390 | 16:06 |
| openstackgerrit | Ngong Ivoline-Clarisse proposed openstack-infra/storyboard master: Fixed hyperlinks to worklist and boards documents https://review.openstack.org/637193 | 16:10 |
| *** takamatsu_ has joined #openstack-infra | 16:11 | |
| openstackgerrit | Tom Barron proposed openstack-infra/openstack-zuul-jobs master: Remove legacy manila artifacts job https://review.openstack.org/632362 | 16:11 |
| *** takamatsu has quit IRC | 16:12 | |
| *** memoussati has quit IRC | 16:15 | |
| *** aojea has quit IRC | 16:20 | |
| *** takamatsu_ has quit IRC | 16:25 | |
| *** takamatsu_ has joined #openstack-infra | 16:28 | |
| *** emccormick has joined #openstack-infra | 16:30 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Rename project to project_name in getPullBySha https://review.openstack.org/637218 | 16:32 |
| emccormick | hello infra folks. Do any of you happen to have admin powers on #openstack-operators? The ops meetup team would like to put something about the upcoming Berlin meetup in the topic for a couple weeks. | 16:32 |
| *** dims has joined #openstack-infra | 16:34 | |
| *** pgaxatte has quit IRC | 16:34 | |
| *** dims has quit IRC | 16:38 | |
| *** markvoelker has joined #openstack-infra | 16:39 | |
| *** ociuhandu_ has quit IRC | 16:41 | |
| *** ociuhandu_ has joined #openstack-infra | 16:42 | |
| *** ociuhandu_ has quit IRC | 16:44 | |
| *** adriancz has quit IRC | 16:44 | |
| smcginnis | Might be useful if emccormick and mihalis68_ have the power in that channel for future updates. | 16:44 |
| smcginnis | I would be willing to help too. | 16:45 |
| *** dims has joined #openstack-infra | 16:45 | |
| corvus | zbr: do not, under any circumstance, fetch a change from gerrit in a job running in zuul | 16:46 |
| openstackgerrit | Mathieu Gagné proposed openstack-infra/project-config master: Revert "Disable inap region due to duplicate IPs" https://review.openstack.org/637222 | 16:46 |
| corvus | zbr: use depends-on for that | 16:48 |
| clarkb | emccormick: smcginnis one sec I'll add emccormick and smcginnis as ops | 16:48 |
| smcginnis | Thanks clarkb | 16:49 |
| *** iurygregory has quit IRC | 16:49 | |
| *** weshay|ruck has joined #openstack-infra | 16:49 | |
| clarkb | I just have to figure out what mode is appropriate | 16:49 |
| corvus | zbr: if you need help figuring out an alternative, i'm happy to help. but do not do what you suggest. (cc fungi) | 16:51 |
| clarkb | emccormick: smcginnis done, you'll need to /op to set the topic but shoudl be set to do that now | 16:52 |
| emccormick | thanks clarkb! | 16:53 |
| fungi | yes, it does certainly seem like a job avoiding relying on zuul's git repository management | 16:53 |
| smcginnis | Much appreciated. | 16:53 |
| *** dims has quit IRC | 16:53 | |
| fungi | corvus: while i agree this seems like a questionable design, the goal of that job is apparently to run some set of tests after further altering the contents of one of the repositories (so the "patching" behavior is tied to the job, not to a particular change being tested) | 16:55 |
| *** dims has joined #openstack-infra | 16:55 | |
| fungi | i'm not immediately seeing how to do that with depends-on, but i also think that storing that detail of the job in some separate gerrit change which is not intended to ever be merged is a very strange idea | 16:56 |
| corvus | fungi: then i think your 'include the patch' idea is the way to go | 16:56 |
| corvus | fungi, zbr: the issue is that our gerrit is not robust enough to have ci jobs fetch refs from it directly. | 16:56 |
| zbr | fungi: corvus : give few moments to write the explanation... | 16:56 |
| fungi | some of this i think likely stems from the tripleo team's ptsd around trying to get changes merged, and a fear thatthey'll be broken for days if they need to try to update that patch inside the repository due to later merge conflicts | 16:57 |
| corvus | i'm not just trying to be bureaucratic here, i'm trying to avoid issues we have seen in the past with zuul performing DOS against gerrit. | 16:58 |
| fungi | zbr: to clarify, you have a job which you want to run with what is essentially a forked version of tripleo-ci, right? | 16:58 |
| corvus | we put *a lot* of work into making zuul and the git farm able to protect gerrit, which is a single server shared by our development community. please do not undermine that work. | 16:59 |
| *** eharney has quit IRC | 16:59 | |
| fungi | is there any reason the behavior provided by the fork can't become optional/switchable behavior of tripleo-ci instead? | 16:59 |
| *** ginopc has quit IRC | 16:59 | |
| *** e0ne has quit IRC | 16:59 | |
| *** ginopc has joined #openstack-infra | 17:00 | |
| *** emccormick has quit IRC | 17:00 | |
| fungi | storing that fork as a perpetually unmerged gerrit change is, to put it bluntly, a very "hacky" design even if fetching arbitrary refs out of the code review system weren't a concern in and of itself | 17:00 |
| *** dpawlik has joined #openstack-infra | 17:01 | |
| *** emccormick has joined #openstack-infra | 17:01 | |
| *** gyee has joined #openstack-infra | 17:01 | |
| *** ginopc has quit IRC | 17:02 | |
| emccormick | clarkm I seem unable to /op myself or set topic. It just says I"m not a channel operator. pardon my weak irc-fu if I'm missing something | 17:02 |
| emccormick | clarkb I seem unable to /op myself or set topic. It just says I"m not a channel operator. pardon my weak irc-fu if I'm missing something | 17:02 |
| emccormick | nice if i'd get the nic right ;) | 17:03 |
| corvus | emccormick: you'll need to ask chanserv to op you -- /msg chanserv help | 17:03 |
| emccormick | aahhh | 17:03 |
| corvus | emccormick: (you can also just ask chanserv to set the topic for you) | 17:03 |
| emccormick | got it! | 17:03 |
| fungi | that's what i always do | 17:05 |
| *** dpawlik has quit IRC | 17:05 | |
| fungi | fewer steps | 17:05 |
| clarkb | I tend to /op then /topic then /deop | 17:05 |
| clarkb | because it is what I can remember :) | 17:06 |
| emccormick | Got it! | 17:06 |
| corvus | i like to keep my topics anonymous :) | 17:06 |
| emccormick | topic now set. Thanks | 17:06 |
| emccormick | haha sensible | 17:06 |
| fungi | i usually find myself needing to set/fix topics in multiple channels at once, so being able to execute a single command per channel is convenient for me | 17:07 |
| fungi | especially since i can do it from an entirely separate buffer and not need to keep switching to buffers for each channel first | 17:08 |
| *** dtantsur is now known as dtantsur|afk | 17:08 | |
| emccormick | I used to know all this stuff in like...1992. I've since forgotten everything :/ | 17:09 |
| fungi | welcome back to 1992! we've missed you | 17:09 |
| corvus | it's just like riding a bike! it's really easy to fall off. | 17:09 |
| fungi | i recommend pads and a helmet | 17:09 |
| *** mattw4 has joined #openstack-infra | 17:10 | |
| emccormick | psh. We didn't wear pads and helmets in 1992! | 17:10 |
| corvus | and we turned out fine! | 17:10 |
| fungi | i have only minor cerebral damage. it builds character | 17:11 |
| emccormick | concussions ftw | 17:11 |
| *** markvoelker has quit IRC | 17:13 | |
| *** mattw4 has quit IRC | 17:14 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Test GithubShaCache https://review.openstack.org/637228 | 17:17 |
| *** rpittau has quit IRC | 17:20 | |
| *** raissa has joined #openstack-infra | 17:21 | |
| *** eernst has joined #openstack-infra | 17:21 | |
| *** kjackal has quit IRC | 17:22 | |
| *** kjackal has joined #openstack-infra | 17:23 | |
| openstackgerrit | Merged openstack-infra/storyboard master: Fixed hyperlinks to worklist and boards documents https://review.openstack.org/637193 | 17:28 |
| zbr | corvus: fungi https://etherpad.openstack.org/p/kolla-and-fedora28 | 17:35 |
| *** eernst has quit IRC | 17:41 | |
| *** ccamacho has quit IRC | 17:42 | |
| *** raissa has quit IRC | 17:44 | |
| corvus | zbr: my question is why not do option (A) but store the patch as a file in the tripleo-ci repo. | 17:46 |
| corvus | zbr: just don't fetch it from gerrit. | 17:46 |
| *** eharney has joined #openstack-infra | 17:46 | |
| zbr | corvus: this patch is very dynamic, people will be working on it. | 17:47 |
| fungi | i don't see why that prevents them from committing it to the repo though | 17:48 |
| fungi | in fact, of that job is run against the repo which the patch is stored in, updates of the patch ought to be entirely self-testing too | 17:48 |
| fungi | er, if that job | 17:49 |
| corvus | ++ | 17:49 |
| zbr | fungi: totally impractical, alternatively it could be fork on github, or a branch of kolla in gerrit. | 17:49 |
| fungi | i'm not seeing the impracticality | 17:50 |
| clarkb | I've put invalid voip.ms credentials in hiera for pbx01.opendev.org and put pbx.openstack.org in the emergency fiel so I can boot a new pbx server on xenial, then cut over the SIP connection once dns is updated | 17:58 |
| smarcet | fungi: clarkb: hello guys, when u get a chance please review https://review.openstack.org/#/c/637084/ its needed for openstackid-dev | 17:59 |
| smarcet | thx :) | 18:00 |
| *** ccamacho has joined #openstack-infra | 18:00 | |
| *** jpena is now known as jpena|off | 18:01 | |
| fungi | smarcet: i did a few hours ago | 18:01 |
| smarcet | fungi: yes sire , i meant to be merged :) | 18:02 |
| *** eernst has joined #openstack-infra | 18:02 | |
| fungi | infra-puppet-core reviewers: 637084 fixes an oversight which is blocking testing the openstackid-dev database move to vexxhost, if you have a moment to take a look | 18:03 |
| *** gfidente has quit IRC | 18:03 | |
| *** derekh has quit IRC | 18:04 | |
| smarcet | fungI: thx sire :) | 18:04 |
| clarkb | looking | 18:05 |
| *** ociuhandu_ has joined #openstack-infra | 18:06 | |
| smarcet | th | 18:06 |
| smarcet | thx | 18:06 |
| *** ociuhandu has quit IRC | 18:09 | |
| clarkb | fungi: did you have any trouble creating rdns records for storyboard-dev01.opendev.org ? | 18:10 |
| *** markvoelker has joined #openstack-infra | 18:10 | |
| fungi | clarkb: i had no problems doing that, no | 18:10 |
| *** ociuhandu_ has quit IRC | 18:10 | |
| fungi | used teh raxdns commands output by the launch script | 18:10 |
| clarkb | ya thats what I'm doing to and get ERROR: 'response' | 18:10 |
| clarkb | I know forward records won't work | 18:10 |
| fungi | for a moment i thought i had problems doing so but then realized that was me trying to create a/aaaa records in rackspace with it | 18:11 |
| fungi | the ptr records were created without error | 18:11 |
| *** paladox has joined #openstack-infra | 18:11 | |
| fungi | i need to go get some lunch. back soon | 18:11 |
| *** eernst has quit IRC | 18:12 | |
| clarkb | heh if I rerun with --debug it says the record already exists | 18:12 |
| clarkb | so maybe the error response was only a fake error | 18:12 |
| clarkb | ya it resolves so maybe everything is ok | 18:12 |
| corvus | error: ok | 18:16 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zone-opendev.org master: Add pbx.opendev.org records https://review.openstack.org/637239 | 18:17 |
| clarkb | maybe don't approve that until I confirm puppet runs and everything is happy | 18:17 |
| clarkb | oh I forgot to bump the serial anyway | 18:18 |
| *** mriedem has quit IRC | 18:18 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zone-opendev.org master: Add pbx.opendev.org records https://review.openstack.org/637239 | 18:18 |
| corvus | we should have a test for that | 18:18 |
| clarkb | ++ | 18:19 |
| clarkb | git diff | grep serial | 18:19 |
| clarkb | something simple like that would get us 95% of the way there (then extra points for checking it is bigger than last value) | 18:19 |
| *** smarcet has quit IRC | 18:20 | |
| *** ijw has joined #openstack-infra | 18:22 | |
| *** mriedem has joined #openstack-infra | 18:24 | |
| *** ociuhandu has joined #openstack-infra | 18:24 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Load buildset registry data from zuul_return https://review.openstack.org/637245 | 18:27 |
| *** ociuhandu has quit IRC | 18:28 | |
| corvus | infra-root: gitea seems to have recovered from the errors i induced yesterday. i'm going to flip the switch on the hostname. | 18:31 |
| *** ijw_ has joined #openstack-infra | 18:31 | |
| openstackgerrit | Merged openstack-infra/zone-opendev.org master: Switch opendev.org to point to gitea https://review.openstack.org/635486 | 18:31 |
| clarkb | exciting | 18:33 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Load buildset registry data from zuul_return https://review.openstack.org/637245 | 18:33 |
| clarkb | the biggest outstanding item remaining is figuring out python kubernetes for the long term? | 18:33 |
| *** ijw has quit IRC | 18:35 | |
| openstackgerrit | Merged openstack-infra/puppet-openstackid master: Fix on mysql ssl certs https://review.openstack.org/637084 | 18:35 |
| corvus | clarkb: yeah, though i'm hoping that will resolve itself soon | 18:36 |
| corvus | clarkb: well, also the k8s-on-openstack playbook doesn't work, but i think mordred wants to rework that a bit | 18:36 |
| corvus | and there's no testing of anything | 18:36 |
| corvus | let's just say "amongst the remaining items" | 18:36 |
| clarkb | I was thinking bout that a bit and we ought to be able to use our existing multinode ansible testing to deploy k8s using the k8s on openstack stuff, we'll just be restricted to not having load balancer stuff I think | 18:37 |
| clarkb | but maybe that gets us 95% of what we want to check anyway? | 18:38 |
| corvus | clarkb: i think one of the things mordred wants to rework is that k8s-on-openstack tries to get the nodes from openstack; i think mordred wants to make it work with already deployed nodes, which would make testing much easier. | 18:38 |
| clarkb | ++ | 18:39 |
| corvus | clarkb: separately, at least at first, i think we can just stand up a k8s and deploy the gitea config with some minor changes (eg, don't use ceph). | 18:39 |
| corvus | that would get us deployment testing of gitea itself, which is the thing we're most likely to change in the near future | 18:39 |
| clarkb | can even tie that to nodepool using the k8s driver ? | 18:40 |
| corvus | clarkb: well, we rely pretty heavily on namespaces, so it may not be the best match there | 18:41 |
| *** ijw_ has quit IRC | 18:42 | |
| tosky | there is a weird spammer on #openstack-unregistered | 18:43 |
| *** markvoelker has quit IRC | 18:43 | |
| clarkb | tosky: ya, but we require registered nicks so unfortunately I Think that means this setup is workign (we just get the spam in the catch all channel) | 18:44 |
| tosky | clarkb: sure, it's just that I haven't seem spam on unregistered in a while | 18:45 |
| AJaeger | infra-root, can we enable inap again? Please +2A https://review.openstack.org/#/c/637222 | 18:46 |
| *** kopecmartin is now known as kopecmartin|off | 18:46 | |
| clarkb | oh right we have static inventory now. I was wondering why puppet didn't run on new pbx | 18:47 |
| clarkb | AJaeger: looking | 18:47 |
| clarkb | thank you mgagne! | 18:47 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Add new pbx01.opendev.org server to inventory https://review.openstack.org/637253 | 18:49 |
| clarkb | infra-root I think we can go ahead and merge https://review.openstack.org/637253 and https://review.openstack.org/637239 I'll just update the IPs in there if I have to rebuild | 18:50 |
| *** psachin has joined #openstack-infra | 18:52 | |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Disable inap region due to duplicate IPs" https://review.openstack.org/637222 | 18:57 |
| *** ykarel|away has quit IRC | 18:59 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Update launch docs for opendev and static inventory https://review.openstack.org/637264 | 19:00 |
| clarkb | infra-root ^ is related notes from what I've learned booting the new server | 19:00 |
| *** roman_g has joined #openstack-infra | 19:04 | |
| *** roman_g has quit IRC | 19:09 | |
| *** luizbag has quit IRC | 19:11 | |
| *** wolverineav has joined #openstack-infra | 19:12 | |
| corvus | does https://38.108.68.64/ error for anyone else? | 19:12 |
| clarkb | yes | 19:13 |
| clarkb | firefox says connection was interrupted | 19:14 |
| *** tjgresha has quit IRC | 19:14 | |
| clarkb | perhaps our load balancer reconfigure didn't apply as expected? | 19:14 |
| corvus | i think it's sometimes working | 19:14 |
| clarkb | I do get redirected from http to https so that bit seems to work | 19:14 |
| corvus | the lb says: | operating_status | ERROR | | 19:16 |
| clarkb | I wonder if it thinks it should be terminating https | 19:16 |
| *** tjgresha has joined #openstack-infra | 19:16 | |
| *** wolverineav has quit IRC | 19:16 | |
| clarkb | no protocol is TCP in the lb config | 19:17 |
| clarkb | only the name is https I doubt it would infer that from the name if protocol is explicity tcp passthrough | 19:17 |
| corvus | now its: | operating_status | DEGRADED | | 19:18 |
| *** wolverineav has joined #openstack-infra | 19:18 | |
| corvus | and my https connection is working | 19:18 |
| corvus | how do we find out more info about a degraded or error lb state? | 19:18 |
| clarkb | johnsom: rm_work ^ may know | 19:18 |
| johnsom | Hi there | 19:19 |
| *** e0ne has joined #openstack-infra | 19:19 | |
| clarkb | johnsom: any idea how we get more info about an openstack load balacner in an error then degraded state? | 19:20 |
| johnsom | You have an Octavia load balancer with which object in operating_status DEGRADED? | 19:20 |
| johnsom | FYI: https://developer.openstack.org/api-ref/load-balancer/v2/index.html#status-codes | 19:20 |
| clarkb | ya is there a request that will show which entities are degraded or in erorr state? | 19:21 |
| johnsom | If it's the load balancer object, then one of the children objects is having a problem. If it's listener, you have more connections than you configured for max connections. If it's the pool, one or more of the members is down. | 19:21 |
| clarkb | corvus: I wonder if the issue had to do with bringing up the backend? and frontend couldn't talk to them | 19:21 |
| johnsom | clarkb https://developer.openstack.org/api-ref/load-balancer/v2/index.html#get-the-load-balancer-status-tree | 19:22 |
| corvus | loadbalancer listener show ... for each of the listeners doesn't indicate any errors | 19:22 |
| *** jamesmcarthur has quit IRC | 19:22 | |
| clarkb | johnsom: that looks like the query we want, now to see if osc exposes it | 19:22 |
| johnsom | Ok, so it's not a load issue. Probably just a member down | 19:22 |
| *** jamesmcarthur has joined #openstack-infra | 19:23 | |
| johnsom | openstack loadbalancer status show <load_balancer> | 19:23 |
| corvus | oh thanks, that's much less typing :) | 19:23 |
| clarkb | corvus: loadbalancer status show | 19:23 |
| corvus | 2 of the pools are degraded | 19:23 |
| clarkb | I'm 30 seconds late | 19:23 |
| clarkb | guessing for http and https? | 19:24 |
| corvus | yeah, i'm guessing so | 19:24 |
| clarkb | possibly because we had to restart the web server to apply tls config? | 19:24 |
| corvus | yeah, that might be racy | 19:24 |
| corvus | confirmed, it's ssh that's working http(s) are not | 19:25 |
| corvus | or rather, they each have one member functioning | 19:25 |
| corvus | presumably the one functioning member failed earlier? i wonder what that was about | 19:26 |
| *** jamesmcarthur has quit IRC | 19:27 | |
| clarkb | healthcheck interval might be long enough that it hasn't noticed they are all back again? | 19:27 |
| clarkb | (assuming k8s did the appropriate restart task) | 19:27 |
| johnsom | I need to step away for a few minutes, ping me if you need more info | 19:27 |
| corvus | clarkb: well, that was 18 hours ago | 19:27 |
| clarkb | corvus: oh right | 19:27 |
| clarkb | corvus: maybe our heathcheck interval is really long? and its just now noticing? though k8s shouldn't removed the pool membersi f they were deleted | 19:28 |
| corvus | clarkb: well, that's assuming that the gitea instances are functioning. i'm assuming the lb is correct and they aren't. | 19:28 |
| corvus | i'm going to try restarting them one at a time, which should tell us whether there's something racy with gitea-init | 19:29 |
| clarkb | ok | 19:29 |
| clarkb | I don't remember the command to do that but I assume it is kubectl -namespace gitea something or other | 19:29 |
| corvus | kubectl --context gitea -n gitea delete pod/gitea-64f8cb784c-pf6fj | 19:30 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Load buildset registry data from zuul_return https://review.openstack.org/637245 | 19:30 |
| corvus | for each of the pods | 19:30 |
| *** gyee has quit IRC | 19:32 | |
| corvus | oh, indeed, 1 of them started, the other 3 are stuck starting up | 19:32 |
| clarkb | learning! | 19:32 |
| *** jamesmcarthur has joined #openstack-infra | 19:33 | |
| corvus | i did kubectl logs on all 4 -- this is the one that's up: http://paste.openstack.org/show/745181/ | 19:33 |
| corvus | the other 3 stop at "Git Version" | 19:33 |
| clarkb | fungi: before you jump into storyboard things care to give a quick look at https://review.openstack.org/637239 https://review.openstack.org/637253 and https://review.openstack.org/637264 for making progress on pbx | 19:33 |
| clarkb | corvus: is it possible there is a port conflict if network namespaces aren't as we expect? | 19:34 |
| clarkb | corvus: that seems unlikely given what k8s is doing for us | 19:34 |
| clarkb | but maybe they can't listen on port 3000 | 19:34 |
| corvus | they do have an anti-affinity rule so they should all be on different hosts | 19:35 |
| * fungi is trying to catch up before "storyboard things" | 19:35 | |
| corvus | and they are | 19:36 |
| *** jamesmcarthur has quit IRC | 19:37 | |
| *** jamesmcarthur has joined #openstack-infra | 19:38 | |
| * clarkb attempts to use some kubectl | 19:39 | |
| clarkb | I will do read only actions | 19:39 |
| *** jamesmcarthur has quit IRC | 19:40 | |
| *** markvoelker has joined #openstack-infra | 19:41 | |
| fungi | clarkb: merge conflict on 637239? | 19:41 |
| *** wolverineav has quit IRC | 19:42 | |
| corvus | oh.... | 19:42 |
| corvus | clarkb: i *think* one of the gitea instances has a lock on the issues index, and the others are waiting for it | 19:42 |
| *** shardy has quit IRC | 19:43 | |
| corvus | though, it's the smaller one, so it seems weird that it would cause a problem | 19:43 |
| *** yamamoto has quit IRC | 19:43 | |
| clarkb | fungi: I don't know why that shows up I don't think there actually is one as there is no conflicts with list | 19:43 |
| corvus | clarkb: i wonder if gitea now requires exclusive locks on that. maybe that's a change in one of our version upgrades. | 19:44 |
| clarkb | oh right we've done some upgrades | 19:44 |
| clarkb | and we've disabled issues in the ui not in the service itself right? | 19:45 |
| clarkb | I wonder if we can avoid something like that by more properly disabling the feature | 19:45 |
| corvus | clarkb: well, maybe we can turn it off, but i expect the next thing after that would be the same lock on the code index | 19:45 |
| corvus | which is one of the main attractions of using gitea | 19:45 |
| *** gyee has joined #openstack-infra | 19:46 | |
| clarkb | hrm | 19:47 |
| clarkb | corvus: which version are we running now? | 19:47 |
| *** wolverineav has joined #openstack-infra | 19:48 | |
| corvus | 1.7.1 | 19:48 |
| *** jamesmcarthur has joined #openstack-infra | 19:48 | |
| clarkb | there is a 1.7.2 from yesterday, but release notes don't say anything about locks (1.7.0 fixed a bunch of lock issues apparently) | 19:48 |
| corvus | i'm deleting the working pod to see what happens with the rest | 19:50 |
| corvus | one of the others got past its block and is serving now | 19:51 |
| corvus | so it's definitely looking like an exclusive lock situation | 19:51 |
| *** wolverineav has quit IRC | 19:53 | |
| clarkb | https://github.com/go-gitea/gitea/issues/2959 remains an open issue I'm skimming it for any possible lock issues | 19:54 |
| fungi | clarkb: "that" being the -2 failure from zuul? shall i just recheck and hope the merge failure was transient? | 19:55 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zone-opendev.org master: Add pbx.opendev.org records https://review.openstack.org/637239 | 19:56 |
| clarkb | fungi: it was conflict on serial due to updating the opendev.org records to point at gitea | 19:57 |
| clarkb | that patchset should make it all happy | 19:57 |
| fungi | yeah, we will *always* merge-conflict on updates to zone files since they should always include a serial update | 19:57 |
| *** psachin has quit IRC | 19:58 | |
| clarkb | corvus: reading that I'm not sure this use case is totally supported but people are using docker-swarm (like k8s) to spin up replacement gitea services when others die | 19:58 |
| openstackgerrit | Merged openstack-infra/zone-opendev.org master: Add pbx.opendev.org records https://review.openstack.org/637239 | 19:58 |
| clarkb | corvus: so maybe we just run this with 2 pods, one active, the other hot standby | 19:58 |
| corvus | clarkb: that won't work for us | 19:58 |
| corvus | also, this worked in a previous version | 19:58 |
| corvus | clarkb: we can't replace a git farm of 8 apache servers with a single go server | 19:59 |
| clarkb | corvus: ya disappointing if they've broken this functionality in the newer release | 19:59 |
| fungi | 10 seconds till storyboard maintenance | 19:59 |
| fungi | #status notice The StoryBoard service on storyboard.openstack.org is offline momentarily for maintenance: http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002666.html | 20:02 |
| openstackstatus | fungi: sending notice | 20:02 |
| *** kjackal has quit IRC | 20:02 | |
| *** kjackal has joined #openstack-infra | 20:02 | |
| -openstackstatus- NOTICE: The StoryBoard service on storyboard.openstack.org is offline momentarily for maintenance: http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002666.html | 20:03 | |
| clarkb | corvus: do you think it is worth an upgrade to 1.7.2 to see if that fixes it? | 20:04 |
| openstackstatus | fungi: finished sending notice | 20:05 |
| clarkb | https://github.com/go-gitea/gitea/commit/b65e37ed722da94f62c90e456413a99e6ec9f419 is in 1.7.2 I think | 20:08 |
| clarkb | I'm now reading to see if the issue fixed mightbe the issue we are seeing | 20:08 |
| corvus | clarkb: couldn't hurt. and worst-case, we just drop the shared aspect of this, and have 8 separate gitea instances. we'd lose some future opportunities, but we can at least do what's on the current docket. | 20:08 |
| clarkb | https://github.com/go-gitea/gitea/issues/5798 | 20:09 |
| clarkb | thats our bug | 20:09 |
| clarkb | interestingly they report seeing the problem in 1.6.4 | 20:10 |
| fungi | storyboard.openstack.org maintenance is concluded. looks like everything's working to me | 20:10 |
| clarkb | a per instance index file is suggested as a possible fix | 20:10 |
| clarkb | corvus: ^ but the person testing it hasn't reported back on whether or not that helps | 20:10 |
| clarkb | fungi: yay | 20:10 |
| corvus | fungi: yep, seems to check out for me | 20:10 |
| fungi | i'll get the old server powered off and snapshotted now | 20:11 |
| clarkb | corvus: ya and "This probably didn't happen to us on 1.5 due to the issue fixed by #5563." | 20:12 |
| clarkb | I think I misread githubs .. between the tags | 20:12 |
| clarkb | it was abbreviating a list of tag the code is in not implying a delta | 20:13 |
| corvus | clarkb: i wonder if i'm misreading lsof/strace, and we're actually waiting on the reindex of the code | 20:13 |
| *** markvoelker has quit IRC | 20:13 | |
| clarkb | possibly https://github.com/go-gitea/gitea/pull/5563 says the issue was writing out version data to the new index | 20:14 |
| clarkb | its possible that no one else can act on that until its fully written out? but then they can increment versions via negotiation of the lock? | 20:14 |
| clarkb | https://github.com/go-gitea/gitea/issues/5325 has more info | 20:15 |
| corvus | i'm finding this network of issues/prs extremely confusing | 20:16 |
| *** wolverineav has joined #openstack-infra | 20:21 | |
| *** wolverineav has quit IRC | 20:27 | |
| *** jamesmcarthur has quit IRC | 20:31 | |
| clarkb | corvus: https://github.com/go-gitea/gitea/blob/master/vendor/github.com/boltdb/bolt/db.go#L179-L189 I think that gives us a bit more info | 20:32 |
| clarkb | and now I need to eat lunch | 20:32 |
| *** agopi is now known as agopi|bbiab | 20:32 | |
| *** ccamacho has quit IRC | 20:36 | |
| corvus | clarkb: on the network front -- when i try to connect to the internal service ip/port of the working gitea web server, i get connection refused about half the time. it looks like the pool members are not, as you might expect, individual gitea pods, but rather, the service ingress port on each of the k8s nodes. so because k8s thinks that all 4 gitea pods are up, it internally routes requests to | 20:36 |
| corvus | all 4 pods. | 20:36 |
| *** agopi|bbiab has quit IRC | 20:37 | |
| *** jamesmcarthur has joined #openstack-infra | 20:38 | |
| *** agopi|bbiab has joined #openstack-infra | 20:43 | |
| *** jamesmcarthur has quit IRC | 20:48 | |
| *** e0ne has quit IRC | 20:51 | |
| *** jamesmcarthur has joined #openstack-infra | 20:52 | |
| *** jamesmcarthur_ has joined #openstack-infra | 20:55 | |
| *** jamesmcarthur_ has quit IRC | 20:57 | |
| *** jamesmcarthur has quit IRC | 20:58 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add new pbx01.opendev.org server to inventory https://review.openstack.org/637253 | 20:58 |
| openstackgerrit | Merged openstack-infra/system-config master: Update launch docs for opendev and static inventory https://review.openstack.org/637264 | 20:58 |
| *** wolverineav has joined #openstack-infra | 21:03 | |
| *** wolverineav has quit IRC | 21:08 | |
| *** markvoelker has joined #openstack-infra | 21:10 | |
| clarkb | corvus: fun | 21:14 |
| clarkb | corvus: my raed of that code is we could probably add support to gitea to have read only index openers (then do one writer and many readers) or have them all write to different indexes | 21:15 |
| corvus | clarkb: when we replicate to gitea, only one of them is going to process that event, so any gitea server would need to be able to write | 21:18 |
| clarkb | ah | 21:19 |
| clarkb | separate indexes wouldn't work either then | 21:19 |
| corvus | yeah, they'd be updated at random | 21:19 |
| corvus | we could potentially have separate indexes *and* git repos, and push to each gitea | 21:20 |
| corvus | they'd keep a shared db. that might work. | 21:20 |
| fungi | so sort of like how we handle the current git farm i guess | 21:20 |
| fungi | but with a common backend | 21:20 |
| clarkb | we may also want to express our interest in fixing https://github.com/go-gitea/gitea/issues/5798 perhaps we can work with them to get a change in to grab a lock rather than hold it indefinitely | 21:20 |
| corvus | (so we might still be able to use a shared wiki) | 21:20 |
| clarkb | that may not fully fix things (pushes could be slow as things lock and unlock) | 21:21 |
| fungi | yeah, a farm of read-only giteas with one (possibly hidden) gitea that can do writes is comparatively easy i guess, but as soon as we want people to potentially interact with the interface in any stateful way that falls apart | 21:22 |
| corvus | i have an interest in it being fixed, but i'm afraid i can't fix it myself. i'm far too overextended on this anyway. | 21:22 |
| *** wolverineav has joined #openstack-infra | 21:23 | |
| corvus | if we can't do shared everything, i'm somewhat tempted to do shared nothing and just build it out like we have the current git farm | 21:23 |
| clarkb | ya more thinking it might draw more effort on their end to it if more people said "hey this is important and a bad regression" than for us to do all the owrk | 21:23 |
| corvus | clarkb: yeah. i wish i had a better handle on what worked previously and when. | 21:23 |
| clarkb | corvus: ya, that may be simpler overall. Then we don't need ha mysql and ha disk and ha indexes | 21:24 |
| corvus | there are too many variables for me to say exactly what happened | 21:24 |
| fungi | i'm compelled by starting with shared-nothing as the shortest path to our primary goals, with the possibly to rearchitect it to turn on stateful features later | 21:24 |
| clarkb | reading the bugs I'm pretty sure https://github.com/go-gitea/gitea/commit/b65e37ed722da94f62c90e456413a99e6ec9f419 is what broke us. It was merged to 1.7.dev then backported to 1.6.something | 21:24 |
| corvus | yeah, no reason we can't go back to this when it's in better shape | 21:24 |
| clarkb | we must've been running 1.5 or 1.6.something less than where that backport ended up | 21:25 |
| clarkb | then when we upgraded it broke us | 21:25 |
| corvus | clarkb: ah, that would make sense then | 21:25 |
| *** wolverineav has quit IRC | 21:25 | |
| corvus | we were running 1.6.0 i think | 21:25 |
| corvus | then 1.6.4, then 1.7.1 | 21:25 |
| fungi | so the 1.6.4 upgrade is where we probably broke but just didn't notice until 1.7.1? | 21:25 |
| fungi | or was the 1.6.x branch backport after .4? | 21:26 |
| corvus | we didn't do much testing of 1.6.4, so while it seemed okay, it's possible we were just really lucky. | 21:26 |
| clarkb | 1.6.4 is reported to be broken at https://github.com/go-gitea/gitea/issues/5798 | 21:26 |
| fungi | got it | 21:27 |
| fungi | so after | 21:27 |
| corvus | oops, 1.6.3 is what we ran | 21:27 |
| corvus | so 1.6.0, 1.6.3, 1.7.1 | 21:27 |
| fungi | er, perhaps before .4 then from the look of that issue? i misunderstood which broken you meant ;) | 21:28 |
| corvus | clarkb: let me see if i can draft up a potentially useful blurb to add to that issue | 21:28 |
| fungi | immaterial now anyway i guess | 21:28 |
| *** wolverineav has joined #openstack-infra | 21:28 | |
| clarkb | it is possible the fix happened in bleve or boltdb. I don't quite understand how the change above would've impacted locking, but the people that should know more than me keep pointing at it in various issues :) | 21:30 |
| *** whoami-rajat has quit IRC | 21:30 | |
| clarkb | its possible we were racing on https://github.com/go-gitea/gitea/blob/master/modules/indexer/indexer.go#L78-L82 | 21:31 |
| clarkb | because the metadata for version wasn't set properly all of the daemons started up and deleted the index then reindexed which creates new files with new file descriptors allowing them to all take the lock | 21:31 |
| clarkb | what is impressive to me if that is what was happening is that the indexes seemed to work :) | 21:32 |
| fungi | evolution through random mutation | 21:32 |
| corvus | oooh :) | 21:32 |
| corvus | clarkb, fungi: do you think http://paste.openstack.org/show/745187/ is at least on the useful side of a "me too" comment? also, should i tell them who "we" are? ;) | 21:34 |
| fungi | expanding "we" could possibly help, but already lgtm | 21:35 |
| clarkb | ya lgtm as is | 21:35 |
| fungi | possible once in production that way we'd have started to see update delays or timeouts due to write lock contention, i suppose | 21:36 |
| clarkb | also cool that flock seems to work on cephfs | 21:37 |
| clarkb | I probably shouldn't be surprised but its always neat to see itnerfaces like that do what you expect | 21:37 |
| *** rh-jelabarre has quit IRC | 21:37 | |
| corvus | clarkb: cephfs has a page somewhere about how posix compliant they are.. and actually how posix compliant doesn't mean anything... and really they're kind of more than posix compliant in some ways... | 21:38 |
| corvus | here it is http://docs.ceph.com/docs/mimic/cephfs/posix/ | 21:38 |
| corvus | i'm poking at setting up shared-nothing gitea | 21:39 |
| clarkb | that is an informative little doc | 21:40 |
| clarkb | puppet is about to run on pbx01.opendev.org. Assuming that goes well I'll be coordinating a switch over. PLan is to stop pbx.openstack.org, restart pbx01.opendev.org with proper voip.ms credentials (update hiera too), update dns for pbx.openstack.org | 21:43 |
| *** markvoelker has quit IRC | 21:43 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Correct host variable in push-to-intermediate-registry https://review.openstack.org/637310 | 21:43 |
| corvus | clarkb, fungi: ^ another tiny change for the registry work | 21:43 |
| clarkb | ok puppet failed because it couldn't find the hieradata. I created an /etc/ansible/hosts/host_vars/pbx01.opendev.org and that didn't end up on the host for puppet apply | 21:47 |
| clarkb | I'm debugging that now | 21:47 |
| clarkb | need that .yaml suffix | 21:49 |
| clarkb | ok next time around should do what I need | 21:50 |
| *** trown is now known as trown|outtypewww | 21:56 | |
| *** rlandy has quit IRC | 21:58 | |
| fungi | why not move to group_vars? | 22:04 |
| *** kjackal has quit IRC | 22:04 | |
| clarkb | fungi: beacuse I've got bad sip credentials to start | 22:05 |
| clarkb | so that the two servers don't fight over that connection | 22:05 |
| fungi | oh, right | 22:05 |
| clarkb | my plan was to make sure puppet does everything else as expected, then update credentials and stop the other server (then I can mv the correct file over to group_vars) | 22:06 |
| clarkb | it is probably overkill but meh | 22:06 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul master: Uncap mypy and ignore out yaml ambiguity https://review.openstack.org/637313 | 22:08 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul master: WIP: Add type hints to zuul.model https://review.openstack.org/637314 | 22:08 |
| *** wolverineav has quit IRC | 22:09 | |
| *** wolverineav has joined #openstack-infra | 22:09 | |
| *** agopi|bbiab has quit IRC | 22:22 | |
| *** wolverineav has quit IRC | 22:27 | |
| *** wolverineav has joined #openstack-infra | 22:28 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Correct host variable in push-to-intermediate-registry https://review.openstack.org/637310 | 22:31 |
| *** wolverineav has quit IRC | 22:33 | |
| clarkb | hrm still not copying the file | 22:33 |
| * clarkb reads the ansible puppet module | 22:33 | |
| *** markvoelker has joined #openstack-infra | 22:40 | |
| *** dave-mccowan has quit IRC | 22:45 | |
| clarkb | ok I'm stumped. ansible-role-puppet looks for host_vars/%s.yaml % ansible_fqdn | 22:47 |
| clarkb | running ansible pbx01.opendev.org -m setup I see the ansible_fqdn fact is pbx01.opendev.org and pbx01.opendev.org.yaml exists in host_vars | 22:47 |
| clarkb | I'm tempted to just switch to using the group except that now I want to understand why this isn't working | 22:48 |
| corvus | clarkb: nothing's coming to mind, sorry | 22:49 |
| clarkb | I know the little helper function is being called because all.yaml is copied and it is set by the helper function | 22:53 |
| clarkb | I've made a group file to see if that works as expected | 22:56 |
| *** EvilienM is now known as EmilienM | 22:56 | |
| *** wolverineav has joined #openstack-infra | 22:59 | |
| clarkb | mordred: assuming that the group version works we should debug why the host_vars version doesn't when you get back from partying in louisiana | 23:01 |
| clarkb | mordred: but I am pretty stumpted right now. I'm guessing the next thing is to add logging to that module and run it in the foreground? | 23:01 |
| clarkb | Its probably something silly I've done on my end :/ | 23:03 |
| *** wolverineav has quit IRC | 23:04 | |
| *** rascasoft has quit IRC | 23:06 | |
| *** jmorgan1 has joined #openstack-infra | 23:13 | |
| *** markvoelker has quit IRC | 23:13 | |
| *** wolverineav has joined #openstack-infra | 23:13 | |
| *** rkukura_ has joined #openstack-infra | 23:15 | |
| *** wolverineav has quit IRC | 23:16 | |
| *** rkukura has quit IRC | 23:18 | |
| *** rkukura_ is now known as rkukura | 23:18 | |
| *** wolverineav has joined #openstack-infra | 23:20 | |
| *** xek has quit IRC | 23:21 | |
| clarkb | ok maybe I was just impatient? it ended up copying both files on the most recent run. I've flipped the service over from pbx.openstack.org to pbx01.opendev.org in DNS (its a CNAME from old name to new name) and with voip.ms connection | 23:21 |
| clarkb | I have confirmed that the voipms DID number works | 23:21 |
| corvus | clarkb: is that subject to the two-run thing? | 23:21 |
| clarkb | corvus: I didn't think so. But we copy the hiera files quite a few minutes before we run puppet and maybe I added the file in between those two? | 23:22 |
| clarkb | corvus: any chance you can test the SIP connection via pbx.openstack.org? dns looks good from here now | 23:22 |
| clarkb | I'll watch it on the next puppet run to make sure it does what I expect as well | 23:23 |
| clarkb | And I'll probably plan to delete the old server on Monday assuming everything still looks good then | 23:23 |
| corvus | clarkb: i just connected to something and am in conference 6000 | 23:24 |
| corvus | clarkb: any possibility it's the old server? | 23:24 |
| clarkb | corvus: I don't think so. But I'll double check that process isn't running | 23:24 |
| *** wolverineav has quit IRC | 23:25 | |
| clarkb | ps shows no asterisk running on the old server | 23:25 |
| corvus | then we're probably good on the sip front | 23:25 |
| clarkb | I can check netstat on new server to double confirm | 23:25 |
| corvus | i just rejoined | 23:25 |
| corvus | (i had hung up right before you said check netstat) | 23:25 |
| clarkb | hrm its udp right? not sure netstat will actually show that as a connection | 23:26 |
| clarkb | I'm pretty certain you are talking to the new server | 23:26 |
| corvus | we're probably good :) | 23:26 |
| *** panda is now known as panda|off | 23:28 | |
| clarkb | #status log pbx01.opendev.org now hosting pbx.openstack.org. Old server to be deleted next week if all looks well then. | 23:28 |
| openstackstatus | clarkb: finished logging | 23:28 |
| *** takamatsu_ has quit IRC | 23:29 | |
| clarkb | I do notice astcanary isn't running on the new server. Maybe newer asterisk doesn't need it? | 23:33 |
| clarkb | oh interesting it is because we don't run with the -p flag anymore | 23:34 |
| *** ekultails has quit IRC | 23:38 | |
| clarkb | The reason for this is the asterisk unit file doesn't specify the -p. The old sysv script and the defaults file which are still both installed default to -p | 23:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Deploy gitea with docker-compose https://review.openstack.org/637330 | 23:41 |
| clarkb | I guess if we find this has problems we can try to change that, but until we have trouble with it sticking to defaults is probably fine | 23:41 |
| corvus | clarkb: hrm. i guess time will tell. | 23:41 |
| corvus | i will miss the tweet.tweet.tweet. | 23:41 |
| clarkb | ha | 23:41 |
| corvus | clarkb, fungi: ^ that change 637330 should have everything we need to do shared-nothing gitea. it uses our existing images, but just does docker compose on a regular server. | 23:42 |
| clarkb | and we'd run N of those regular servers? makes sense | 23:42 |
| corvus | yep | 23:42 |
| corvus | i did not use host networking there, mostly because this was the easiest way to map ssh to a high port (which we'll need to do on these servers since there's no vip, and they'll be running a normal ssh already) | 23:43 |
| corvus | if we really want to use host networking, i'm sure we can figure something out. | 23:43 |
| *** ijw has joined #openstack-infra | 23:43 | |
| corvus | that should be a complete change ready to go. i've tested it locally. (that's a bonus of this method, testing both in zuul and locally is a bit easier given our current toolset) | 23:44 |
| corvus | don't be alarmed by the passwords -- they're all random just for the tests. i did not accidentally copy the prod passwords :) | 23:45 |
| *** betherly has quit IRC | 23:45 | |
| *** yamamoto has joined #openstack-infra | 23:45 | |
| corvus | (i usually use more obvious test passwords, but enough of these needed to be legitimate secret keys, i just did them all) | 23:45 |
| clarkb | re the host networking problem, remember that docker fights out iptables rules updates | 23:45 |
| clarkb | (its not the end of the world, restart docker after updating iptables) | 23:46 |
| corvus | hrm. | 23:46 |
| corvus | maybe it's worth telling sshd to run on a high port | 23:46 |
| *** wolverineav has joined #openstack-infra | 23:47 | |
| clarkb | the current compose does 222:22 probably not a big deal if we already expose it as a high port publicly? | 23:48 |
| corvus | gitea-ssh runs on 22, that maps it to 222 | 23:49 |
| clarkb | ya | 23:49 |
| corvus | if we want to use host networking, we need to tell gitea's opensshd to run on something other than 22 | 23:49 |
| *** gyee has quit IRC | 23:50 | |
| *** hwoarang has quit IRC | 23:50 | |
| *** yamamoto has quit IRC | 23:50 | |
| corvus | it gets its config file from the upstream docker image | 23:50 |
| corvus | so we could either sed it in our docker image, or replace it | 23:51 |
| clarkb | gotcha | 23:51 |
| clarkb | I'm not opposed to trying this (I think we are learning all the things trying stuff) | 23:51 |
| corvus | this is the perfect example of what i want to use the intermediate registry for -- i'd like to make a change to the gitea docker image, then make a change to the docker-compose | 23:52 |
| clarkb | and restarting docker might not be so bad with a proxy/load balancer in front | 23:52 |
| *** wolverineav has quit IRC | 23:52 | |
| corvus | if that job were in place, we could test this without landing either of those | 23:52 |
| clarkb | ++ | 23:52 |
| clarkb | getting close right? | 23:52 |
| corvus | hope so :) i just rechecked it | 23:53 |
| *** gyee has joined #openstack-infra | 23:54 | |
| *** hwoarang has joined #openstack-infra | 23:54 | |
| corvus | meh. maybe i'll just embed it. it's extremely unlikely to change. | 23:54 |
| clarkb | puppet just ran on pbx01 and didn't change anything so I think that is done for now | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!