| *** dustinc has quit IRC | 00:01 | |
| fungi | ahh, https://github.com/percona/percona-xtradb-cluster | 00:03 |
|---|---|---|
| clarkb | its built on galera right? | 00:04 |
| fungi | so we're all right? percona server + galera basically | 00:04 |
| clarkb | I seem to recall there was some relationship there | 00:04 |
| clarkb | ya and percona server is oracle mysql with patches | 00:04 |
| fungi | "Percona XtraDB Cluster is based on Percona Server and Codership wsrep API using galera replication library that provides Multimaster Cluster based on synchronous replication." | 00:05 |
| ianw | damn, just needed "serverless" and that would have won buzzword bingo | 00:06 |
| mordred | bingo! | 00:08 |
| mordred | (and yes, sorry - got distracted - totally basedon galera | 00:09 |
| *** smarcet has quit IRC | 00:21 | |
| *** rlandy|rover is now known as rlandy|rover|bbl | 00:23 | |
| *** rascasoft has joined #openstack-infra | 00:24 | |
| clarkb | ianw: not sure if you ntoiced, but I'd like to bring up the lets encrypt discussion again as its going to be relevant to any opendevification of services that we do in the near future | 00:25 |
| clarkb | I pointed fungi at it too fwiw | 00:25 |
| fungi | yup | 00:27 |
| clarkb | ok http://logs.openstack.org/11/624011/1/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/c86035d/job-output.txt.gz#_2019-01-07_23_10_14_368698 is our ssh problem in limestone. http://logs.openstack.org/11/624011/1/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/c86035d/zuul-info/inventory.yaml says both nodes ran on the same hypervisor | 00:29 |
| clarkb | I think that rules out switch gear or anything in between hypervisors like that | 00:29 |
| clarkb | the two VMs are one IP apart from each other so are in the same subnet | 00:30 |
| clarkb | logan-: ^ fyi I think that rules out a bunch of potential problems | 00:30 |
| logan- | yeah for sure | 00:30 |
| *** rascasoft has quit IRC | 00:31 | |
| clarkb | security groups should be wide open, but maybe we should double check that | 00:31 |
| clarkb | we know sshd is running and listening on at least ipv6 addrs | 00:32 |
| clarkb | it could potentially be not listening on ipv4 (thought that doesn't explain the ovs vxlan failure that mimic this issue) | 00:32 |
| clarkb | also vxlan is udp not tcp | 00:32 |
| clarkb | fungi: ^ you don't have any good ideas do you? :P | 00:32 |
| fungi | this was two nodes on the same hypervisor failing to reach one another? | 00:33 |
| logan- | is it possible to autohold a multinode group? | 00:33 |
| clarkb | fungi: yes ssh from one to another over their private ipv4 interfaces failed and they are on the same hypervisor | 00:34 |
| clarkb | logan-: ya, but it is job oriented (and I don't think we can filter by cloud currently?) | 00:34 |
| clarkb | the ports for the two instances should be plugged into the same ovs switch and then see each other locally | 00:35 |
| logan- | would be great to do some ping tests while tcpdumping the tap interfaces and examine the lxb forwarding table on one of these broken pairs | 00:36 |
| clarkb | oh are they linux bridge not ovs? similar situation in that case, they'd be on the same bridge on the hypervisor | 00:36 |
| logan- | right | 00:36 |
| clarkb | logan-: maybe we can aritificially test this booting our centos7 image on the same hypervisor then ssh between them? | 00:37 |
| clarkb | if it is somewhat deterministic that may be an easier reproduction method than trying to hold the job in limestone | 00:37 |
| *** Swami has joined #openstack-infra | 00:38 | |
| logan- | i can create a flavor that only schedules to this particular hv, one sec | 00:38 |
| fungi | yeah, wondering if there's ebtables complication at play or something | 00:39 |
| clarkb | we've also got a few pairs running right now I think | 00:39 |
| clarkb | let me see if I can just jump on ssh/telnet port 22 | 00:40 |
| clarkb | I can ping and ssh between two randomly chosen centos7 nodes there | 00:42 |
| clarkb | so maybe this isn't deterministic (unless host_id does play a part somehow) | 00:42 |
| logan- | yeah it seems to be somewhat intermittent | 00:43 |
| logan- | nodepool.2 flavor will always schedule on ed7b1ca2718fc1b2a54ae0881c6760c05caf172968fce6936dc41602 | 00:43 |
| clarkb | cool give me a minute and I'll boot two test nodes there | 00:44 |
| clarkb | I can ping and ssh from both sides (though I'm only confirm the ssh connection connects not that I can fully ssh in) | 00:48 |
| clarkb | seems like this isn't determistic on a host known to do it | 00:49 |
| clarkb | I've also double checked the security group rules and they seem fine. We have an ingress rule allowing 0.0.0.0/0 without specifying any ports and an egress that doesn't specify anything which should mean its all allowed as expected | 00:59 |
| *** Swami has quit IRC | 00:59 | |
| *** rascasoft has joined #openstack-infra | 00:59 | |
| *** yamamoto has joined #openstack-infra | 01:02 | |
| *** rascasoft has quit IRC | 01:05 | |
| fungi | could it be _just_ the vxlan encapsulation which isn't working there" | 01:07 |
| fungi | ?> | 01:07 |
| *** wolverineav has quit IRC | 01:07 | |
| clarkb | fungi: possibly, except that I think the tripleo ssh connection between two nodes never gets vxlan tunneled | 01:08 |
| *** wolverineav has joined #openstack-infra | 01:09 | |
| clarkb | fungi: those hosts will be directly attached to the same ovs switch or linux bridge (depending on cloud setup) and should interface there before any encapsulation happens | 01:09 |
| clarkb | fungi: we've also seen failure of the ovs vxlan setup in other jobs (we ping between the two nodes) which could be a different issue entirely and be related to vlxan encapsulation | 01:09 |
| clarkb | curiously this tripleo job's multinode networking setup doesn't seem to ping between the nodes over the vxlan addrs (which in turn go over the private ipv4 addr) | 01:11 |
| *** wolverineav has quit IRC | 01:13 | |
| clarkb | maybe we only do that ping as part of the role tests | 01:15 |
| clarkb | ya that is part of the role test and not the role itself | 01:15 |
| *** zhangfei has joined #openstack-infra | 01:16 | |
| *** roman_g has quit IRC | 01:18 | |
| clarkb | logan-: fungi: I've got to go and find dinner, but thinking ahead maybe we boot new nodes on that flavor until something breaks? or work with corvus and shrews to see if we can hold nodes on a specific provider (I think pabelanger wasalso looking at this or wat it dmsimard also for this issue) | 01:20 |
| clarkb | on the tripleo side we may want to drive log collection from the executor so that we can have logs in these cases too | 01:21 |
| clarkb | (maybe that will help us see something important?) | 01:21 |
| anteaya | got an email back from the person who had difficulty with the wiki, I got an apology | 01:21 |
| anteaya | I explained I fixed the issue, they said they are trying to figure out irc | 01:21 |
| anteaya | so they might pop up sometime | 01:21 |
| clarkb | I'll leave clarkb-test1 and clarkb-test2 running in limestone too as maybe it degrades after some period of time and I can recheck ping and ssh tomorrow morning | 01:22 |
| *** wolverineav has joined #openstack-infra | 01:27 | |
| *** wolverineav has quit IRC | 01:32 | |
| *** rascasoft has joined #openstack-infra | 01:38 | |
| *** rascasoft has quit IRC | 01:44 | |
| *** hwoarang has joined #openstack-infra | 01:44 | |
| dmsimard | clarkb: I looked at how we might be able to reserve on specific providers but fell short of time to get a better understanding of the model | 01:45 |
| *** hwoarang_ has quit IRC | 01:46 | |
| *** shardy has quit IRC | 01:57 | |
| *** jamesmcarthur has joined #openstack-infra | 01:58 | |
| *** hwoarang has quit IRC | 02:01 | |
| *** hwoarang has joined #openstack-infra | 02:02 | |
| *** jamesmcarthur has quit IRC | 02:03 | |
| *** shardy has joined #openstack-infra | 02:04 | |
| *** bobh_ has joined #openstack-infra | 02:04 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:06 | |
| *** bobh_ has quit IRC | 02:08 | |
| *** bhavikdbavishi has quit IRC | 02:09 | |
| *** rkukura has quit IRC | 02:15 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:20 | |
| *** armax has quit IRC | 02:31 | |
| *** rlandy|rover|bbl is now known as rlandy|rover | 02:33 | |
| *** hongbin has joined #openstack-infra | 02:37 | |
| *** psachin has joined #openstack-infra | 02:41 | |
| *** ykarel has joined #openstack-infra | 02:57 | |
| *** ykarel has quit IRC | 03:04 | |
| *** ykarel has joined #openstack-infra | 03:05 | |
| *** yamamoto has quit IRC | 03:05 | |
| *** ramishra has joined #openstack-infra | 03:11 | |
| *** fuentess has quit IRC | 03:11 | |
| *** apetrich has quit IRC | 03:16 | |
| *** whoami-rajat has joined #openstack-infra | 03:16 | |
| *** yamamoto has joined #openstack-infra | 03:22 | |
| *** yamamoto has quit IRC | 03:27 | |
| *** yamamoto has joined #openstack-infra | 03:27 | |
| *** hwoarang_ has joined #openstack-infra | 03:30 | |
| *** hwoarang has quit IRC | 03:31 | |
| *** shardy has quit IRC | 03:52 | |
| *** shardy has joined #openstack-infra | 03:53 | |
| *** portdirect has joined #openstack-infra | 03:57 | |
| *** hongbin has quit IRC | 03:57 | |
| ianw | "mkswap: error: /root/swapfile is mounted; will not make swapspace" | 04:09 |
| ianw | http://logs.openstack.org/15/628815/1/check/tripleo-buildimage-overcloud-full-centos-7/2e8ed6a/ara-report/ | 04:09 |
| ianw | ahh,weird, this is a second call to swap setup in run-v3.yaml ... i guess this is a tripleo specific failure. at first i was worried it indicated node reuse | 04:11 |
| *** udesale has joined #openstack-infra | 04:12 | |
| *** spsurya has joined #openstack-infra | 04:22 | |
| *** bhavikdbavishi has quit IRC | 04:25 | |
| *** rfolco has quit IRC | 04:34 | |
| *** rfolco has joined #openstack-infra | 04:34 | |
| ianw | ok, already addressed https://review.openstack.org/#/c/628984/ | 04:35 |
| *** yamamoto has quit IRC | 04:36 | |
| *** rf0lc0 has joined #openstack-infra | 04:37 | |
| *** rfolco has quit IRC | 04:39 | |
| *** wolverineav has joined #openstack-infra | 04:40 | |
| *** rf0lc0 has quit IRC | 04:40 | |
| *** rfolco has joined #openstack-infra | 04:41 | |
| *** bhavikdbavishi has joined #openstack-infra | 04:44 | |
| *** dims has quit IRC | 04:47 | |
| *** dims has joined #openstack-infra | 04:48 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: source-repositories: Replace documentation http with https links https://review.openstack.org/619699 | 04:50 |
| clarkb | ianw: any chance the tripleo ssh failures in limestone make sense to you? | 04:54 |
| *** dims has quit IRC | 04:56 | |
| clarkb | hrm my ssh key isn't loaded on this machine maybe I will call it a night | 04:56 |
| *** dims has joined #openstack-infra | 04:56 | |
| *** tmorin has joined #openstack-infra | 05:05 | |
| *** tmorin has quit IRC | 05:08 | |
| *** spsurya has quit IRC | 05:10 | |
| *** spsurya has joined #openstack-infra | 05:13 | |
| openstackgerrit | Merged openstack/diskimage-builder master: source-repositories: Replace documentation http with https links https://review.openstack.org/619699 | 05:16 |
| *** bobh_ has joined #openstack-infra | 05:19 | |
| *** ykarel has quit IRC | 05:24 | |
| *** bobh_ has quit IRC | 05:26 | |
| *** ykarel has joined #openstack-infra | 05:51 | |
| *** hwoarang has joined #openstack-infra | 05:54 | |
| *** hwoarang_ has quit IRC | 05:57 | |
| *** rkukura has joined #openstack-infra | 05:57 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/629095 | 06:03 |
| *** hwoarang_ has joined #openstack-infra | 06:08 | |
| *** hwoarang has quit IRC | 06:11 | |
| *** kjackal has joined #openstack-infra | 06:17 | |
| *** agopi__ has joined #openstack-infra | 06:23 | |
| *** gyee has quit IRC | 06:24 | |
| openstackgerrit | weiyuanjun proposed openstack/diskimage-builder master: update spelling errors https://review.openstack.org/629096 | 06:31 |
| *** agopi__ has quit IRC | 06:33 | |
| *** agopi__ has joined #openstack-infra | 06:34 | |
| *** agopi__ has quit IRC | 06:36 | |
| *** yamamoto has joined #openstack-infra | 06:37 | |
| *** yamamoto has quit IRC | 06:42 | |
| *** sshnaidm has quit IRC | 06:52 | |
| *** eumel8 has quit IRC | 06:53 | |
| *** rcernin has quit IRC | 06:58 | |
| *** quiquell|off is now known as quiquell | 07:09 | |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/629095 | 07:16 |
| *** tmorin has joined #openstack-infra | 07:29 | |
| *** pgaxatte has joined #openstack-infra | 07:33 | |
| ianw | AJaeger: yeah, dib gate is blocked until 628984 | 07:35 |
| *** pcaruana has joined #openstack-infra | 07:42 | |
| *** jbadiapa has joined #openstack-infra | 07:43 | |
| *** quiquell is now known as quiquell|brb | 07:44 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul-jobs master: upload-pypi: add option to register packages https://review.openstack.org/629018 | 07:49 |
| *** slaweq has joined #openstack-infra | 07:50 | |
| AJaeger | ianw: ah - thanks! | 07:50 |
| *** wolverineav has quit IRC | 07:51 | |
| *** kjackal has quit IRC | 07:51 | |
| *** ykarel is now known as ykarel|lunch | 07:51 | |
| *** slaweq has quit IRC | 07:53 | |
| *** slaweq has joined #openstack-infra | 07:55 | |
| *** dkehn has quit IRC | 07:58 | |
| *** bobh_ has joined #openstack-infra | 08:02 | |
| *** rpittau has joined #openstack-infra | 08:05 | |
| *** JpMaxMan has quit IRC | 08:05 | |
| *** mwhahaha has quit IRC | 08:05 | |
| *** jungleboyj has quit IRC | 08:06 | |
| *** jungleboyj has joined #openstack-infra | 08:06 | |
| *** JpMaxMan has joined #openstack-infra | 08:06 | |
| *** mwhahaha has joined #openstack-infra | 08:06 | |
| *** bobh_ has quit IRC | 08:07 | |
| *** ccamacho has joined #openstack-infra | 08:09 | |
| *** ginopc has joined #openstack-infra | 08:10 | |
| *** quiquell|brb is now known as quiquell | 08:11 | |
| *** priteau has joined #openstack-infra | 08:12 | |
| *** ssbarnea has joined #openstack-infra | 08:14 | |
| *** ssbarnea|bkp2 has quit IRC | 08:16 | |
| *** e0ne has joined #openstack-infra | 08:16 | |
| *** aojea has joined #openstack-infra | 08:16 | |
| *** shardy has quit IRC | 08:19 | |
| *** shardy has joined #openstack-infra | 08:20 | |
| *** kjackal has joined #openstack-infra | 08:23 | |
| *** xek has joined #openstack-infra | 08:24 | |
| *** sshnaidm has joined #openstack-infra | 08:31 | |
| *** pcrews has quit IRC | 08:34 | |
| *** tosky has joined #openstack-infra | 08:37 | |
| *** sshnaidm has quit IRC | 08:47 | |
| *** evrardjp_ has joined #openstack-infra | 08:48 | |
| *** evrardjp__ has joined #openstack-infra | 08:49 | |
| *** jpena|off is now known as jpena | 08:50 | |
| *** wolverineav has joined #openstack-infra | 08:51 | |
| *** evrardjp has quit IRC | 08:51 | |
| *** evrardjp_ has quit IRC | 08:52 | |
| *** evrardjp__ has quit IRC | 08:53 | |
| *** wolverineav has quit IRC | 08:55 | |
| *** gfidente has joined #openstack-infra | 09:00 | |
| *** evrardjp has joined #openstack-infra | 09:03 | |
| *** jpich has joined #openstack-infra | 09:03 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/git-review master: Switch from testrepository to stestr https://review.openstack.org/628655 | 09:12 |
| *** bobh_ has joined #openstack-infra | 09:15 | |
| *** tmorin has quit IRC | 09:19 | |
| *** rpittau has quit IRC | 09:19 | |
| *** rpittau has joined #openstack-infra | 09:19 | |
| *** tmorin has joined #openstack-infra | 09:19 | |
| *** dtantsur|afk is now known as dtantsur | 09:21 | |
| *** ykarel|lunch is now known as ykarel | 09:22 | |
| *** tmorin has quit IRC | 09:25 | |
| *** apetrich has joined #openstack-infra | 09:26 | |
| openstackgerrit | Will Szumski proposed openstack-dev/pbr master: Do not globally replace path prefix https://review.openstack.org/629006 | 09:28 |
| *** derekh has joined #openstack-infra | 09:33 | |
| *** rpittau has quit IRC | 09:49 | |
| *** Miouge has left #openstack-infra | 09:50 | |
| *** rpittau has joined #openstack-infra | 09:51 | |
| *** sshnaidm has joined #openstack-infra | 09:55 | |
| *** priteau has quit IRC | 10:17 | |
| *** bobh_ has quit IRC | 10:19 | |
| *** kjackal has quit IRC | 10:24 | |
| *** kjackal has joined #openstack-infra | 10:24 | |
| *** bhavikdbavishi has quit IRC | 10:25 | |
| *** lpetrut has joined #openstack-infra | 10:26 | |
| *** rpittau has quit IRC | 10:34 | |
| *** rpittau has joined #openstack-infra | 10:35 | |
| frickler | prometheanfire: seems the dib gentoo builds are failing due to some python dependency stuff, if you want to take a look http://logs.openstack.org/31/618431/2/check/dib-functests-bionic-python3-extras/8fe89c0/logs/gentoo_build-succeeds.FAIL.log | 10:39 |
| *** zhangfei has quit IRC | 10:39 | |
| prometheanfire | looking | 10:42 |
| prometheanfire | frickler: I'll ask if we changed our default python | 10:43 |
| prometheanfire | PYTHON_SINGLE_TARGET="python3_6 -python3_5 (-python3_7)" | 10:43 |
| prometheanfire | PYTHON_TARGETS="python3_5 -python3_6 (-python3_7)" | 10:43 |
| prometheanfire | those should be the same | 10:43 |
| prometheanfire | there are vars (that should be set) that control it | 10:43 |
| *** priteau has joined #openstack-infra | 10:44 | |
| prometheanfire | frickler: started happening around the 23rd? | 10:51 |
| frickler | prometheanfire: I didn't check the logs, just noticed this for a current patch | 10:52 |
| frickler | prometheanfire: seems we try to pin to python3.5 here https://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/elements/gentoo/environment.d/00-gentoo-envars.bash#L5 | 10:52 |
| *** kjackal has quit IRC | 10:52 | |
| prometheanfire | project-config/nodepool/nodepool.yaml | 10:53 |
| prometheanfire | overrides it | 10:53 |
| *** jpich has quit IRC | 10:53 | |
| frickler | prometheanfire: the job seems to have been failing since beginning of december, but that may have other reasons http://zuul.openstack.org/builds?job_name=dib-functests-bionic-python3-extras | 10:53 |
| *** jpich has joined #openstack-infra | 10:54 | |
| prometheanfire | think dev-util/glib-utils may need to be added to https://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/elements/gentoo/pre-install.d/02-gentoo-04-install-desired-python#L32 | 10:54 |
| frickler | prometheanfire: the override seems to be for our actual infra build only, not for the dib func tests? | 10:54 |
| prometheanfire | I'll build a test image today and see what's going on | 10:54 |
| prometheanfire | frickler: oh, this is the dib build? | 10:55 |
| prometheanfire | dib bug not infra bug? | 10:55 |
| prometheanfire | in that case, yes, we should just change the default | 10:55 |
| frickler | prometheanfire: these are the test jobs for a dib patch, yes | 10:55 |
| prometheanfire | sorry, infra channel :P | 10:56 |
| *** udesale has quit IRC | 10:57 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: change to python36 for gentoo https://review.openstack.org/629145 | 11:00 |
| *** yamamoto has joined #openstack-infra | 11:02 | |
| prometheanfire | we'll see if that works | 11:02 |
| *** ccamacho has quit IRC | 11:03 | |
| *** rpittau has quit IRC | 11:04 | |
| *** rpittau has joined #openstack-infra | 11:05 | |
| *** pbourke has quit IRC | 11:08 | |
| *** pbourke has joined #openstack-infra | 11:10 | |
| *** yamamoto has quit IRC | 11:11 | |
| *** roman_g has joined #openstack-infra | 11:17 | |
| *** rpittau has quit IRC | 11:20 | |
| *** rpittau has joined #openstack-infra | 11:20 | |
| *** yamamoto has joined #openstack-infra | 11:23 | |
| *** lpetrut has quit IRC | 11:26 | |
| *** yamamoto has quit IRC | 11:29 | |
| frickler | prometheanfire: thanks, that looks o.k. for gentoo now, the next failure seems to be from debian: http://logs.openstack.org/45/629145/1/check/dib-functests-bionic-python3-extras/f068bb4/logs/ | 11:32 |
| *** ccamacho has joined #openstack-infra | 11:33 | |
| *** rpittau has quit IRC | 11:35 | |
| *** rpittau has joined #openstack-infra | 11:36 | |
| *** kjackal has joined #openstack-infra | 11:37 | |
| tosky | AJaeger: hi, do you know why the gerritbot regexp that I introduced in http://git.openstack.org/cgit/openstack-infra/project-config/commit/gerritbot/channels.yaml?id=09e2a307f00ea272e6664d3f4cae79effc5609d2 to catch all sahara repositories is not working? | 11:47 |
| frickler | tosky: I guess it will need a "^" at the beginning to mark it as regex, like here http://git.openstack.org/cgit/openstack-infra/project-config/tree/gerritbot/channels.yaml?id=09e2a307f00ea272e6664d3f4cae79effc5609d2#n7 | 11:49 |
| *** tellesnobrega has joined #openstack-infra | 11:50 | |
| tosky | oh | 11:51 |
| tosky | I see, thanks | 11:51 |
| *** dkehn has joined #openstack-infra | 11:52 | |
| *** bhavikdbavishi has joined #openstack-infra | 11:55 | |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: sahara: basic jobs for the new repos, fix IRC notifications https://review.openstack.org/629068 | 11:57 |
| tosky | recycling reviews | 11:57 |
| *** yamamoto has joined #openstack-infra | 12:00 | |
| *** d0ugal has quit IRC | 12:01 | |
| *** rh-jelabarre has joined #openstack-infra | 12:01 | |
| *** rpittau is now known as rpittau|lunch | 12:03 | |
| *** rpittau|lunch has quit IRC | 12:05 | |
| *** rpittau|lunch has joined #openstack-infra | 12:06 | |
| *** tellesnobrega has left #openstack-infra | 12:06 | |
| *** gfidente has quit IRC | 12:11 | |
| *** gfidente has joined #openstack-infra | 12:13 | |
| openstackgerrit | Will Szumski proposed openstack-dev/pbr master: Fix white space handling in file names https://review.openstack.org/629161 | 12:13 |
| *** rpittau|lunch has quit IRC | 12:20 | |
| *** ccamacho has quit IRC | 12:20 | |
| *** rpittau|lunch has joined #openstack-infra | 12:20 | |
| *** d0ugal has joined #openstack-infra | 12:22 | |
| *** wolverineav has joined #openstack-infra | 12:28 | |
| *** wolverineav has quit IRC | 12:32 | |
| *** rpittau|lunch has quit IRC | 12:35 | |
| *** jpena is now known as jpena|lunch | 12:37 | |
| *** rpittau|lunch has joined #openstack-infra | 12:37 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/openstack-zuul-jobs master: Implement workaround for ansible unmount bug https://review.openstack.org/629169 | 12:47 |
| ssbarnea | AJaeger: clarkb fungi ^^ | 12:49 |
| *** ccamacho has joined #openstack-infra | 12:53 | |
| *** ccamacho has quit IRC | 12:54 | |
| *** ccamacho has joined #openstack-infra | 12:54 | |
| mordred | ssbarnea: fascinating | 12:58 |
| ssbarnea | mordred: indeed, what can I say.... still this could have being spotted with an idempotency test (something that molecule can do) | 12:59 |
| ssbarnea | we found it by accident by running the role twice | 12:59 |
| *** bhavikdbavishi has quit IRC | 13:00 | |
| ssbarnea | as it was imported from another place. | 13:00 |
| *** yamamoto has quit IRC | 13:00 | |
| ssbarnea | i guess we have to use workaround and remove it once the fix reaches our version of ansible, right? | 13:00 |
| *** rpittau|lunch is now known as rpittau | 13:01 | |
| *** bhavikdbavishi has joined #openstack-infra | 13:01 | |
| *** yamamoto has joined #openstack-infra | 13:02 | |
| mordred | yah. although it seems the workaround isn't super terrible, thankfully | 13:02 |
| ssbarnea | mordred: can i use depends-on to test it? | 13:03 |
| *** yamamoto has quit IRC | 13:04 | |
| *** rpittau has quit IRC | 13:06 | |
| *** rpittau has joined #openstack-infra | 13:06 | |
| *** rpittau has quit IRC | 13:07 | |
| *** szaher has joined #openstack-infra | 13:08 | |
| *** rpittau has joined #openstack-infra | 13:08 | |
| mordred | ssbarnea: yes, I believe so | 13:11 |
| openstackgerrit | Luigi Toscano proposed openstack-infra/project-config master: sahara: basic jobs for the new repos, fix IRC notifications https://review.openstack.org/629068 | 13:14 |
| *** trown|outtypewww is now known as trown | 13:17 | |
| *** rascasoft has joined #openstack-infra | 13:19 | |
| *** kgiusti has joined #openstack-infra | 13:19 | |
| *** udesale has joined #openstack-infra | 13:20 | |
| *** agopi has joined #openstack-infra | 13:24 | |
| *** zigo has joined #openstack-infra | 13:30 | |
| *** bhavikdbavishi has quit IRC | 13:32 | |
| *** psachin has quit IRC | 13:33 | |
| *** jpena|lunch is now known as jpena | 13:34 | |
| *** ykarel is now known as ykarel|away | 13:35 | |
| *** jamesmcarthur has joined #openstack-infra | 13:42 | |
| *** jhesketh has quit IRC | 13:45 | |
| *** szaher has quit IRC | 13:47 | |
| *** jhesketh has joined #openstack-infra | 13:47 | |
| *** agopi has quit IRC | 13:48 | |
| openstackgerrit | lei zhang proposed openstack-infra/storyboard master: word highlight https://review.openstack.org/629189 | 13:49 |
| *** rpittau has quit IRC | 13:52 | |
| *** rpittau has joined #openstack-infra | 13:52 | |
| *** szaher has joined #openstack-infra | 13:52 | |
| *** rpittau_ has joined #openstack-infra | 13:55 | |
| *** rpittau has quit IRC | 13:57 | |
| *** rpittau_ is now known as rpittau | 13:57 | |
| *** d0ugal has quit IRC | 14:02 | |
| *** sthussey has joined #openstack-infra | 14:02 | |
| *** d0ugal has joined #openstack-infra | 14:04 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Use template for lower-constraints https://review.openstack.org/626745 | 14:04 |
| *** iurygregory has joined #openstack-infra | 14:06 | |
| *** jento has quit IRC | 14:07 | |
| iurygregory | Hello everyone, in ironic we are renaming the jobs to match the zuulv3 pattern https://review.openstack.org/#/c/629173/, the definition of the job is in the ironic repository but according to the error i got i should change in https://github.com/openstack/ironic-tempest-plugin/blob/master/zuul.d/project.yaml#L23, any ideas on how i should rename the job in both repositories? | 14:11 |
| *** zul has joined #openstack-infra | 14:12 | |
| *** d0ugal has quit IRC | 14:12 | |
| frickler | iurygregory: probably you need three steps: 1. create jobs with new names as copy of the existing ones, 2. rename jobs in ironic-tempest-plugin, 3. remove old jobs | 14:13 |
| frickler | iurygregory: or drop jobs from ironic-tempest-plugin while doing the rename | 14:13 |
| iurygregory | frickler, tks o/ | 14:13 |
| *** boden has joined #openstack-infra | 14:14 | |
| *** bobh_ has joined #openstack-infra | 14:16 | |
| *** bobh_ has quit IRC | 14:21 | |
| *** ykarel|away has quit IRC | 14:21 | |
| *** jcoufal has joined #openstack-infra | 14:27 | |
| *** jamesmcarthur has quit IRC | 14:28 | |
| ssbarnea | clarkb: take a look at http://logs.openstack.org/74/611574/8/check/barbican-dogtag-devstack-functional-fedora-latest/12945cd/job-output.txt.gz#_2019-01-08_07_49_43_581105 -- i suspect this may be caused by ansible.cfg not being loaded due to permissions. | 14:30 |
| ssbarnea | the job is broken for more than a month, found it while trying to merge unrelated change. | 14:31 |
| *** priteau has quit IRC | 14:33 | |
| *** fuentess has joined #openstack-infra | 14:35 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Implement workaround for ansible unmount bug https://review.openstack.org/629169 | 14:35 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/devstack-gate master: Fix ansible.cfg loading by assuring o-w permissions https://review.openstack.org/629197 | 14:40 |
| *** shardy has quit IRC | 14:41 | |
| *** hrubi has quit IRC | 14:43 | |
| *** eernst has joined #openstack-infra | 14:43 | |
| *** zul has quit IRC | 14:44 | |
| *** jamesmcarthur has joined #openstack-infra | 14:48 | |
| *** jamesmcarthur has quit IRC | 14:50 | |
| *** jamesmcarthur has joined #openstack-infra | 14:50 | |
| *** shardy has joined #openstack-infra | 14:51 | |
| *** shardy has quit IRC | 14:51 | |
| *** shardy has joined #openstack-infra | 14:51 | |
| openstackgerrit | Merged openstack/diskimage-builder master: change to python36 for gentoo https://review.openstack.org/629145 | 14:59 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Default ssh keys to empty list https://review.openstack.org/629201 | 14:59 |
| openstackgerrit | Will Szumski proposed openstack-dev/pbr master: Fix white space handling in file names https://review.openstack.org/629161 | 15:04 |
| *** ykarel|away has joined #openstack-infra | 15:04 | |
| *** eernst has quit IRC | 15:05 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/devstack-gate master: Replace deprecated ansible --sudo with --become https://review.openstack.org/629205 | 15:06 |
| *** ykarel|away is now known as ykarel | 15:09 | |
| *** bhavikdbavishi has joined #openstack-infra | 15:13 | |
| *** eernst has joined #openstack-infra | 15:14 | |
| *** smarcet has joined #openstack-infra | 15:19 | |
| *** ramishra has quit IRC | 15:20 | |
| *** smarcet has quit IRC | 15:21 | |
| *** ykarel is now known as ykarel|away | 15:21 | |
| *** markvoelker has joined #openstack-infra | 15:23 | |
| *** smarcet has joined #openstack-infra | 15:25 | |
| *** markvoelker has quit IRC | 15:26 | |
| *** jento has joined #openstack-infra | 15:31 | |
| *** armax has joined #openstack-infra | 15:41 | |
| fungi | #status log restarted bind9 service on adns1.opendev.org in order to get it to properly recognize and re-sign the updated zuulci.org zone | 15:43 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstack-zuul-jobs master: Updated laravel jobs to include php7 repo https://review.openstack.org/629035 | 15:43 |
| openstackstatus | fungi: finished logging | 15:43 |
| fungi | corvus: clarkb: ^ that seemed to do the trick | 15:43 |
| fungi | hopefully it'll maintain the signed zone for real from now on | 15:44 |
| smarcet | fungi:mordred: morning when u have some free time please review https://review.openstack.org/629035 thx ! | 15:47 |
| *** eernst has quit IRC | 15:50 | |
| *** eernst has joined #openstack-infra | 15:51 | |
| openstackgerrit | Merged openstack-infra/zuul master: Default ssh keys to empty list https://review.openstack.org/629201 | 15:53 |
| *** shardy has quit IRC | 15:54 | |
| *** e0ne has quit IRC | 15:56 | |
| fungi | corvus: for the other domains, have we been providing just the sha2-256 (digest type #2) ds resource record to the registrar, or including a version with sha-1 (digest type #1) as well? | 15:58 |
| corvus | fungi: i *think* sha2 | 15:59 |
| *** bobh has joined #openstack-infra | 15:59 | |
| *** hamerins has joined #openstack-infra | 16:00 | |
| fungi | thanks | 16:00 |
| fungi | https://docs.openstack.org/infra/system-config/dns.html#adding-a-zone seems to imply that | 16:00 |
| fungi | i guess with rfc 4509 being almost 13 years old now, that's probably sufficient | 16:02 |
| *** udesale has quit IRC | 16:02 | |
| fungi | odds are almost nobody had dnssec implemented prior to 2006 anyway, and so basically everyone who supports dnssec supports sha2-256 digests | 16:03 |
| *** wolverineav has joined #openstack-infra | 16:04 | |
| *** bobh has quit IRC | 16:06 | |
| *** szaher has quit IRC | 16:08 | |
| *** wolverineav has quit IRC | 16:08 | |
| *** szaher has joined #openstack-infra | 16:09 | |
| corvus | fungi: and even csc didn't have a problem with it :) | 16:10 |
| fungi | pure insanity! | 16:11 |
| *** gfidente has quit IRC | 16:11 | |
| *** gfidente has joined #openstack-infra | 16:12 | |
| fungi | anyway, i've passed the ds rr for zuulci.org along to jamesmcarthur now that the opendev.org nameservers are serving the latest version of the zone (not that it's in any way urgent) | 16:12 |
| clarkb | fungi: do you want to wait on working dnssec before removing the old servers? or are we ready to clean those up nowish? | 16:13 |
| fungi | i think it should be safe to tear them down now | 16:13 |
| * fungi was actually just about to say that ;) | 16:13 | |
| *** ykarel|away has quit IRC | 16:17 | |
| *** ykarel|away has joined #openstack-infra | 16:17 | |
| *** diablo_rojo has joined #openstack-infra | 16:18 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Updated laravel jobs to include php7 repo https://review.openstack.org/629035 | 16:20 |
| *** pcaruana has quit IRC | 16:20 | |
| AJaeger | jd_: we need PTL +1 on https://review.openstack.org/619235 (or -1 ;), could you review, please? | 16:21 |
| *** e0ne has joined #openstack-infra | 16:25 | |
| fungi | jd_: and same for https://review.openstack.org/#/c/619538https://review.openstack.org/619538 | 16:26 |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/devstack-gate master: Fix ansible.cfg loading by assuring o-w permissions https://review.openstack.org/629197 | 16:29 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Nameservers are now managed with ansible https://review.openstack.org/629257 | 16:32 |
| clarkb | fungi: corvus ^ thats the config mgmt side of the cleanup | 16:32 |
| *** ccamacho has quit IRC | 16:38 | |
| *** gyee has joined #openstack-infra | 16:38 | |
| spotz | fungi are you or jimmy in charge of the new discuss list? | 16:39 |
| corvus | clarkb, fungi, mordred: what's the command to create a venv on bridge again? | 16:39 |
| mordred | corvus: python3 -m virtualenv --python=python3 foo is what I've been using | 16:39 |
| clarkb | corvus: python3 /usr/lib/python3/dist-packages/virtualenv.py --python=python3 ~/launch-env | 16:40 |
| clarkb | mordred's is less typing | 16:40 |
| corvus | thx; i found "-m venv" seaching online but that is not happy | 16:40 |
| *** zaro has quit IRC | 16:41 | |
| corvus | -m virtualenv works | 16:41 |
| fungi | spotz: if you mean moderator-slave, that's me. see the bottom of http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss | 16:43 |
| spotz | fungi: Just wondering if we might need to send a reminder out to folks some how. I had subbed when it was first announced but hadn't been receiving anything. I resubbed this morning and mail is flowing again | 16:47 |
| fungi | spotz: yeah, not sure who to remind? but i can take a look at the subscription log and see what the deal was with yours | 16:49 |
| spotz | fungi: It might have been me, but if someone hadn't mentioned sending something to the list I woudn't have checked cause I get mail from the user and foundation list and it's just after the holidays | 16:50 |
| *** imacdonn has quit IRC | 16:50 | |
| *** imacdonn has joined #openstack-infra | 16:51 | |
| corvus | clarkb, fungi, mordred: have you seen this error? http://paste.openstack.org/show/740681/ | 16:51 |
| mordred | corvus: never | 16:52 |
| clarkb | I haven't seen it either | 16:52 |
| *** jpich has quit IRC | 16:52 | |
| fungi | spotz: looks like you tried to subscribe on september 20 but you never followed up on the subscription confirmation reply | 16:53 |
| corvus | dhellmann: ^ have you seen http://paste.openstack.org/show/740681/ ? | 16:53 |
| *** shardy has joined #openstack-infra | 16:53 | |
| clarkb | fwiw my two limestone instances can still ping each other | 16:54 |
| clarkb | I think my next step is to delete one of them and replace it in a loop to see if I can make a node that is unpingable | 16:54 |
| clarkb | (thats sort of a stab in the dark though) | 16:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Run k8s-on-openstack to manage k8s control plane https://review.openstack.org/626965 | 16:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add resources for deploying rook and xtradb to kuberenets https://review.openstack.org/626054 | 16:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Increase number of k8s nodes to 4 https://review.openstack.org/628031 | 16:56 |
| jd_ | fungi: AJaeger: I asked him to just create the repo but i don't think it fits in Telemetry, he's chatting with dhellmann too | 16:57 |
| fungi | corvus: yeah, i think debian split the venv module out of the stdlib and put it in a separate python3-venv package, so ubuntu presumably inherited that design choice | 16:58 |
| *** ginopc has quit IRC | 16:58 | |
| *** rpittau has quit IRC | 16:59 | |
| corvus | infra-root: i need to change the vexxhost openstackci password | 16:59 |
| fungi | corvus: thanks for the heads up | 16:59 |
| clarkb | if you need working osc the one in my homedir on bridge does work as of yesterday booting limestone instances | 16:59 |
| spotz | fungi: id10t error:) | 17:00 |
| fungi | spotz: i make at least six of those every morning before breakfast | 17:00 |
| spotz | fungi: hehe, I mainly ride horses before breakfast so HOPE I don't do to many of those that early! | 17:01 |
| corvus | #status log changed vexxhost openstackci password | 17:03 |
| openstackstatus | corvus: finished logging | 17:03 |
| corvus | infra-root: password is updated in vexxhost, ansible host vars, and password storage | 17:03 |
| corvus | i'm guessing nodepool may have errors for the next little while | 17:03 |
| corvus | (also, i suspect this would have been very difficult to accomplish without the static inventory, so yay for that) | 17:04 |
| mordred | corvus: yay! | 17:05 |
| mordred | corvus: I'm going to manually update /etc/openstack/clouds.yaml on bridge with the udpated password | 17:05 |
| *** pgaxatte has quit IRC | 17:05 | |
| mordred | corvus: the new password works - I'm re-running the k8s ansible deploy which uses it from clouds.yaml as the first step | 17:07 |
| corvus | clarkb: for my sanity, can you use your venv to get a server listing in vexxhost sjc1? | 17:07 |
| *** e0ne has quit IRC | 17:08 | |
| dhellmann | corvus : that error is related to a new cmd2 release, I think? https://bugs.launchpad.net/python-cliff/+bug/1810213 | 17:10 |
| openstack | Launchpad bug 1810213 in cliff "EXCEPTION of type 'FrozenInstanceError' occurred with message: ''" [Undecided,Triaged] | 17:10 |
| *** shardy has quit IRC | 17:10 | |
| clarkb | corvus: it fails on auth. I'm using the all-clouds.yaml file though, let me try using clouds.yaml | 17:11 |
| mordred | clarkb: yah - I only uipdated clouds.yaml | 17:11 |
| clarkb | clouds.yaml works with my osc | 17:11 |
| dhellmann | AJaeger , jd_ , fungi : we should probably decouple the work of creating the repo(s) from the governance change so they can get on with their work. If the repo isn't going to be hosted under Telemetry, they have to decide whether to go through the trouble of creating their own team. | 17:13 |
| dhellmann | AJaeger , jd_ , fungi : I left a comment about that, as well as the repo name | 17:15 |
| *** jamesmcarthur has quit IRC | 17:15 | |
| corvus | dhellmann, clarkb: using clarkb's old venv, i can run "openstack [...] server list" from the command line and it works, but running "openstack [...]" and then typing "server list" from the prompt gets me a "ValueError" (though i note that's different from the FrozenInstanceError) | 17:15 |
| dhellmann | corvus : yes, the logic for interactive mode and cli mode is a little different, and the bug only comes up in interactive mode | 17:16 |
| dhellmann | the cmd2 folks rewrote a bunch of stuff a while back and we're still finding corner cases in cliff | 17:16 |
| dhellmann | I'm not sure if downgrading cmd2 actually fixes the problem, or if it's just that no one has noticed until recently | 17:17 |
| corvus | is there a way to get a working interactive mode? i guess maybe i can try to pip freeze my months-old venv? | 17:17 |
| dhellmann | I'm not sure. | 17:17 |
| dhellmann | I don't think the fix is that hard, but I haven't had time to try my theory (we need to create a new Statement instead of trying to modify the one we're given) | 17:18 |
| dhellmann | they use a library that tries to make the object immutable, because it's subclassed from str | 17:19 |
| *** jamesmcarthur has joined #openstack-infra | 17:19 | |
| *** quiquell is now known as quiquell|off | 17:19 | |
| jamesmcarthur | fungi: ds record for zuulci.org has been added | 17:20 |
| clarkb | mordred: your hold on 0001566885 please don't delete that | 17:22 |
| clarkb | mordred: it may have also caught an instance of the ssh connectivity issues in limestone | 17:23 |
| corvus | dhellmann: are you saying that 'interactive usage of the openstack client' has been broken for quite some time? | 17:23 |
| clarkb | logan-: mordred: limestone instance 6b6916a6-2dd2-4a57-b8bf-432c15fc6818 running on hypervisor f1aa0af441eac757d5d6ad886e8a4724eebf11ed0ff2e56260217915 is reachable via global ipv6 (I can ssh in that way) but it cannot ping its neighbors on the private 10.4.70.0/24 network and its neighbors cannot ping it | 17:24 |
| clarkb | logan-: mordred: there is a good chance that the job failed due to this (though I'm not positive) so it would liekly be helpful if we can debug the 10 net connectivity issue without disturbing the host too much (I'm not sure what sdk thing mordred was trying to debug) | 17:25 |
| mordred | clarkb: neat! so - I don't actually need it for the thing I was trying to debug - feel free to take it over as much as you want | 17:25 |
| logan- | awesome | 17:25 |
| clarkb | mordred: thanks | 17:25 |
| corvus | dhellmann: fwiw, "pip install python-openstackclient; pip install cmd2==0.8.9" seems to work | 17:26 |
| clarkb | logan-: if you want me to add your ssh key to the instance I can do that too (if it helps with debugging). THe interfaces on the host look fine to me (and working ipv6 would seem to agree) | 17:26 |
| logan- | that would be great, thank you. https://github.com/logan2211.keys | 17:26 |
| mordred | corvus, dhellmann: we have a <0.9 exclsion in global-requirements for python2 but not for python3 - perhaps we shoudl just make the exclusion apply to both? | 17:27 |
| corvus | oh, so if i were running under python2, i wouldn't have seen this? | 17:28 |
| mordred | yeah | 17:28 |
| clarkb | logan-: 2607:ff68:100:54:f816:3eff:fecf:7c2a is ipv6 addr and 10.4.70.19 is private ipv4 addr that seems to not work. Your keys should be in place | 17:28 |
| logan- | i'm in, thanks | 17:29 |
| clarkb | fwiw I started to do ping tests against all current 10.4.70 addrs listed by nova server list from my test nodes. 10.4.70.19 showed up as failing and the host wasn't deleted so I looekd further and found it was held by mordred | 17:29 |
| mordred | corvus, dhellmann: remote: https://review.openstack.org/629269 Apply cmd2 upper bound to python3 | 17:29 |
| corvus | mordred, dhellmann: yeah, so it seems like adding that exclusion to py3 is a good idea, because i think right now, folks using osc with python3 are probably having a very bad experience :) | 17:29 |
| clarkb | I mostly stumbled onto it, so don't have a good "this is how to debug this in the future" answer | 17:29 |
| clarkb | logan-: another data point, is the host can ping its gateway, 10.4.70.1 but not 10.4.70.83 (my test node) | 17:32 |
| *** jamesmcarthur has quit IRC | 17:32 | |
| logan- | yeah it is weird, i can ping some test nodes from it, but some fail. v4 internet gateway works. it seems like maybe a l2pop issue. | 17:33 |
| *** kaiokmo has quit IRC | 17:33 | |
| *** elod has quit IRC | 17:34 | |
| logan- | arp -n shows that it is not receiving arp responses for some nodes when pinging at random | 17:35 |
| openstackgerrit | Will Szumski proposed openstack-dev/pbr master: Fix white space handling in file names https://review.openstack.org/629161 | 17:35 |
| *** dustinc has joined #openstack-infra | 17:36 | |
| clarkb | arp problems could explain why it was seen on a single hypervisor too | 17:37 |
| clarkb | and ipv6 doesn't arp | 17:37 |
| *** jamesmcarthur has joined #openstack-infra | 17:45 | |
| *** elod has joined #openstack-infra | 17:46 | |
| *** trown is now known as trown|lunch | 17:48 | |
| logan- | it seems like linuxbridge agent started before the networking was fully up on a couple servers. you can see that here http://paste.openstack.org/raw/740685/ on 6125 and 6116. they look normal after a linuxbridge agent restart http://paste.openstack.org/raw/740686/ and i haven't found any nodes that fail to ping | 17:49 |
| fungi | clarkb: well, v6 has neighbor discovery, which is sort of like arp | 17:52 |
| fungi | jamesmcarthur: thanks!!! very fast again | 17:52 |
| logan- | it doesn't really explain why we would have seen failures between nodes on the same host though, like we did yesterday | 17:53 |
| *** jamesmcarthur has quit IRC | 17:56 | |
| clarkb | (thinking out loud here) does linuxbrdige agent need to bind to an interface in order to curate l2pop or arp type data? | 17:57 |
| fungi | unfortunately i don't know enough of how that's configured to form any conjecture over what could cause layer 2 connectivity issues between two instances on the same host. are they connected via simple linux kernel bridge(8) devices? | 17:57 |
| clarkb | if that is the case it could potentailly still be an issue on the same host? | 17:57 |
| clarkb | fungi: I think part of the issue here is that neutron doesn't use arp | 17:57 |
| clarkb | it uses l2pop | 17:57 |
| logan- | i'm not certain on the cause, its possible restarting linuxbridge-agent on those nodes caused it to resync the bridge table and wallpaper over the issue. it could just be randomly failing to setup ports properly | 17:58 |
| clarkb | logan-: we have an e-r query to track it at least. If you think you've restarted all the necessary agents we should either see the problem go away if that was it or continue to see it later if not | 17:58 |
| *** ykarel|away has quit IRC | 17:58 | |
| logan- | but i have seen issues in the past with linuxbridge forwarding issues when the agent starts before networking isnt fully started and it fails to bind the physical interface to its bridge. on 2nd though that doesnt really seem to be the case here because iirc you would see _no_ connectivity to those nodes in that case | 17:59 |
| *** derekh has quit IRC | 18:00 | |
| logan- | ok sounds good, guess we'll see what happens. thanks for grabbing that node | 18:00 |
| clarkb | freedesktop.org has a whole docuemnt on how to fix ^ with systemd units fwiw | 18:00 |
| clarkb | https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ | 18:00 |
| clarkb | (a lot of things get this wrong, even in official distro packaging) | 18:01 |
| dhellmann | mordred , corvus : after the global rule lands we may want to add rules to osc itself, too. | 18:02 |
| *** jamesmcarthur has joined #openstack-infra | 18:02 | |
| logan- | yep -- the startup issue I think is a bug in OSA with the service unit where it uses After=network.target instead of After=network-online.target | 18:03 |
| mordred | dhellmann: ++ | 18:05 |
| dhellmann | mordred , corvus : or maybe cliff, I guess. either way, if one of you does that I can +2 | 18:05 |
| fungi | logan-: does indeed sound racu | 18:05 |
| fungi | racy | 18:06 |
| mordred | dhellmann: I'll get a patch up | 18:06 |
| clarkb | hrm we have 2 hours of meetings coming up shortly. /me is going to go find breakfast now | 18:06 |
| clarkb | logan-: ya you can get away with after network.target if you bind to 0.0.0.0 or :: or 127.0.0.1 or ::1 | 18:08 |
| clarkb | logan-: those addrs are special in linux and it handles them just fine, but if you need to bind to specific interfaces you need to wait for network-online.target to make thsoe available | 18:09 |
| *** hamerins has quit IRC | 18:09 | |
| *** hamerins has joined #openstack-infra | 18:11 | |
| *** whoami-rajat has quit IRC | 18:12 | |
| *** rkukura has quit IRC | 18:12 | |
| clarkb | hrm I don't think we index our zuul jobs ansible playbook/role test job logs | 18:14 |
| clarkb | and with that I really need to find food | 18:14 |
| *** rf0lc0 has joined #openstack-infra | 18:14 | |
| *** rfolco has quit IRC | 18:15 | |
| AJaeger | dhellmann, jd_ thanks for your comments on ceilometer-dashboard | 18:15 |
| *** rf0lc0 is now known as rfolco | 18:16 | |
| *** panda is now known as panda|off | 18:17 | |
| *** gfidente is now known as gfidente|afk | 18:17 | |
| *** Swami has joined #openstack-infra | 18:22 | |
| *** e0ne has joined #openstack-infra | 18:23 | |
| *** armax has quit IRC | 18:26 | |
| *** wolverineav has joined #openstack-infra | 18:29 | |
| *** jpena is now known as jpena|off | 18:30 | |
| *** wolverineav has quit IRC | 18:30 | |
| *** wolverineav has joined #openstack-infra | 18:31 | |
| *** rkukura has joined #openstack-infra | 18:43 | |
| *** dtantsur is now known as dtantsur|afk | 18:45 | |
| *** lbragsta_ has joined #openstack-infra | 18:52 | |
| *** lbragstad has quit IRC | 18:52 | |
| *** bhavikdbavishi has quit IRC | 18:52 | |
| *** trown|lunch is now known as trown | 18:55 | |
| *** lbragsta_ is now known as lbragstad | 18:55 | |
| clarkb | I'm cleaning up my test nodes in limestone now | 18:56 |
| *** elod has quit IRC | 18:59 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid master: Migration to PHP 7.x https://review.openstack.org/611936 | 18:59 |
| *** kaiokmo has joined #openstack-infra | 19:01 | |
| *** armax has joined #openstack-infra | 19:01 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid master: Migration to PHP 7.x https://review.openstack.org/611936 | 19:11 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid master: Migration to PHP 7.x https://review.openstack.org/611936 | 19:13 |
| *** elod has joined #openstack-infra | 19:21 | |
| *** gfidente|afk has quit IRC | 19:26 | |
| *** hamerins has quit IRC | 19:34 | |
| *** hamerins has joined #openstack-infra | 19:35 | |
| *** e0ne has quit IRC | 19:39 | |
| *** jamesmcarthur has quit IRC | 19:41 | |
| *** jamesmcarthur has joined #openstack-infra | 19:42 | |
| *** e0ne has joined #openstack-infra | 19:43 | |
| *** jamesmcarthur has quit IRC | 19:52 | |
| ianw | is there like a test number for pbx.openstack.org? | 19:54 |
| *** ccamacho has joined #openstack-infra | 19:54 | |
| *** jamesmcarthur has joined #openstack-infra | 19:55 | |
| fungi | 6000 is what we've mostly tested on | 19:56 |
| clarkb | I finally got it working on android fwiw. I had to use some third party sip client, put it bogus auth info, then let it fail and dial a specific sip url after that | 19:57 |
| clarkb | its weird that all the cleints assume authentication | 19:57 |
| ssbarnea | does anyone know a way avoid https://github.com/pypa/setuptools/issues/1543 ? --- making http_proxy work with pip/setuptools? | 19:57 |
| ianw | clarkb: is that csipsimple? | 19:58 |
| ianw | i know i have had that working once, when a ptg or something was bridged | 19:58 |
| clarkb | mizudroid? | 19:58 |
| *** ccamacho has quit IRC | 19:58 | |
| corvus | let's talk in #opendev about the k8s stuff | 20:00 |
| *** gary_perkins has quit IRC | 20:01 | |
| *** armax has quit IRC | 20:03 | |
| *** bobh has joined #openstack-infra | 20:03 | |
| *** gary_perkins has joined #openstack-infra | 20:06 | |
| *** bobh has quit IRC | 20:10 | |
| *** slittle1 has joined #openstack-infra | 20:16 | |
| slittle1 | Anyone know how to create a branch on an openstack hosted git ? | 20:17 |
| slittle1 | I've read https://docs.openstack.org/fuel-docs/latest/plugindocs/fuel-plugin-sdk-guide/create-environment/repository-branching/repository-branching-create.html | 20:18 |
| fungi | slittle1: sorry i haven't had a chance to reply to your e-mail yet | 20:18 |
| slittle1 | Neither procedure outlined on that web page is working | 20:19 |
| slittle1 | Are special privileges required ? | 20:19 |
| fungi | slittle1: yes, branch creation requires a specific acl rule | 20:19 |
| slittle1 | what is the procedure to set that up? | 20:21 |
| fungi | slittle1: took me a minute to dig up the right spot in the infra manual: https://docs.openstack.org/infra/manual/creators.html#creation-of-branches | 20:21 |
| fungi | there's also a lot about feature branches in particular at https://docs.openstack.org/infra/manual/drivers.html#feature-branches which includes info on the acl you'll need to be able to push merge commits between them and your main branches | 20:21 |
| *** jamesmcarthur_ has joined #openstack-infra | 20:23 | |
| *** jamesmcarthur has quit IRC | 20:26 | |
| fungi | slittle1: looking at the acls you have, for example https://git.openstack.org/cgit/openstack-infra/project-config/tree/gerrit/acls/openstack/stx-metal.config already includes create permission for members of the starlingx-release group | 20:26 |
| fungi | slittle1: so any member of https://review.openstack.org/#/admin/groups/1921,members can go to https://review.openstack.org/#/admin/projects/openstack/stx-metal,branches and use the ui there to create a new branch | 20:28 |
| * dtroyer catching up | 20:28 | |
| dtroyer | right, it should just be a group membership thing for us | 20:28 |
| fungi | that group also seems to be self-managed, so anyone in that group can add anyone else to that group too | 20:29 |
| dansmith | is it just me or has the gerrit UI become really (really) annoying since the new year? like, all kinds of jumping around when creating a new comment, typing, etc | 20:34 |
| dansmith | there's always a little of that, but it seems like a major change lately | 20:34 |
| fungi | i don't believe the software has changed in many, many months | 20:39 |
| *** smarcet has quit IRC | 20:40 | |
| *** spsurya has quit IRC | 20:41 | |
| ssbarnea | there are few recent CR on https://review.openstack.org/#/q/project:openstack-infra/devstack-gate+status:open which could be useful | 20:44 |
| SpamapS | mordred: shade is now 1.30.0 in Debian unstable. :) | 20:44 |
| SpamapS | and no RC bugs means it might even make it into the next stable release of Debian. ;) | 20:44 |
| fungi | smarcet: so coming back to your question from earlier, the legacy/laravel-openstackid-unittests/run.yaml playbook seems to include the bindep role by reference, followed by an anonymous list of tasks. i guess what you want is a way to reorder those so that the ppa addition shell task happens before the tasks from the bindep role | 20:45 |
| *** tosky has quit IRC | 20:49 | |
| *** kgiusti has left #openstack-infra | 20:55 | |
| *** kjackal has quit IRC | 20:56 | |
| *** kjackal has joined #openstack-infra | 20:56 | |
| *** smarcet has joined #openstack-infra | 20:56 | |
| *** xek has quit IRC | 21:02 | |
| *** xek has joined #openstack-infra | 21:02 | |
| *** dustinc has quit IRC | 21:03 | |
| *** armax has joined #openstack-infra | 21:08 | |
| *** jbadiapa has quit IRC | 21:10 | |
| *** hamerins has quit IRC | 21:14 | |
| *** hamerins has joined #openstack-infra | 21:15 | |
| *** slaweq has quit IRC | 21:28 | |
| *** jamesdenton has quit IRC | 21:35 | |
| *** jcoufal has quit IRC | 21:47 | |
| *** jamesdenton has joined #openstack-infra | 21:56 | |
| *** armax has quit IRC | 21:58 | |
| *** boden has quit IRC | 22:01 | |
| *** smarcet has quit IRC | 22:05 | |
| *** trown is now known as trown|outtypewww | 22:07 | |
| *** slaweq has joined #openstack-infra | 22:23 | |
| *** e0ne has quit IRC | 22:27 | |
| *** kmarc_ has quit IRC | 22:29 | |
| *** kmarc_ has joined #openstack-infra | 22:32 | |
| *** kmarc_ has quit IRC | 22:32 | |
| *** kmarc_ has joined #openstack-infra | 22:33 | |
| *** kmarc_ has quit IRC | 22:34 | |
| *** kmarc_ has joined #openstack-infra | 22:36 | |
| *** armax has joined #openstack-infra | 22:53 | |
| *** rcernin has joined #openstack-infra | 22:53 | |
| *** kjackal has quit IRC | 22:55 | |
| *** d0ugal has joined #openstack-infra | 22:56 | |
| clarkb | dansmith: ya I've noticed some of that in firefox since I got back from holidays | 22:56 |
| clarkb | dansmith: I wonder if some reelase of firefox has made it worse | 22:56 |
| dansmith | clarkb: I'm on chrome | 22:56 |
| clarkb | hrm, we haven't chagned the js at all so odd that multiple browsers would do it | 22:56 |
| clarkb | but not impossible | 22:56 |
| dansmith | it is ... super annoying | 22:56 |
| *** tosky has joined #openstack-infra | 23:01 | |
| *** jesusaur has quit IRC | 23:02 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Nameservers are now managed with ansible https://review.openstack.org/629257 | 23:13 |
| clarkb | corvus: infra-root ^ I realized there was a bit more cleanup we should do there | 23:13 |
| *** hamerins has quit IRC | 23:13 | |
| clarkb | if we can get that in the next step is removing the old servers | 23:14 |
| *** rascasoft has quit IRC | 23:16 | |
| *** rlandy|rover is now known as rlandy|rover|bbl | 23:16 | |
| *** dustinc has joined #openstack-infra | 23:23 | |
| clarkb | logan-: not to completely overwhelm you with all the weird cloud behaviors, but http://logs.openstack.org/84/608984/7/gate/openstack-tox-py27/97fede3/job-output.txt.gz#_2019-01-08_21_59_31_329767 and other tests like it fail more often on limestone than other clouds (though there are other cloud cases too). bd7224f4352ae7cdf04e64151b2efb0315b0aeca4fb3c03b3124de18 and | 23:26 |
| clarkb | 77681beb0ddf7a44597e1e8fa6f648861b5edf5445510df6083cdb72 are the host ids that this most often happens on (there is one case on 704a6e4d2ae61ad0bf113de69b52cb6414dadb287241358ebaf1c7b2) but all others are those two hypervisors according to the logstash logs | 23:26 |
| clarkb | logan-: anything stand out io wise on those hypervisors? no swapping or unhappy disks or maybe kvm writethrough is different on them? | 23:26 |
| clarkb | I guess limestone is running on ceph so could be network related? | 23:26 |
| *** dustinc has left #openstack-infra | 23:27 | |
| *** rascasoft has joined #openstack-infra | 23:28 | |
| *** armax has quit IRC | 23:29 | |
| *** armax has joined #openstack-infra | 23:30 | |
| *** jamesmcarthur_ has quit IRC | 23:43 | |
| *** jamesmcarthur has joined #openstack-infra | 23:45 | |
| *** jamesmcarthur has quit IRC | 23:50 | |
| *** armax has quit IRC | 23:54 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Update to Fedora 29 https://review.openstack.org/618431 | 23:56 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!