Thursday, 2019-09-26

*** goldyfruit_ has joined #openstack-kolla00:04
*** igordc has joined #openstack-kolla00:44
*** BjoernT has joined #openstack-kolla02:04
*** r3ap3r has joined #openstack-kolla02:36
*** dave-mccowan has joined #openstack-kolla03:12
*** igordc has quit IRC03:18
*** igordc has joined #openstack-kolla03:18
*** dave-mccowan has quit IRC03:47
*** BjoernT has quit IRC04:07
*** BjoernT has joined #openstack-kolla04:13
*** BjoernT has quit IRC04:29
*** BjoernT has joined #openstack-kolla04:31
*** janki has joined #openstack-kolla04:35
*** dpawlik has joined #openstack-kolla05:06
*** BjoernT has quit IRC05:17
*** BjoernT_ has joined #openstack-kolla05:17
*** BjoernT_ has quit IRC05:17
*** BjoernT has joined #openstack-kolla05:18
*** BjoernT has quit IRC05:18
*** BjoernT_ has joined #openstack-kolla05:18
*** BjoernT_ has quit IRC05:19
*** BjoernT has joined #openstack-kolla05:23
*** BjoernT has quit IRC05:23
*** BjoernT_ has joined #openstack-kolla05:23
*** BjoernT_ has quit IRC05:24
*** BjoernT has joined #openstack-kolla05:25
*** BjoernT has quit IRC05:25
*** BjoernT_ has joined #openstack-kolla05:26
*** BjoernT_ has quit IRC05:26
*** BjoernT has joined #openstack-kolla05:27
*** BjoernT has quit IRC05:28
*** BjoernT_ has joined #openstack-kolla05:28
*** BjoernT_ has quit IRC05:29
*** BjoernT has joined #openstack-kolla05:29
*** BjoernT has quit IRC05:30
*** BjoernT has joined #openstack-kolla05:32
*** BjoernT has quit IRC05:32
*** BjoernT has joined #openstack-kolla05:33
*** BjoernT has quit IRC05:33
*** BjoernT has joined #openstack-kolla05:33
*** BjoernT has quit IRC05:34
*** BjoernT_ has joined #openstack-kolla05:34
*** BjoernT_ has quit IRC05:35
*** BjoernT has joined #openstack-kolla05:35
*** BjoernT has quit IRC05:37
*** tonythomas has quit IRC05:46
*** radeks has joined #openstack-kolla05:54
yoctozeptomorning06:09
*** skramaja has joined #openstack-kolla06:29
*** pcaruana has joined #openstack-kolla06:30
*** cah_link has joined #openstack-kolla06:35
yoctozeptomgoddard: looks like rabbitmq in centos cannot handle ipv6 addresses ;/06:40
yoctozeptoor it's centos in general cause it takes it a while to get local ipv6 address reachable via ping for starters06:41
*** gfidente has joined #openstack-kolla06:47
*** igordc has quit IRC06:53
mnasiadkamorning07:05
mnasiadkaso, ceph doesn't like our new vxlan thingie07:10
mnasiadkaI mean ceph-ansible07:10
mnasiadkamaybe because we don't set the docker mtu?07:11
*** tonythomas has joined #openstack-kolla07:15
*** ivve has joined #openstack-kolla07:19
egonzalezmorning07:22
openstackgerritRadosław Piliszek proposed openstack/kolla master: Use C.UTF-8 locale  https://review.opendev.org/68480807:29
yoctozeptomnasiadka: it's haproxy giving 504 for keystone07:29
mnasiadkayoctozepto: where?07:30
*** hjensas has joined #openstack-kolla07:30
yoctozeptoin recent ceph jobs failures07:30
yoctozeptoyou meant something else? ;p07:30
mnasiadkanaah, I mean my ceph-ansible CI job07:31
mnasiadkamons have problems reaching a quorum when using vxlan07:32
mnasiadkawhen I use parent interface - it works fine07:32
mnasiadkaI'll recheck just to be sure and try to find the issue...07:33
yoctozeptohmm, odd07:33
yoctozeptokolla ceph works fine07:33
yoctozeptobut we run on host networking07:34
yoctozeptoif ceph-ansible uses docker bridge, then it could be the case07:34
mnasiadkaI don't think it does, let me double check07:34
mnasiadkayeah, NetworkMode: "host"07:36
mnasiadkaso that shouldn't be the issue07:36
yoctozeptothen it should be just fine like ours07:38
yoctozeptomnasiadka: https://cb16041cfb3c54cedd2e-24bc61d83ed5aece64ab40b405cf025c.ssl.cf5.rackcdn.com/676376/72/check/kolla-ansible-centos-source-ceph-ansible/12caf3b/primary/logs/system_logs/ss.txt.gz07:42
yoctozeptoceph-mon seems to be listening at the public address07:42
yoctozeptois it the falure case or the "fixed" case?07:42
yoctozeptoah yeah07:43
yoctozeptohere is the local case: https://5845c76054a956b85ff7-2e4c3ae56b094a6634f1d9ca742de644.ssl.cf5.rackcdn.com/676376/71/check/kolla-ansible-centos-source-ceph-ansible/f7de896/primary/logs/system_logs/ss.txt.gz07:43
yoctozeptoindeed no connections established07:43
mnasiadkainteresting :)07:43
yoctozeptoall in time wait07:44
*** jaicaa has quit IRC07:44
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637607:45
*** jaicaa has joined #openstack-kolla07:46
*** FlorianFa has joined #openstack-kolla07:52
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637607:53
mnasiadkayoctozepto: ok, seems like it takes mtu from eth0, let me try setting public_network and see what happens07:54
*** gregoryo has joined #openstack-kolla07:54
yoctozeptomnasiadka: network selection looks fine, mtu too, something else must be awry then07:58
*** rgogunskiy has joined #openstack-kolla08:09
*** rpittau|afk is now known as rpittau08:12
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637608:14
*** gregoryo has quit IRC08:17
*** k_mouza has joined #openstack-kolla08:25
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637608:29
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637608:43
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Update Neutron reference docs  https://review.opendev.org/68473008:57
*** luksky has joined #openstack-kolla09:00
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Update Neutron reference docs  https://review.opendev.org/68473009:03
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637609:07
openstackgerritMark Goddard proposed openstack/kolla-ansible master: DNM: Always enable TLS in CI  https://review.opendev.org/68497809:10
openstackgerritMark Goddard proposed openstack/kolla-ansible master: DNM: Always enable TLS in CI  https://review.opendev.org/68497809:15
openstackgerritIsaac Prior proposed openstack/kolla-ansible master: Parse MariaDB log messages in different formats into Fluentd  https://review.opendev.org/68498809:17
openstackgerritMerged openstack/kolla-ansible master: Add support for libvirt+tls  https://review.opendev.org/65044809:19
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Update Neutron reference docs  https://review.opendev.org/68473009:25
mnasiadkauhh, why oh why docutils do not support :caption: in :code-block: ...09:31
openstackgerritMark Goddard proposed openstack/kayobe master: Build pdf doc  https://review.opendev.org/68261609:32
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Update Neutron reference docs  https://review.opendev.org/68473009:33
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Remove delegate_to from service-ks-register tasks  https://review.opendev.org/68500809:39
openstackgerritMerged openstack/kolla-ansible master: Fix deployment of Grafana  https://review.opendev.org/68453409:42
openstackgerritMerged openstack/kolla-ansible master: CI: Fix check-logs.sh  https://review.opendev.org/68335609:42
openstackgerritMerged openstack/kolla-ansible master: [prometheus] Added documentation for deployment and setting extra options  https://review.opendev.org/68409009:42
openstackgerritMerged openstack/kayobe-config master: Update links to docs and IRC for kolla governance  https://review.opendev.org/67451109:50
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Check containers during kolla-ansible check  https://review.opendev.org/59973509:52
openstackgerritMerged openstack/kayobe-config-dev master: Update links to docs and IRC for kolla governance  https://review.opendev.org/67450810:04
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Update Neutron reference docs  https://review.opendev.org/68473010:15
*** BjoernT has joined #openstack-kolla10:23
mgoddardwhat's with these 504 gateway timeouts in the ceph jobs?10:28
openstackgerritRadosław Piliszek proposed openstack/kolla master: Use C.UTF-8 locale  https://review.opendev.org/68480810:30
yoctozeptomgoddard: networking death magic :<10:30
mgoddardooh10:30
mgoddard2019-09-25 18:19:31.811769 Script timed out before returning headers: keystone-wsgi-admin10:31
*** pcaruana has quit IRC10:32
yoctozeptoyeah, it's keystone all around10:34
yoctozeptoatm10:34
mgoddardinteresting, haproxy round robin kills us: https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a06/675659/21/check/kolla-ansible-centos-source-ceph/a061590/primary/logs/kolla/haproxy/haproxy_latest.b593646f241d7f557238ab523de2c9e4c.txt.gz10:36
mgoddardeach request includes 3 API calls, so we always end up back on the broken on (assuming the others aren't also broken)10:36
yoctozeptoah, yeah, because we control reissue at the client level, not api10:40
yoctozeptowhat's wrong with that secondary1 anyway10:41
mgoddardpass10:41
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: External Ceph: keys as variables  https://review.opendev.org/68503210:44
openstackgerritOndrej Duchon proposed openstack/kolla-ansible master: Corrected notifications for auditing.  https://review.opendev.org/67854410:51
openstackgerritMerged openstack/kolla-ansible master: Switch default cloudkitty storage backend to influxdb  https://review.opendev.org/68155410:58
openstackgerritMerged openstack/kolla-ansible stable/rocky: Fix for haproxy precheck failing on CentOS running non-root  https://review.opendev.org/68357610:58
openstackgerritMerged openstack/kolla-ansible stable/queens: Fix for haproxy precheck failing on CentOS running non-root  https://review.opendev.org/68358210:58
*** pcaruana has joined #openstack-kolla10:58
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: Switch default cloudkitty storage backend to influxdb  https://review.opendev.org/68177211:02
yoctozeptomgoddard: maybe report to keystoners? there is no way to get more info about it, we already run debug and it gave nothing11:03
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: DNM: Testing cloudkitty in Stein  https://review.opendev.org/68154811:03
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: External Ceph: keys as variables  https://review.opendev.org/68503211:03
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637611:04
mgoddardyoctozepto: it does seem to have started since VXLAN though right?11:08
openstackgerritOndrej Duchon proposed openstack/kolla-ansible master: Added support for custom logging.conf  https://review.opendev.org/67981211:11
stingrayzahello folks o/11:13
mgoddardhi stingrayza11:13
yoctozeptomgoddard: since vxlan we use haproxy so indeed 504 started with haproxy...11:15
yoctozepto;p11:15
mgoddardyoctozepto: true, although it was always in use in single node11:16
yoctozeptomgoddard: but then all services are single11:16
yoctozeptostill, we got some 504 there11:16
mgoddardyes11:16
yoctozeptonot sure if deadly11:16
yoctozeptowe did not retry before either11:16
yoctozeptoso each 504 was "meh, random"11:16
yoctozeptowell, it's still random ;p11:16
openstackgerritMark Goddard proposed openstack/kolla-ansible master: DNM: Always enable TLS in CI  https://review.opendev.org/68497811:17
mnasiadkaI love our intermittent CI failures :)11:18
*** kplant has joined #openstack-kolla11:20
yoctozeptomnasiadka: there is a reason for them for sure11:20
yoctozeptobut it's hard to find when you don't get access to the broken deploy11:20
mnasiadkayoctozepto: go to #openstack-infra and ask for it :)11:21
mnasiadka(but it's random, so you don't know which one :D)11:21
yoctozeptonah, I'm thinking about using my infra for that, but I need other things sorted out first11:21
yoctozeptobut we can get there, we can :-)11:21
yoctozeptoyou know, a simple: deploy/destroy cycle until you fail11:22
yoctozeptoand then message me and leave it alone11:22
*** BjoernT has quit IRC11:23
yoctozepto(knowing my luck, it would get fixed by the time I have time to look at it)11:23
yoctozeptobut anyways11:23
mnasiadkahaha11:25
mnasiadkabtw, setting public_network to vxlan cidr fixed the ceph-ansible issue11:25
yoctozeptohmm, sounds like ceph bug11:28
yoctozeptobut glad it fixed it11:28
yoctozeptowonder what it tried to do11:29
yoctozeptoas it listened on proper addresses11:29
yoctozeptodocument it11:29
mnasiadkayoctozepto: well, not a bug - by default it's 0.0.0.0/0 - so it takes MTU of the NIC through which it has default routing11:36
mnasiadkamgoddard, yoctozepto: what should be the approach in making ceph keys (external ceph) as variables? Set the same values that we had in the past and just allow the user to set them - or make them by default streamlined with ceph-ansible?11:38
mgoddardmnasiadka: probably can't change defaults11:38
mnasiadkaYeah, that's my thinking - let's not break things11:39
mnasiadkaat least not before removing kolla-ceph completely11:39
*** ondrejduchon has joined #openstack-kolla11:40
yoctozeptomnasiadka: well, it makes more sense to set mtu from the listening interface but who would ask me ;p11:44
yoctozeptomgoddard, mnasiadka: +1 for letting ppl override without changing the defaults11:44
mnasiadkaso let me update to the defaults, and let's get over with it :)11:45
openstackgerritMerged openstack/kolla master: Fluentd: td-agent repository for Debian  https://review.opendev.org/68411911:46
openstackgerritMerged openstack/kolla-ansible stable/stein: Fix kolla_toolbox error handling  https://review.opendev.org/68234411:46
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: External Ceph: keys as variables  https://review.opendev.org/68503211:55
*** pcaruana has quit IRC12:05
*** janki has quit IRC12:05
*** k_mouza has quit IRC12:10
*** k_mouza has joined #openstack-kolla12:15
*** pcaruana has joined #openstack-kolla12:19
*** goldyfruit_ has quit IRC12:23
openstackgerritRadosław Piliszek proposed openstack/kolla master: Placement: Debian pkg has migration script in different path  https://review.opendev.org/68368012:23
openstackgerritRadosław Piliszek proposed openstack/kolla master: added broken images to UNBUILDABLE_IMAGES  https://review.opendev.org/68368112:25
openstackgerritRadosław Piliszek proposed openstack/kolla master: added broken images to UNBUILDABLE_IMAGES  https://review.opendev.org/68368112:25
*** vmixor has joined #openstack-kolla12:31
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637612:37
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Update Neutron reference docs  https://review.opendev.org/68473012:44
openstackgerritMerged openstack/kolla-ansible stable/stein: Cloudkitty InfluxDB Storage backend via Kolla-ansible  https://review.opendev.org/68154512:47
*** BjoernT has joined #openstack-kolla12:51
*** dpawlik has quit IRC13:10
*** rgogunskiy_ has joined #openstack-kolla13:15
*** rgogunskiy has quit IRC13:18
*** dpawlik has joined #openstack-kolla13:28
*** goldyfruit_ has joined #openstack-kolla13:31
openstackgerritViktor Michalek proposed openstack/kolla master: Neutron: fwaas should be installed also in Debian  https://review.opendev.org/68367813:47
*** k_mouza_ has joined #openstack-kolla13:49
*** priteau has joined #openstack-kolla13:50
*** k_mouza_ has quit IRC13:51
*** k_mouza_ has joined #openstack-kolla13:51
*** rgogunskiy_ has quit IRC13:52
*** k_mouza has quit IRC13:54
*** priteau has quit IRC13:55
*** BjoernT has quit IRC13:57
*** BjoernT has joined #openstack-kolla14:00
*** goldyfruit_ has quit IRC14:08
*** goldyfruit_ has joined #openstack-kolla14:16
*** ivve has quit IRC14:36
*** k_mouza_ has quit IRC14:52
*** k_mouza has joined #openstack-kolla15:00
*** skramaja has quit IRC15:15
*** goldyfruit___ has joined #openstack-kolla15:25
*** goldyfruit_ has quit IRC15:28
*** goldyfruit_ has joined #openstack-kolla15:34
*** goldyfruit___ has quit IRC15:37
*** dpawlik has quit IRC15:43
*** dpawlik has joined #openstack-kolla15:43
*** cah_link has quit IRC15:44
openstackgerritViktor Michalek proposed openstack/kolla master: Horizon: fix for Debian  https://review.opendev.org/68367715:48
*** priteau has joined #openstack-kolla15:48
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Update Neutron reference docs  https://review.opendev.org/68473015:49
openstackgerritRadosław Piliszek proposed openstack/kolla master: Neutron: fwaas should be installed also in Debian  https://review.opendev.org/68367815:53
*** gfidente has quit IRC15:54
*** pcaruana has quit IRC15:57
*** rpittau is now known as rpittau|afk15:58
*** ktibi has joined #openstack-kolla16:02
openstackgerritMerged openstack/kolla master: added broken images to UNBUILDABLE_IMAGES  https://review.opendev.org/68368116:03
openstackgerritViktor Michalek proposed openstack/kolla master: Horizon: fix for Debian  https://review.opendev.org/68367716:03
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637616:04
*** igordc has joined #openstack-kolla16:14
openstackgerritOndrej Duchon proposed openstack/kolla-ansible master: Added support for custom logging.conf  https://review.opendev.org/67981216:15
*** ktibi has quit IRC16:19
openstackgerritViktor Michalek proposed openstack/kolla master: Horizon: fix for Debian  https://review.opendev.org/68367716:24
*** BjoernT has quit IRC16:26
*** ondrejduchon has quit IRC16:30
*** BjoernT has joined #openstack-kolla16:33
*** ktibi has joined #openstack-kolla16:35
*** priteau has quit IRC16:37
*** hrw has joined #openstack-kolla16:38
hrwmorning16:38
*** k_mouza has quit IRC16:39
mgoddardmorning hrw16:39
mgoddardhave you moved to west coast US?16:40
hrwconference16:40
hrw~curse docker-py for lack of buildkit support16:41
*** luksky has quit IRC16:41
*** igordc has quit IRC16:43
hrwturns out that it can improve build times, is present in 18.09+ docker and we can not use it as docker-py just does not handle it16:48
hrwhttps://github.com/docker/docker-py/issues/223016:48
hrwmgoddard: centos:8 images will be next week but it does not change anything16:48
hrwmgoddard: RDO will do train with centos716:48
mgoddardhrw: I thought RDO would support centos8?16:49
mgoddardchanged their minds?16:49
yoctozeptohrw: how much does it improve the build time? (and how?)16:50
hrwhttp://eavesdrop.openstack.org/irclogs/%23rdo/%23rdo.2019-09-25.log.html#t2019-09-25T15:19:2516:50
hrwmgoddard: 'centos 8 is too late'16:50
hrwyoctozepto: more aggressive caching iirc.16:51
hrwyoctozepto: there will be session about it tomorrow16:51
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Add a job that *only* deploys updated containers  https://review.opendev.org/64596616:51
hrwhttps://review.rdoproject.org/etherpad/p/moving-rdo-to-centos816:51
hrw- RDO Train GA will be on CentOS7 only.16:52
hrw- Have a full RDO Trunk Train repo for CentOS8 in addition to CentOS7 post-GA.16:52
mgoddardhrw: "and later add support for it"16:52
mgoddardsounds like it may get CentOS8 support later16:52
hrwmgoddard: so the question is "do we wait or go for c7"16:53
mgoddardrelease without it16:53
mgoddardif it's easy to add support, we could backport it after release16:53
hrwok16:53
yoctozeptohrw: we are lame at caching our stuff, need to fix that bug with plugins16:57
*** k_mouza has joined #openstack-kolla17:03
*** itlinux has quit IRC17:04
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Remove Nova legacy upgrade  https://review.opendev.org/67819017:05
*** k_mouza has quit IRC17:07
hrwwow. built 264 docker images in 33 minutes17:08
*** henriqueof has joined #openstack-kolla17:09
*** henriqueof1 has quit IRC17:09
*** EmilienM is now known as MniLieEm17:15
*** MniLieEm is now known as EmilienM17:15
hrwin tmpfs17:15
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for self-signed SSL certificates  https://review.opendev.org/66451717:27
*** igordc has joined #openstack-kolla17:28
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637617:30
hrwnext set of same images: 9:5217:47
*** iclon_ has quit IRC17:53
*** dpawlik has quit IRC17:56
*** iclon has joined #openstack-kolla17:56
*** pcaruana has joined #openstack-kolla18:04
*** factor has joined #openstack-kolla18:05
*** ktibi has quit IRC18:07
hrwwe should prioritize openstack-base to be built as quick as possible - best would be right after base one18:09
*** signed8bit has quit IRC18:10
yankcrimeei18:12
yankcrimela18:12
yankcrimextcerei18:12
yankcrimel18:13
yankcrimesorry, my terminal completely messed itself up :<18:14
*** signed8bit has joined #openstack-kolla18:18
*** igordc has quit IRC18:29
hrwdiscussion for next week: adding priority to images18:30
*** igordc has joined #openstack-kolla18:32
*** pcaruana has quit IRC18:36
*** luksky has joined #openstack-kolla18:47
*** BjoernT has quit IRC19:05
*** BjoernT has joined #openstack-kolla19:07
*** ivve has joined #openstack-kolla19:14
*** dave-mccowan has joined #openstack-kolla19:22
*** dave-mccowan has quit IRC19:27
*** dave-mccowan has joined #openstack-kolla19:28
*** kplant has quit IRC19:41
*** BjoernT_ has joined #openstack-kolla20:04
*** BjoernT has quit IRC20:06
*** ktibi has joined #openstack-kolla20:49
*** ktibi has quit IRC20:49
*** radeks has quit IRC20:59
*** hrw has quit IRC21:09
*** goldyfruit_ has quit IRC21:14
*** luksky has quit IRC21:25
*** BjoernT_ has quit IRC22:01
*** vmixor has quit IRC22:46
*** ivve has quit IRC22:50
*** goldyfruit_ has joined #openstack-kolla23:01

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!