Friday, 2019-09-27

*** BjoernT has joined #openstack-kolla00:14
*** KeithMnemonic has quit IRC00:18
*** BjoernT has quit IRC02:38
*** gregoryo has joined #openstack-kolla03:10
*** dave-mccowan has quit IRC03:24
*** igordc has quit IRC04:08
*** janki has joined #openstack-kolla04:26
*** skramaja has joined #openstack-kolla04:36
*** radeks has joined #openstack-kolla04:55
*** hrw has joined #openstack-kolla05:07
*** dpawlik has joined #openstack-kolla05:16
*** tonythomas has quit IRC05:40
*** Luzi has joined #openstack-kolla05:42
*** cah_link has joined #openstack-kolla06:00
*** Guest29634 has quit IRC06:16
*** gregoryo has quit IRC06:16
*** benj has joined #openstack-kolla06:32
*** benj is now known as Guest5786106:32
*** Luzi has quit IRC06:40
*** Luzi has joined #openstack-kolla06:56
*** negronjl has quit IRC07:00
mnasiadkamorning07:03
*** negronjl has joined #openstack-kolla07:05
*** rpittau|afk is now known as rpittau07:05
*** pcaruana has joined #openstack-kolla07:21
*** vedup19 has joined #openstack-kolla07:22
*** negronjl has quit IRC07:24
*** negronjl has joined #openstack-kolla07:27
*** pbing19 has joined #openstack-kolla07:31
pbing19When i do code level chages in horizon and build using kolla .. my chnages is not reflecting07:33
pbing19even when i can see changes inside container horizon folder07:33
*** luksky has joined #openstack-kolla07:35
openstackgerritMerged openstack/kolla master: ppc64le: Ensure that rabbitmq-server.repo exists before trying sed  https://review.opendev.org/68399707:35
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637607:36
*** yan0s has joined #openstack-kolla07:37
*** priteau has joined #openstack-kolla07:43
*** stingrayza has quit IRC07:57
*** stingrayza has joined #openstack-kolla08:01
mgoddardmornin'08:06
*** priteau has quit IRC08:09
*** k_mouza has joined #openstack-kolla08:17
*** k_mouza has quit IRC08:17
*** k_mouza has joined #openstack-kolla08:18
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Remove /etc/hosts entries pointing hostname to localhost  https://review.opendev.org/68523308:26
*** ivve has joined #openstack-kolla08:30
*** rgogunskiy has joined #openstack-kolla08:33
*** pcaruana has quit IRC08:52
openstackgerritMerged openstack/kolla-ansible master: Update Neutron reference docs  https://review.opendev.org/68473009:02
*** pcaruana has joined #openstack-kolla09:03
*** ondrejduchon has joined #openstack-kolla09:16
*** rpittau is now known as rpittau|bbl09:24
openstackgerritMark Goddard proposed openstack/kolla-ansible master: DNM: Always enable TLS in CI  https://review.opendev.org/68497809:28
ondrejduchonmgoddard: Hello, I managed logging.conf functionality for other projects and services. https://review.opendev.org/#/c/679812/4 Check it please.09:31
mgoddardhi ondrejduchon. Thanks, I'll have a look when I have time09:32
openstackgerritArx Cruz proposed openstack/kolla master: Adding python3-pyngus package to openstack base  https://review.opendev.org/68530209:41
*** Guest57861 has quit IRC09:44
*** k_mouza has quit IRC09:47
*** benj has joined #openstack-kolla09:47
*** benj is now known as Guest9408309:48
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637609:59
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Ceph-Ansible CI  https://review.opendev.org/67637609:59
*** skramaja has quit IRC10:02
*** k_mouza has joined #openstack-kolla10:03
*** k_mouza has quit IRC10:17
*** k_mouza has joined #openstack-kolla10:19
*** hamzaachi has joined #openstack-kolla10:39
*** hamzaachi has quit IRC10:39
*** luksky has quit IRC10:49
*** vmixor has joined #openstack-kolla11:05
*** hamzaachi has joined #openstack-kolla11:09
*** vedup19 has quit IRC11:10
*** pbing19 has quit IRC11:10
*** luksky has joined #openstack-kolla11:15
*** kplant has joined #openstack-kolla11:18
*** hamzaachi has quit IRC11:38
*** Luzi has quit IRC11:52
yoctozeptoafternoon kollars11:54
*** zul has quit IRC12:06
*** kplant has quit IRC12:07
*** kplant has joined #openstack-kolla12:07
*** rgogunskiy has quit IRC12:08
*** iclon has quit IRC12:08
*** rgogunskiy has joined #openstack-kolla12:08
*** dpawlik has quit IRC12:09
*** dpawlik has joined #openstack-kolla12:09
*** iclon has joined #openstack-kolla12:11
*** goldyfruit_ has quit IRC12:14
*** dpawlik has quit IRC12:16
*** rpittau|bbl is now known as rpittau12:36
openstackgerritIsaac Prior proposed openstack/kolla-ansible master: Parse MariaDB log messages in different formats into Fluentd  https://review.opendev.org/68498812:39
*** tonythomas has joined #openstack-kolla12:40
*** dpawlik has joined #openstack-kolla12:42
openstackgerritMark Goddard proposed openstack/kolla-ansible master: DNM: Always enable TLS in CI  https://review.opendev.org/68497812:43
*** arxcruz|rover has joined #openstack-kolla12:44
arxcruz|roverhey guys, i found this problem here https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_15c/685302/1/check/kolla-build-centos-source/15c63ba/kolla/build/000_FAILED_monasca-grafana.txt.gz12:44
arxcruz|roverrake requires ruby 2.2 but the version installed is 2.012:45
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Remove /etc/hosts entries pointing hostname to localhost  https://review.opendev.org/68523312:48
mgoddardarxcruz|rover: yeah, I just noticed that12:51
mgoddardruby issues again :(12:52
mgoddardhrw: your favourite12:52
mgoddardperhaps we can pin rake12:53
*** dpawlik has quit IRC12:53
mgoddard13.0 was release today12:53
mgoddardI'll raise a bug12:54
*** cah_link has quit IRC12:55
*** dpawlik has joined #openstack-kolla12:56
mgoddardI think I have a fix, just testing12:57
*** hamzaachi has joined #openstack-kolla12:57
*** dpawlik has quit IRC13:00
arxcruz|rovermgoddard: ok, thanks13:03
*** weshay has joined #openstack-kolla13:03
arxcruz|rovermgoddard: may i have the lp and the patch ?13:04
mgoddardarxcruz|rover: https://bugs.launchpad.net/kolla/+bug/184564713:04
openstackLaunchpad bug 1845647 in kolla train "monasca-grafana fails to build due to rake 13.0 release" [Critical,Triaged] - Assigned to Mark Goddard (mgoddard)13:04
mgoddardjust testing the patch13:04
arxcruz|rovermgoddard: cool thanks :)13:05
*** pcaruana has quit IRC13:09
openstackgerritMark Goddard proposed openstack/kolla master: Fix monasca-grafana - pin rake to <13  https://review.opendev.org/68534813:14
mgoddardarxcruz|rover: ^13:15
arxcruz|rovermgoddard: thanks! i'll add a depends on13:15
mnasiadkayoctozepto, mgoddard, hrw: re support matrix - ovsdpdk is marked unbuildable on CentOS, should I mark it as N? I think it should be buildable with some template overrides or something?13:21
*** BjoernT has joined #openstack-kolla13:21
*** goldyfruit_ has joined #openstack-kolla13:22
mgoddardmnasiadka: yeah, if we're not sure then go for unbuildable13:23
mnasiadkaOk13:26
*** luksky has quit IRC13:29
yoctozeptomgoddard: add note pls, thx (monasca patch)13:30
*** dave-mccowan has joined #openstack-kolla13:31
openstackgerritMark Goddard proposed openstack/kolla master: Fix monasca-grafana - pin rake to <13  https://review.opendev.org/68534813:32
mgoddardyoctozepto: noted13:32
yoctozeptomnasiadka: https://review.opendev.org/68534813:35
mnasiadkaAt your service13:36
*** yan0s has quit IRC13:37
vmixorhi there, may I ask for another review of https://review.opendev.org/68367713:44
*** yan0s has joined #openstack-kolla13:49
jovial[m]what's the policy on prometheus exporters? If I add https://github.com/soundcloud/ipmi_exporter would that be accepted?13:49
kplantthat sounds a little out of scope for this project, no?13:51
*** yan0s has quit IRC13:52
openstackgerritScott Solkhon proposed openstack/kolla master: Add Prometheus RabbitMQ exporter  https://review.opendev.org/68479913:53
*** hamzaachi has quit IRC13:55
*** dave-mccowan has quit IRC13:58
*** goldyfruit_ has quit IRC13:59
*** goldyfruit_ has joined #openstack-kolla14:00
*** goldyfruit_ has quit IRC14:00
mgoddardjovial[m]: do they publish an image?14:00
mgoddardand does it require any config, or just command line options?14:01
*** pcaruana has joined #openstack-kolla14:04
jovial[m]mgoddard: I can't see an offical docker image although there are references in the makefile. It only accepts command line arguments as far as I can tell. You give it a list of ipmi addresses through prometheus config.14:07
mgoddardjovial[m]: ok. I was just wondering if there is a way we can support arbitrary exporters14:08
mgoddardjovial[m]: add a custom service definition with how to run the image, and overrides for prometheus.yml14:08
mgoddardwould that be enough?14:08
jovial[m]mgoddard: also, I told a lie - you have to give it IPMI credentials via a config file14:08
mgoddardjovial[m]: it won't take environment variables?14:09
jovial[m]not that I'm aware of. There was this commit: https://github.com/soundcloud/ipmi_exporter/commit/e11e76ed5c059edc752a101fb600a393d73d776a14:10
mgoddardjovial[m]: looks like more about how the exporter communicates with freeipmi14:11
*** luksky has joined #openstack-kolla14:12
jovial[m]but isn't freeipmi just a set of command line tools?14:14
*** luksky has quit IRC14:17
jovial[m]mgoddard: could there be some way of specifying config files to bind mount in aswell?14:22
*** munimeha1 has joined #openstack-kolla14:23
mgoddardjovial[m]: could be, but you need to generate those on the remote hosts14:25
mgoddardwhich requires tasks14:25
jovial[m]for instance, could we add config files as key in the `prometheus_services` definitions and then you could pass in extra services.14:27
jovial[m]some what similar to this: https://github.com/stackhpc/stackhpc.ssm/blob/master/tasks/config.yml#L1314:29
mnasiadkamgoddard: wouldn’t it be easier if we would allow running user defined roles? You can have a milion of node exporters, and if we start to build them - we’re screwed.14:29
jovial[m]user defined roles would be nice :)14:32
mgoddardthat would be nice. Getting into a bigger feature there though14:33
mgoddardI guess with dynamic includes you could just do:14:34
mgoddardinclude_role:14:34
mgoddard  name: "{{ item }}"14:34
mgoddardwith_items: "{{ prometheus_custom_roles }}"14:34
jovial[m]how would the role be downloaded?14:34
*** pcaruana has quit IRC14:35
*** hamzaachi has joined #openstack-kolla14:46
*** radeks has quit IRC14:49
mgoddardjovial[m]: I think that might be out of scope14:52
mgoddardjust assume it's in the role path14:52
*** ondrejduchon has quit IRC14:52
*** BjoernT has quit IRC14:53
openstackgerritMark Goddard proposed openstack/kolla-ansible master: DNM: Always enable TLS in CI  https://review.opendev.org/68497814:54
jovial[m]and the custom roles would just support all the `kolla_actions`? Could we support overriding role default variables as well?14:54
*** BjoernT has joined #openstack-kolla14:56
*** ivve has quit IRC14:58
*** rgogunskiy has quit IRC14:58
mgoddardjovial[m]: yeah, variables would work as normal14:59
*** hamzaachi has quit IRC15:00
*** pbing19 has joined #openstack-kolla15:01
*** vedup19 has joined #openstack-kolla15:01
jovial[m]mgoddard: can I work on a patch to add that then?15:03
mgoddardjovial[m]: sure. we're getting towards feature freeze so it might not make train15:06
*** hamzaachi has joined #openstack-kolla15:21
*** hamzaachi has quit IRC15:22
*** hamzaachi has joined #openstack-kolla15:22
*** arxcruz|rover is now known as arxcruz|zzz15:22
*** ktibi has joined #openstack-kolla15:23
openstackgerritArx Cruz proposed openstack/kolla master: Adding python3-pyngus package to openstack base  https://review.opendev.org/68530215:29
openstackgerritArx Cruz proposed openstack/kolla master: Adding python3-pyngus package to openstack base  https://review.opendev.org/68530215:30
*** k_mouza has quit IRC15:31
*** k_mouza_ has joined #openstack-kolla15:31
*** pcaruana has joined #openstack-kolla15:36
*** vedup19 has quit IRC15:43
*** pbing19 has quit IRC15:43
*** ktibi has quit IRC15:46
*** rpittau is now known as rpittau|afk15:48
*** hamzaachi has quit IRC15:50
*** jbadiapa has joined #openstack-kolla15:56
*** hamzaachi has joined #openstack-kolla15:57
mgoddardinteresting issue hitting kayobe CI: https://bugs.launchpad.net/kolla-ansible/+bug/184568116:01
openstackLaunchpad bug 1845681 in kolla-ansible train "kolla_toolbox module fails with docker 1.12.0 in master" [High,Triaged]16:01
*** hamzaachi has quit IRC16:05
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: CI: add Debian/binary job  https://review.opendev.org/68367516:10
yoctozeptomgoddard: argh :/16:11
mgoddardyoctozepto: yeah :(16:11
*** vedup19 has joined #openstack-kolla16:12
*** pbing19 has joined #openstack-kolla16:12
hrwmorning16:13
*** iniazi_ has quit IRC16:13
hrwthe good part: from Tuesday I will be in european time16:14
hrws/ in / on /16:14
hrwdoes someone understand python queue stuff?16:18
openstackgerritSorin Sbarnea proposed openstack/kolla master: Adding python3-pyngus package to openstack base  https://review.opendev.org/68530216:18
openstackgerritSorin Sbarnea proposed openstack/kolla master: Adding python3-pyngus package to openstack base  https://review.opendev.org/68530216:19
*** iniazi has joined #openstack-kolla16:24
openstackgerritViktor Michalek proposed openstack/kolla master: CI: add Debian/binary job  https://review.opendev.org/68367516:27
*** k_mouza has joined #openstack-kolla16:32
openstackgerritSorin Sbarnea proposed openstack/kolla master: Adding python*-pyngus package to openstack base  https://review.opendev.org/68530216:34
*** k_mouza_ has quit IRC16:34
openstackgerritSorin Sbarnea proposed openstack/kolla master: Adding python*-pyngus package to openstack base  https://review.opendev.org/68530216:36
*** k_mouza has quit IRC16:37
openstackgerritViktor Michalek proposed openstack/kolla master: Horizon: fix for Debian  https://review.opendev.org/68367716:38
*** zbr|ruck is now known as zbr16:41
*** dpawlik has joined #openstack-kolla16:42
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Horizon: fix for Debian  https://review.opendev.org/68367716:44
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Fluentd: fix missing /var/run/fluentd  https://review.opendev.org/68367616:47
hrwseparated both16:47
hrwvmixor: now it is like it was supposed to be16:47
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: CI: add Debian/binary job  https://review.opendev.org/68367516:51
hrwand back to two dependson16:51
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Placement: Debian pkg has migration script in different path  https://review.opendev.org/68368016:53
openstackgerritMerged openstack/kolla master: Fix monasca-grafana - pin rake to <13  https://review.opendev.org/68534816:54
*** k_mouza has joined #openstack-kolla16:57
*** k_mouza has quit IRC17:01
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Copy Nova role as a basis for the Nova cell role  https://review.opendev.org/67565817:01
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Add service-rabbitmq role  https://review.opendev.org/68397417:01
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Factor out nova-cell role  https://review.opendev.org/67565917:01
openstackgerritMark Goddard proposed openstack/kolla stable/stein: Fix monasca-grafana - pin rake to <13  https://review.opendev.org/68541017:03
openstackgerritMark Goddard proposed openstack/kolla stable/rocky: Fix monasca-grafana - pin rake to <13  https://review.opendev.org/68541117:03
mgoddardyoctozepto, hrw, mnasiadka: ^ couple of stable patches to unblock CI on those branches17:04
hrwok17:04
hrwdone in both17:05
hrwvmixor: can you take a look at your merged patches and mark worthy ones as stable/stein backports?17:07
*** weshay is now known as weshay|ruck17:08
*** igordc has joined #openstack-kolla17:09
hrwok, debian/binary ci should be sorted17:14
hrwtime to look at some queue17:14
*** radeks has joined #openstack-kolla17:20
*** pcaruana has quit IRC17:23
hrwDebian/binary CI job went green in own patch ;D17:27
* hrw off17:33
openstackgerritViktor Michalek proposed openstack/kolla master: Neutron: use iptables-legacy  https://review.opendev.org/68367917:36
*** hrw has quit IRC17:38
*** itlinux has joined #openstack-kolla17:39
openstackgerritViktor Michalek proposed openstack/kolla master: Neutron: use iptables-legacy  https://review.opendev.org/68367917:46
openstackgerritViktor Michalek proposed openstack/kolla master: Neutron: use iptables-legacy  https://review.opendev.org/68367917:46
*** janki has quit IRC17:49
*** hamzaachi has joined #openstack-kolla17:50
openstackgerritSorin Sbarnea proposed openstack/kolla master: Adding python*-pyngus package to openstack base  https://review.opendev.org/68530217:59
*** dpawlik has quit IRC18:06
*** dciabrin_ has joined #openstack-kolla18:19
*** dciabrin has quit IRC18:20
*** KeithMnemonic has joined #openstack-kolla18:41
openstackgerritRadosław Piliszek proposed openstack/kolla master: Neutron: fwaas should be installed also in Debian  https://review.opendev.org/68367818:54
*** hamzaachi has quit IRC18:55
*** itlinux has quit IRC19:01
openstackgerritMerged openstack/kolla master: Debian: switch fluentd to td-agent on x86_64  https://review.opendev.org/68455119:01
openstackgerritMerged openstack/kolla master: Install RabbitMQ in kolla-toolbox image  https://review.opendev.org/68474219:02
openstackgerritMerged openstack/kolla-ansible stable/stein: Switch default cloudkitty storage backend to influxdb  https://review.opendev.org/68177219:02
openstackgerritMerged openstack/kolla-ansible stable/queens: Ensure keepalived is restarted during upgrade  https://review.opendev.org/68425219:02
*** radeks has quit IRC19:12
openstackgerritViktor Michalek proposed openstack/kolla master: Neutron: use iptables-legacy  https://review.opendev.org/68367919:16
yoctozeptoarar19:20
yoctozeptomore ruby breakage havoc19:20
yoctozeptoin rocky and below for ubuntu19:20
yoctozeptoError installing fluent-plugin-elasticsearch:19:20
yoctozeptofaraday requires Ruby version >= 2.3.19:20
yoctozeptomgoddard, mnasiadka, hrw: ^19:20
*** hamzaachi has joined #openstack-kolla19:20
kplanti've got mariadb breakage after a pull,upgrade from stable/stein and docker hub :-|19:20
kplantcurse mariadb19:21
mnasiadkayoctozepto: that’s why I like td-agent with deps embedded... curse ruby...19:21
yoctozeptorefer https://bugs.launchpad.net/kolla/+bug/184570519:22
openstackLaunchpad bug 1845705 in kolla rocky "ubuntu fluentd image unbuildable on rocky and older - old ruby" [Critical,Triaged]19:22
yoctozeptokplant: what kind of breakage19:23
kplantRUNNING HANDLER [mariadb : wait for slave mariadb] both slaves run through the 10 retries and fail19:23
*** KeithMnemonic has quit IRC19:23
kplantthe containers are restarting endlessly - trying to find out why in logs19:24
kplant2019-09-27 19:25:11 0 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)19:25
kplantthere we go19:25
kplantthey're all failing that they can't connect to each other19:27
kplanti'm going to try to use images from my registry instead of docker hub - hopefully that's the issue :-|19:30
kplanti just switched over to give the newest a shot19:30
kplantany other ideas? i already tried kolla-ansible -i INVENTORY mariadb_recovery19:33
yoctozeptokplant: you did rocky to stein there?19:35
kplantnope19:35
kplantjust a git pull and then kolla-ansible pull19:35
kplantweird.. i fixed it19:36
kplanti switched back to my registry19:36
kplantdid a pull, upgrade (failed)19:36
kplantthen did a mariadb_recovery and it succeeded that time19:36
yoctozeptoyou did stein->stein?19:38
yoctozeptowith upgrade?19:38
kplantyes19:38
yoctozeptono wonder it failed19:38
kplant(:19:38
kplantshould that have a been a pull,deploy ?19:38
kplantand not a pull,upgrade19:38
yoctozeptoit applies workarounds speific to rocky19:38
yoctozeptoyeah19:38
yoctozeptopull deploy19:38
kplantgood to know it was me19:38
kplantand thankfully it was just my lab19:39
kplantbut i mariadb_recovery eventually fixed it19:39
yoctozeptomaybe we should improve the docs19:39
yoctozeptoyeah, it does :-)19:39
kplantit is nice how easy it was to fix19:39
kplantkolla++19:39
yoctozepto:-)19:39
yoctozeptohttps://bugs.launchpad.net/kolla-ansible/+bug/184570819:42
openstackLaunchpad bug 1845708 in kolla-ansible "improve docs regarding upgrade (major/minor)" [Undecided,New]19:42
yoctozeptobeing lazy before bed19:42
yoctozeptogood night channel19:42
kplantnight night19:43
*** negronjl has quit IRC19:44
*** negronjl has joined #openstack-kolla19:47
*** kplant has quit IRC19:47
*** jbadiapa has quit IRC19:50
*** igordc has quit IRC19:52
*** igordc has joined #openstack-kolla20:23
openstackgerritMerged openstack/kolla master: Placement: Debian pkg has migration script in different path  https://review.opendev.org/68368020:32
*** luksky has joined #openstack-kolla20:40
openstackgerritMerged openstack/kolla stable/stein: Fix monasca-grafana - pin rake to <13  https://review.opendev.org/68541020:42
*** vedup19 has quit IRC21:02
*** pbing19 has quit IRC21:02
*** negronjl has quit IRC21:16
*** negronjl has joined #openstack-kolla21:19
*** munimeha1 has quit IRC21:35
*** hamzaachi has quit IRC21:40
*** BjoernT has quit IRC21:40
*** igordc has quit IRC22:01
*** igordc has joined #openstack-kolla22:01
*** vmixor has quit IRC22:01
*** igordc has quit IRC22:17
*** signed8bit has quit IRC22:26
*** igordc has joined #openstack-kolla22:29
*** igordc has quit IRC23:32
*** igordc has joined #openstack-kolla23:34
*** igordc has quit IRC23:38
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: DNM: Always enable TLS in CI  https://review.opendev.org/68497823:49

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!