Friday, 2020-04-03

*** k_mouza has joined #openstack-kolla00:21
*** pbing19 has quit IRC00:37
*** zhanglong has joined #openstack-kolla01:14
*** cah_link has quit IRC01:34
*** k_mouza has quit IRC01:36
*** noxoid has quit IRC02:11
*** k_mouza has joined #openstack-kolla03:37
*** k_mouza has quit IRC03:41
*** skramaja has joined #openstack-kolla03:48
*** ykarel|away is now known as ykarel03:54
*** skramaja has quit IRC03:58
*** skramaja_ has joined #openstack-kolla03:58
*** kcnomad has quit IRC04:01
*** Tony31 has quit IRC04:22
*** evrardjp has quit IRC04:36
*** evrardjp has joined #openstack-kolla04:37
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for encrypting backend Keystone HAProxy traffic  https://review.opendev.org/71398605:01
*** JangwonLee_ has joined #openstack-kolla05:51
*** JangwonLee__ has quit IRC05:54
*** JangwonLee__ has joined #openstack-kolla05:55
*** JangwonLee_ has quit IRC05:59
*** Tony31 has joined #openstack-kolla06:08
Tony31morning06:09
*** bjolo has joined #openstack-kolla06:17
*** srini_ has joined #openstack-kolla06:18
srini_morning Tony3106:18
Tony31Morning srini_ how are you doing?06:19
Tony31Hope you and your family are keeping well06:20
srini_I am doing fine, Yes everyone is good. thanks for asking06:21
*** igordc has quit IRC06:21
Tony31good to hear srini_06:25
srini_Tony31, quick update, we are really considering running control-planes in Ovirt https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/director_installation_and_usage/creating-virtualized-control-planes06:31
srini_Tony31, Thanks for the suggestion :)06:32
Tony31Just checking this link06:35
Tony31a few months ago I Was looking for ovirt / openstack integrations06:35
osmanlicilegimorning06:35
Tony31morning osmanlicilegi06:35
osmanlicilegihow do you do folks?06:36
mnasiadkamorning06:38
Tony31morning mnasiadka06:38
Tony31All good osmanlicilegi06:38
Tony31@srini_ - thanks for the link, I am doing this at the moment. I have a task in my list to try and run a performance optimised VM. For the new openstack I am working on, I've not done that though but I do have 3 x controllers06:40
Tony31if you run haproxy on ovirt, you need to set the network to have no filter. By default it does not allow mac spoof and because of the floating virtual ip and virtual mac address for kolla / kayobe haproxy, it does not work unless you configure RHV06:41
Tony31otherwise, all good :)06:41
srini_Tony31, thanks for the tip, :)06:42
Tony31:)06:46
Tony31I have an issue with `kolla_internal_tls_cert:` the error during the deploy is cert validation failed. I am trying to use the same cert for the internal API as well as the external endpoint - is it possible?06:57
Tony31the cert is signed by godaddy and I can access the external / horizon using https without any issue or error. But when I try to apply the same to the internal api i get a validation error.06:58
Tony31thought of asking here in case you guys already ran into the same issue ?06:58
yoctozeptoTony31: does the internal fqdn agree? I would suspect name validation07:01
yoctozeptomorning folks07:01
yoctozeptobline: any and all help is appreciated07:02
Tony31yoctozepto - it's a wildcard cert *.domain.com internal api is stack-int.domain.com and external is stack.domain.com07:03
Tony31so it should be working as I understand it. I previously used *.domain.com for jira-staging.domain.com and the "-staging" didnt cause any validation issue07:04
*** hjensas|afk is now known as hjensas07:05
*** mrunge_ is now known as mrunge07:05
*** bengates has joined #openstack-kolla07:11
*** cah_link has joined #openstack-kolla07:16
*** bengates has quit IRC07:16
*** bengates has joined #openstack-kolla07:17
*** ykarel is now known as ykarel|afk07:21
yoctozeptoTony31: ack, makes sense; is there nothing more about that validation failure? no hints?07:30
Tony31snippet:07:37
Tony31'auth_url is correct. %s' % e)\nkeystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. SSL exception connecting to https://kayostack-int.domain.com:35357: HTTPSConnectionPool(host='kayostack-int.domain.com', port=35357): Max retries exceeded with url: / (Caused by SSLError(SSLError(\"bad handshake: Error([('SSL routines',07:37
Tony31'tls_process_server_certificate', 'certificate verify failed')],)\",),))\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}07:37
*** ykarel|afk is now known as ykarel07:37
Tony31im going to re-deploy without a `-` in the dns and see if it makes a difference, since there is no `-` in the external dns name and it is working. it's the only difference I can see07:38
openstackgerritPierre Riteau proposed openstack/kayobe stable/queens: Install python-openstackclient using upper constraints  https://review.opendev.org/71720807:44
*** jbadiapa has joined #openstack-kolla07:45
*** ktibi has joined #openstack-kolla07:47
*** cah_link has quit IRC07:51
*** ykarel is now known as ykarel|afk07:55
*** rpittau|afk is now known as rpittau07:56
*** JangwonLee__ has quit IRC08:06
*** JangwonLee__ has joined #openstack-kolla08:06
*** ykarel|afk is now known as ykarel|lunch08:09
mgoddardTony31: hi. Internal TLS is a much bigger change than external TLS, which is why it was only recently added08:16
mgoddardTony31: with external TLS, only external API consumers need to use TLS08:16
mgoddardTony31: for internal TLS, all the openstack services communicate with each other using TLS08:16
mgoddardTony31: the python requests library can be quite selective in where it will accept CA certs08:17
mgoddardTony31: there have been quite a few changes in master for internal TLS. Have you set openstack_cacert?08:18
*** cah_link has joined #openstack-kolla08:21
mgoddardyoctozepto: this is the link that fails in the monasca-grafana build: http://mirror.ord.rax.opendev.org:8080/registry.npmjs/@types%2fcolor-name08:21
mgoddardlooks like it is still 40408:21
Tony31hi mgoddard thanks for this mate. I've not set "openstack_cacert"08:22
Tony31. I just done a search for that and it's found in venv/ and src. Is this something I need to manually add somewhere in the kayobe config?08:22
mgoddardTony31: it should be set in the kolla globals.yml file - for kayobe this is etc/kayobe/kolla/globals.yml in kayobe-config08:23
mgoddardTony31: we thought it would not be required for publicly signed certs, but probably it was not tested08:23
*** scottsol has joined #openstack-kolla08:27
cosmicsoundhi08:28
cosmicsoundis someone using gocchi source in grafana, what auth method is best or working?08:29
Tony31mgoddard the godaddy ca cert should be trusted by the system by the local CA cert store as it's a trusted cert authority. The "openstack_cacert" is not in kolla/globals.yml but i could add it and point it to the same as `kolla_internal_fqdn_cacert:`08:29
mgoddardTony31: I'd suggest it's worth a try08:33
mgoddardwhere is that error coming from btw?08:33
Tony31mgoddard it's during the reconfigure deployment08:45
mgoddardTony31: ok08:45
Tony31mgoddard I've just completed a fresh deployment once again. I'll try adding the cert config back again now and re-test, because I've also changed the dns name. If it still fails then i'll add the openstack_cacert and post back :) Thanks for this08:45
*** rmart04 has joined #openstack-kolla08:49
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: CI: Use upper constraints to install clients  https://review.opendev.org/71725908:52
yoctozeptomgoddard: yeah, I'm wondering why servers still haven't reconfigured08:54
mgoddardyoctozepto: are you sure they have not?08:54
yoctozeptomgoddard: sorry, they did, we both looked at old failure08:55
*** pbing19 has joined #openstack-kolla08:55
yoctozeptomgoddard: see https://a28720641abd1053931d-69746c322fe5e3cf8e2976f654b96d92.ssl.cf5.rackcdn.com/717075/1/check/kolla-build-centos8-binary/9e67bb6/08:55
yoctozeptomgoddard: no scaring me!08:55
yoctozeptomgoddard: ruling this issue out08:56
mgoddardyoctozepto: cool so monasca-grafana issue solved?08:56
*** hrw has quit IRC08:57
*** hrw has joined #openstack-kolla08:57
*** k_mouza has joined #openstack-kolla08:59
yoctozeptomgoddard: yeah, master is green09:00
yoctozeptomgoddard: overall status: amber09:00
hrwmorgen09:00
yoctozeptomgoddard: merge me those unittest.mock before ci decides to go red again09:00
mgoddardhi hrw09:00
yoctozeptohrw: buongiorno, signore09:01
hrwyoctozepto: приве́т!09:01
yoctozeptohrw: хахаха09:01
*** sorin-mihai_ has joined #openstack-kolla09:02
openstackgerritMark Goddard proposed openstack/kolla stable/stein: Fix multiple CI issues  https://review.opendev.org/71700809:02
yoctozeptomgoddard: ganesha needs some care09:02
mgoddardyoctozepto: I think I just patched the wrong line09:03
mgoddardyoctozepto: there are two centos mirror seds in stein & earlier, cherry pick got the wrong one09:03
*** sorin-mihai has quit IRC09:03
*** sorin-mihai__ has joined #openstack-kolla09:04
mrungehi, at a quick glance, I couldn't find where the replacement of repo urls for centos is done09:05
mrungeany pointers?09:05
hrwargh.09:05
mrungethere is still the issue, that additional repos are using a mirrorlist which is disliked by ci09:05
*** sorin-mihai_ has quit IRC09:07
mnasiadkamrunge: tests/templates/template-override.j2 in kolla09:07
mnasiadkamrunge: well, one repo - the collectd one :)09:07
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/rocky: CI: Use upper constraints to install clients  https://review.opendev.org/71726209:07
mrungemnasiadka, are the tests being executed before build?09:07
yoctozeptomgoddard: ok09:07
mnasiadkamrunge: no, it's just a file location, not really a test :)09:08
mnasiadkayoctozepto, mgoddard: ganesha had a problem because it was https://mirror.centos.org, and I thought I fixed it?09:08
mgoddardmnasiadka: the patch wasn't merged, and didn't propoerly apply to stein & earlier09:09
mnasiadkawell, that it's not easily backportable was obvious09:09
mnasiadkaI wish we had a better mechanism for setting mirrors :)09:09
hrwone week without any changes in master...09:10
openstackgerritMark Goddard proposed openstack/kolla stable/rocky: Fix multiple CI issues  https://review.opendev.org/71702409:12
hrwmgoddard: can we get RP+2 on important patches?09:12
mgoddardhrw: ok09:13
mrungesigh, the replacement in tests/templates/template-override.j2 works with the repo, do we have a stale mirror again?09:13
hrwI do not follow kolla development daily and have a feeling that last time whenever I read channel it is 'CI is fucked up again'09:13
hrwwhich makes it hard to follow important patches in review09:13
mnasiadkamrunge: it runs once a day I think09:16
mnasiadkamrunge: I mean the mirror-update playbook09:16
mrungeokay, that might explain this situation09:16
mrungethank you mnasiadka09:16
mrungeI'll recheck later today then09:17
yoctozeptohrw: yeah, we can officially announce the kolla fubar days09:18
yoctozeptothere is a lot of fire fighting, both proactive and reactive09:18
mrungemnasiadka, hmmm, the updated repo config is already on the mirror node being used :-/09:19
yoctozeptoINFO:kolla.common.utils.fluentd:curl (https://codeload.github.com/monasca/fluentd-monasca/tar.gz/0.1.1): response: 500, time: 6.129151, size: 2609:19
yoctozeptoINFO:kolla.common.utils.fluentd:[91mtar: This does not look like a tar archive09:19
hrwmgoddard, yoctozepto: I think that after Ussuri release we should run some analytics on it and decide which things we remove as giving most of issues.09:19
yoctozeptoalways monasca to blame09:19
hrwas I have a feeling that ~50% of time we spend on fixing CI rather on development09:20
openstackgerritMark Goddard proposed openstack/kayobe master: Use unittest.mock instead of PyPI mock  https://review.opendev.org/71726609:23
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: CI: Add https match for mirror.centos.org to template-overrides  https://review.opendev.org/71654809:23
hrwopstools use baseurl=http://09:23
mnasiadkamrunge: seems so, just checked in a fresh container and it worked - so maybe just run a recheck ;)09:24
mrungemnasiadka, apparently Piotr already hit recheck :) The last ci run used the older package09:25
mnasiadkaok, so problem solved (I hope) :)09:26
*** ykarel|lunch is now known as ykarel09:26
openstackgerritPierre Riteau proposed openstack/kayobe stable/queens: Fix multiple CI failures  https://review.opendev.org/71720809:38
mgoddardhrw: we can do a review of breakages if you like09:38
hrwAm I right? We have to fix rocky to fix stein to fix train to fix master?09:38
mgoddardhrw: possibly to get upgrade jobs going09:39
hrwmgoddard: I see that patches flow so will look at RP+2 ones and at own ones09:39
mgoddardhrw: master isn't broken09:39
*** pbing19 has quit IRC09:40
*** pbing19 has joined #openstack-kolla09:40
mnasiadkamgoddard: actually - do we really need nfs-ganesha repo now in master - since ceph is gone?09:41
mnasiadkaor maybe it's sucked in as dep in centos-release-ceph-nautilus, ok09:41
mgoddardmnasiadka: correct09:42
yoctozeptomnasiadka: just need more reliable url replace mechanism, that's all09:42
mnasiadkaso we just disable it and never use again - that's why master works09:42
yoctozeptomnasiadka: it works because you fixed it09:43
mnasiadkayoctozepto: well, we could fetch baseurl using yum-config-manager and set it to mirror - if sed is not reliable :D09:43
mgoddardalthough we don't use centos-release-ceph-nautilus on master09:43
mgoddardmaster was never broken09:43
hrwmgoddard: we use it for ceph client libs iirc09:43
mnasiadkamgoddard: because there is no centos-release-ceph-nautilus in c8 yet ;)09:44
mnasiadkahrw: mgoddard meant we don't use the ,,repo'' package, just define the repo manually09:44
hrwk09:44
yoctozeptomnasiadka: it's in a ducking comment09:48
mnasiadkayoctozepto: so yum-config-manager won't print it, geez09:49
*** e0ne has joined #openstack-kolla09:54
mnasiadkayoctozepto: so we're left with sed, it hasn't failed us in the past, we just didn't think http:// will change into https:// ;)09:54
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Custom haproxy script for monitoring galera  https://review.opendev.org/71021309:56
hrwmnasiadka: https://review.opendev.org/#/c/715164/ +2?09:57
patchbotpatch 715164 - kolla-ansible - CI: Always use upper-constraints - 4 patch sets09:57
mnasiadkahrw: +809:59
hrwmuchas gracias, señor10:00
*** gfidente|afk is now known as gfidente10:02
openstackgerritPierre Riteau proposed openstack/kolla-ansible stable/queens: DNM: Try using a mirror of the legacy YUM Docker repository  https://review.opendev.org/71727610:02
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible stable/stein: CI: Use upper constraints to install clients  https://review.opendev.org/71725910:06
hrwadded depends-on on kolla patch10:06
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible stable/stein: CI: Use upper constraints to install clients  https://review.opendev.org/71725910:07
hrwops. unlooped10:07
hrwfsck10:07
yoctozeptomnasiadka: and many other things...10:09
mnasiadkayoctozepto: well, as always :D10:18
*** rpittau is now known as rpittau|bbl10:29
*** born2bake has joined #openstack-kolla10:44
*** zhanglong has quit IRC10:44
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Custom haproxy script for monitoring galera  https://review.opendev.org/71021310:48
*** cah_link has quit IRC10:54
openstackgerritRadosław Piliszek proposed openstack/kolla master: Fix glance-api privsep errors  https://review.opendev.org/71505110:57
openstackgerritBharat Kunwar proposed openstack/kolla stable/train: CI: Add https match for mirror.centos.org to template-overrides  https://review.opendev.org/71661611:11
openstackgerritBharat Kunwar proposed openstack/kolla stable/train: Bump up magnum and manila versions  https://review.opendev.org/71633911:11
openstackgerritBharat Kunwar proposed openstack/kolla stable/train: CI: Add https match for mirror.centos.org to template-overrides  https://review.opendev.org/71661611:11
*** cah_link has joined #openstack-kolla11:14
Tony31mgoddard - same cert verify issue after changing dns and adding the variable11:18
Tony31mgoddard I dont think it's a config error because I copied the yaml for the cert that was working for the external API.11:19
Tony31mgoddard also initially I had the same certificate CA bundle for both variables kolla_internal_fqdn_cacert: and kolla_external_fqdn_cacert:11:19
hrw5 patches at the gate. good11:20
Tony31mgoddard im thinking to leave out internal TLS to move forward, is there anything else I could try to rule out being a config issue or a bug?11:20
mgoddardTony31: I would suggest using the kolla_toolbox container to iterate and find the problem11:22
mgoddarddocker exec -it kolla_toolbox openstack <args>11:22
mgoddardif you can get that to work, then that would be a good start11:22
mgoddardyou may need the --os-cacert argment11:22
Tony31mgoddard sorry mate I'm not on the same page. This maybe too far advanced for me. I tried to run curl to check on this. I think this sounds like it's a bit more advanced than checking curl or logs for user-readable error logs :)11:23
mgoddardTony31: we have the openstack CLI installed in kolla-toolbox. You should be able to use it to reach the APIs via the internal network11:24
Tony31mgoddard - ok thanks Mark I'll look it up :)11:25
mgoddardnp11:25
mgoddardyoctozepto: Looks like we have a CI dependency loop between kolla and k-a.11:25
mgoddardkolla-toolbox image broken so we can't fix k-a11:26
mgoddardk-a clients broken for init-runonce so we can't fix kolla11:26
mgoddardnon-voting deploy jobs in kolla :(11:26
mgoddardWe could really use a proper promotion process for our images, so we don't publish those that fail deploy jobs11:27
hrwmgoddard: or force k-a jobs to build images until fixed?11:28
mgoddardwe should publish as <release>-stage, then retag11:28
mgoddardhrw: that is a better solution, thanks11:28
mgoddardalthough upgrade jobs wouldn't pass11:29
hrwmgoddard: do a patch which adds 'build by default' and makes upgrades not voting. rebase fixes on top. we merge, fix situiation, revert11:29
*** scottsol has quit IRC11:30
mgoddardhrw: no upgrade jobs in rocky, so we could fix rocky, then fix stein tomorrow using fresh rocky images11:30
hrwhttps://review.opendev.org/#/c/717024/ - mnasiadka yoctozepto +2 presto!11:31
patchbotpatch 717024 - kolla (stable/rocky) - Fix multiple CI issues - 4 patch sets11:31
mgoddardhrw: it failed11:32
hrwjoking a bit ;D11:32
yoctozeptomgoddard: indeed it seems so11:33
mgoddardhrw: flaw in your plan - images won't build without kolla patch11:34
hrwI love deploops11:36
*** pescobar has quit IRC11:37
hrwthere are moments when I do not give a shit and just move to other project/thing11:37
hrwmaybe it is time to just go under desk and improve cabling in my desktop ;D11:37
yoctozeptowe should make another pair of stickers: "don't publish broken images"11:38
yoctozeptowith dead koala logo11:39
openstackgerritMark Goddard proposed openstack/kolla stable/rocky: Fix multiple CI issues  https://review.opendev.org/71702411:41
openstackgerritMark Goddard proposed openstack/kolla stable/stein: Fix multiple CI issues  https://review.opendev.org/71700811:44
hrwyoctozepto: so you suggest that publish jobs should build, deploy all-in-one, test and then publish?11:44
yoctozeptomgoddard: commented, I think we are missing # variants in some places11:44
hrw+2 to both11:45
yoctozeptohrw: yeah, but it's a too-lengthy process11:45
openstackgerritMerged openstack/kolla master: Bump prometheus_openstack_exporter to 1.0.0  https://review.opendev.org/71552111:45
openstackgerritMerged openstack/kolla-ansible master: CI: Always use upper-constraints  https://review.opendev.org/71516411:45
openstackgerritMerged openstack/kolla master: Use unittest.mock instead of PyPI mock  https://review.opendev.org/71707511:45
openstackgerritMerged openstack/kolla-ansible master: Be less confusing about custom Docker registry  https://review.opendev.org/71560111:45
mgoddardhrw, yoctozepto: not a single job, but dependent jobs11:45
mgoddardwe should publish to a master-stage tag, then a deploy job runs against those images11:46
mgoddardif it passes, a third job promotes the images11:46
mgoddardone issue is that it's not very fine grained, so a bad apple spoils the bunch11:46
mgoddardbut that's probably true anyway and old images are better than a broken image11:47
yoctozeptomgoddard: and lengthy11:47
yoctozeptolarge surface of breakage11:48
mgoddardwhat do you mean?11:48
openstackgerritMark Goddard proposed openstack/kolla stable/stein: Fix multiple CI issues  https://review.opendev.org/71700811:50
*** pescobar has joined #openstack-kolla11:50
*** sorin-mihai_ has joined #openstack-kolla11:51
openstackgerritMark Goddard proposed openstack/kolla stable/rocky: Fix multiple CI issues  https://review.opendev.org/71702411:51
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/rocky: CI: Use upper constraints to install clients  https://review.opendev.org/71726211:52
*** sorin-mihai__ has quit IRC11:53
openstackgerritLin PeiWen proposed openstack/kolla-ansible master: manila share container name variable  https://review.opendev.org/71729311:56
yoctozeptomgoddard: network connectivity likes to fail from time to time - if we create large chains of network-based actions then we have more occassions to fail to publish11:57
openstackgerritRadosław Piliszek proposed openstack/kolla stable/rocky: Fix multiple CI issues  https://review.opendev.org/71702412:00
openstackgerritRadosław Piliszek proposed openstack/kolla stable/rocky: Fix multiple CI issues  https://review.opendev.org/71702412:00
openstackgerritRadosław Piliszek proposed openstack/kolla stable/stein: Fix multiple CI issues  https://review.opendev.org/71700812:01
yoctozeptogo, go, go, go12:02
*** bengates_ has joined #openstack-kolla12:02
*** bengates has quit IRC12:03
*** rpittau|bbl is now known as rpitau12:08
*** rpitau is now known as rpittau12:09
*** bengates has joined #openstack-kolla12:11
*** bengates_ has quit IRC12:11
openstackgerritMichal Nasiadka proposed openstack/kolla master: neutron: Add ovn-octavia-driver  https://review.opendev.org/71729612:14
openstackgerritMichal Nasiadka proposed openstack/kolla master: neutron: Add ovn-octavia-driver  https://review.opendev.org/71729612:16
*** pescobar has quit IRC12:22
openstackgerritMichal Nasiadka proposed openstack/kolla master: neutron: Add ovn-octavia-driver  https://review.opendev.org/71729612:25
ykarelmgoddard, can u check comment https://review.opendev.org/#/c/710822/12:25
patchbotpatch 710822 - kolla (stable/train) - Build collectd image for CentOS8 (MERGED) - 3 patch sets12:25
*** heikkine has quit IRC12:29
*** bengates has quit IRC12:39
*** bengates has joined #openstack-kolla12:39
mgoddardykarel: makes sense. Can you propose a patch?12:40
ykarelmgoddard, okk doing12:40
openstackgerrityatin proposed openstack/kolla stable/train: Ignore epel disable  https://review.opendev.org/71730012:44
ykarelmgoddard, check please ^12:44
*** Fl1nt has joined #openstack-kolla12:45
Fl1ntGood afternoon everyone!12:45
Fl1ntquick question, was just reviewing https://opendev.org/openstack/kolla-ansible/src/branch/master/tools/kolla-ansible bash script, and I was wondering:12:46
Fl1ntI know about the ${VAR} notation in bash12:47
Fl1ntbut what that @${VAR} often use mean?12:47
*** scottsol has joined #openstack-kolla12:48
hrwykarel: gentle -1 given12:49
ykarelhrw, ack checking12:49
ykarelok fixing12:50
openstackgerritMerged openstack/kolla master: CI: Add https match for mirror.centos.org to template-overrides  https://review.opendev.org/71654812:53
hrwykarel: commit message needs to be clear message. "where: what\n\nwhy"12:53
ykarelhrw, got it12:54
openstackgerrityatin proposed openstack/kolla stable/train: Collectd: Ignore failure of --disable epel repos  https://review.opendev.org/71730012:54
hrwykarel: I have simple rule: if I need to read code to find out what change does then commit message is wrong12:54
hrwykarel: s/--//?12:54
hrw'collect: ignore failure of disabling epel repos'12:55
openstackgerritBharat Kunwar proposed openstack/kolla stable/train: Bump up magnum and manila versions  https://review.opendev.org/71633912:56
ykarelhrw, okk fixing12:56
mgoddardFl1nt: it's for ansible. ansible-playbook -e @<file>12:56
*** pbing19 has quit IRC12:58
openstackgerrityatin proposed openstack/kolla stable/train: Collectd: ignore failure of disabling epel repos  https://review.opendev.org/71730012:59
*** srini_ has quit IRC13:00
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: kolla-toolbox: use requirements for pip constraints  https://review.opendev.org/71708013:00
hrwpep8 fixes13:00
hrwykarel: +213:00
ykarelhrw, Thanks13:00
openstackgerritPiotr Rabiega proposed openstack/kolla master: add dpdk_telemetry and logparser, use centos-release-opstools for C8  https://review.opendev.org/71592313:01
openstackgerritPierre Riteau proposed openstack/kolla-ansible stable/queens: CI: Use upper constraints to install clients  https://review.opendev.org/71730213:10
Fl1ntoh ok, thx @mgoddard so that's a specific ansible char in order for him to load bunch of vars from file, noticed.13:14
*** sorin-mihai_ has quit IRC13:18
*** sorin-mihai_ has joined #openstack-kolla13:21
hrwhttps://review.opendev.org/#/c/713135 anyone?13:23
patchbotpatch 713135 - kolla - CI: Avoid tox - 33 patch sets13:23
*** k_mouza has quit IRC13:26
openstackgerritMerged openstack/kolla-ansible master: Use unittest.mock instead of PyPI mock  https://review.opendev.org/71707313:29
openstackgerritPierre Riteau proposed openstack/kolla-ansible stable/queens: CI: Use upper constraints to install clients  https://review.opendev.org/71730213:40
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: CI: Avoid tox  https://review.opendev.org/71313513:47
hrwrebased13:47
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Remove default(omit) from openstack_cacert in templates  https://review.opendev.org/71730813:49
openstackgerritMerged openstack/kolla master: Revert "openstack-base: pin setuptools < 46 to get horizon working"  https://review.opendev.org/71516613:53
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: CI: Ansiblify setup of builders  https://review.opendev.org/71520214:00
hrwrebased14:00
hrwbbl14:01
*** noxoid has joined #openstack-kolla14:14
*** Chaserjim has quit IRC14:16
*** TrevorV has joined #openstack-kolla14:18
*** cdearborn has joined #openstack-kolla14:25
*** EmilienM is now known as EvilienM14:40
openstackgerritMerged openstack/kolla stable/stein: Fix multiple CI issues  https://review.opendev.org/71700814:42
hrwso which branches we have working now?14:45
*** pbing19 has joined #openstack-kolla14:48
*** Chaserjim has joined #openstack-kolla14:49
openstackgerritPierre Riteau proposed openstack/kayobe master: Prevent openrc files from using wrong OS_CACERT value  https://review.opendev.org/71731414:49
*** ykarel is now known as ykarel|away14:52
*** Chaserjim has quit IRC14:57
openstackgerritPierre Riteau proposed openstack/kolla-ansible stable/queens: DNM: Try using mirrors of the legacy YUM Docker repository  https://review.opendev.org/71727615:06
openstackgerritPierre Riteau proposed openstack/kolla-ansible stable/queens: CI: Use upper constraints to install clients  https://review.opendev.org/71730215:06
*** klindgren_ has quit IRC15:06
*** klindgren has joined #openstack-kolla15:07
*** e0ne has quit IRC15:12
*** yoctozepto has quit IRC15:17
*** cah_link has quit IRC15:17
openstackgerritPierre Riteau proposed openstack/kayobe master: Prevent openrc files from using wrong OS_CACERT value  https://review.opendev.org/71731415:18
*** yoctozepto8 has joined #openstack-kolla15:18
*** Wasaac has quit IRC15:18
*** Wasaac has joined #openstack-kolla15:19
*** hjensas is now known as hjensas|afk15:21
*** happyhemant has joined #openstack-kolla15:26
*** k_mouza has joined #openstack-kolla15:27
*** k_mouza has quit IRC15:31
*** igordc has joined #openstack-kolla15:32
*** k_mouza has joined #openstack-kolla15:34
openstackgerritMerged openstack/kolla stable/rocky: Fix multiple CI issues  https://review.opendev.org/71702415:38
*** JamesBenson has joined #openstack-kolla15:38
*** cah_link has joined #openstack-kolla15:40
*** rpittau is now known as rpittau|afk15:43
openstackgerritMarcin Juszkiewicz proposed openstack/kolla stable/rocky: Disable Rocky periodic publishing jobs  https://review.opendev.org/71513615:46
hrwrebased15:46
hrwhttps://review.opendev.org/#/c/713135/ anyone? so https://review.opendev.org/715202 will merge too15:49
patchbotpatch 713135 - kolla - CI: Avoid tox - 34 patch sets15:49
patchbotpatch 715202 - kolla - CI: Ansiblify setup of builders - 9 patch sets15:49
openstackgerritMarcin Juszkiewicz proposed openstack/kolla stable/stein: Fix invalid test in cinder data migration script  https://review.opendev.org/71516315:53
hrwrebased15:53
hrwso we have fixed rocky. and fixed stein15:54
hrwtrain needs stein images so need some +2+W on monday. then we need to publish working images to get master upgrade jobs working. Right?15:55
*** bengates has quit IRC15:55
born2bakeHi, I am wondering why cinder-volume is failing for me. Kolla-ansible 9.0, train, ubuntu, sources: enable_cinder: "yes", enable_cinder_backup: "yes", using external ceph - cinder_backend_ceph: "yes"15:57
born2bakenova, glance are working fine15:57
born2bakecinder-volume.log ERROR cinder.cmd.volume [-] Configuration for cinder-volume does not specify "enabled_backends". Using DEFAULT section to configure drivers is not supported since Ocata.15:57
hrwno idea15:59
hrwbut I do not deploy15:59
born2bakehow can I debug it if container does not show any logs?16:01
*** e0ne has joined #openstack-kolla16:02
born2bake2020-04-03 17:54:58.589 7 WARNING cinder.scheduler.filter_scheduler [req-53f559bd-225f-4820-ae33-2f26aca86bcd e10497c216a94c23a881d4ab14618da3 7d971ea300594b0d933389e8dcc19276 - default default] No weighed backend found for volume with properties: {'id': '5072a931-765d-49e3-8519-573c9d0c23da', 'name': '__DEFAULT__', 'description': 'Default Volume Type', 'is_public': True, 'projects': [], 'extra_specs': {}, 'qos_specs_id': None,16:03
born2bake'created_at': '2020-04-03T12:59:13.000000', 'updated_at': '2020-04-03T12:59:13.000000', 'deleted_at': None, 'deleted': False}16:03
*** sorin-mihai_ has quit IRC16:03
*** sorin-mihai_ has joined #openstack-kolla16:03
born2bakeseems like similar issue https://bugs.launchpad.net/openstack-ansible/+bug/176823116:04
openstackLaunchpad bug 1768231 in openstack-ansible "Cinder filter scheduler not enabled with multiple storage back ends" [Medium,In progress]16:04
born2bakebut I do not have multiple storage back end...16:04
*** jimcrowleyibm has joined #openstack-kolla16:09
*** dking_desktop has joined #openstack-kolla16:12
openstackgerritMark Goddard proposed openstack/kayobe master: Avoid unconditional fact gathering  https://review.opendev.org/71631516:14
mgoddardhrw: train images are fine because we use upper-constraints16:15
mgoddardwaiting on rocky & stein images to fix rocky -> train16:15
born2bakeokay it seems like kolla's automatic external ceph support with cinder is broken. I followed stein guide and created cinder.conf files manually and its working now.16:16
openstackgerritRadosław Piliszek proposed openstack/kolla stable/train: Collectd: ignore failure of disabling epel repos  https://review.opendev.org/71730016:22
openstackgerritMark Goddard proposed openstack/kayobe master: Use Ansible connection reset support in docker role  https://review.opendev.org/71733216:31
*** evrardjp has quit IRC16:36
*** evrardjp has joined #openstack-kolla16:37
*** skramaja_ has quit IRC16:40
openstackgerritPierre Riteau proposed openstack/kolla-ansible stable/queens: CI: Use upper constraints to install clients  https://review.opendev.org/71730216:41
yoctozepto8mgoddard: https://review.opendev.org/717310 oslo running away from mock drop ;/16:42
patchbotpatch 717310 - oslotest - Revert "Switch to unittest.mock from mock" - 1 patch set16:42
mgoddardyoctozepto8: not too surprising :)16:43
yoctozepto8mgoddard: well, yeah16:43
*** yoctozepto8 is now known as yoctozepto16:43
*** gfidente is now known as gfidente|afk16:43
yoctozeptoI wonder why bouncer failed to change nick16:43
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Avoid tox (and clean up gate setup)  https://review.opendev.org/71313416:46
*** sorin-mihai_ has quit IRC16:52
*** sorin-mihai has joined #openstack-kolla16:55
*** Tony31 has quit IRC16:56
*** sorin-mihai has quit IRC16:57
*** scottsol has quit IRC17:01
*** sorin-mihai has joined #openstack-kolla17:01
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: Avoid tox (and clean up gate setup)  https://review.opendev.org/71313417:05
*** ktibi has quit IRC17:11
openstackgerritRadosław Piliszek proposed openstack/kolla master: neutron: Add ovn-octavia-driver  https://review.opendev.org/71729617:16
cosmicsoundis this https://docs.openstack.org/kolla-ansible/latest/reference/storage/external-ceph-guide.html supposed to work on train also with the variables in globals.yml, does this mean the .conf files in /etk/kolla/config are not needed ?17:21
*** Fl1nt has quit IRC17:22
*** jbadiapa has quit IRC17:29
*** JangwonLee__ has quit IRC17:33
*** JangwonLee__ has joined #openstack-kolla17:34
*** k_mouza has quit IRC17:40
yoctozeptocosmicsound: no, for train it is different17:41
yoctozeptoswitch 'latest' to 'train'17:41
*** dking_desktop has left #openstack-kolla17:45
*** dking_desktop has joined #openstack-kolla17:45
cdearbornhey folks, I work on the Dell iDRAC driver in the Ironic project.  We're working on transitioning to the Redfish protocol and have started using the Sushy project to do so.  We have a new plugin to Sushy called sushy-oem-idrac (https://opendev.org/x/sushy-oem-idrac) that we need to include in the container that runs ironic-conductor.  Any idea what needs to be done to make that happen?  I really know17:48
cdearbornvirtually nothing about Kolla.17:48
hrwcdearborn: is it present in requirements.txt?17:52
cdearbornhrw: ironic has a driver-requirements.txt, and it is present there.  not sure if that is sufficient...17:53
cdearbornhttps://github.com/openstack/ironic/blob/master/driver-requirements.txt#L2317:54
hrwlet me check17:55
hrwbuilding image17:55
cdearbornhrw: thx!17:55
*** sean-k-mooney has quit IRC17:56
cosmicsoundceph_pool = gnocchi is not needed anymore in gnocchi/18:01
*** cah_link1 has joined #openstack-kolla18:07
*** cah_link has quit IRC18:09
*** cah_link1 is now known as cah_link18:09
yoctozeptocosmicsound: it gets simpler in ussuri, yeah; less conditionals all around18:11
cosmicsound+118:14
*** rmart04 has quit IRC18:38
openstackgerritMerged openstack/kolla stable/stein: Fix invalid test in cinder data migration script  https://review.opendev.org/71516318:45
*** k_mouza has joined #openstack-kolla19:06
cosmicsoundso qcow2 is for images is not recommended in ceph19:13
cosmicsoundrather raw19:13
kplantcorrect19:13
cosmicsoundnoted that downn19:16
kplantiirc you can still upload qcow2 images to glance19:17
cosmicsoundive oticed19:17
kplanthowever when you create an instance, it will copy the qcow2 into a raw container19:17
kplantso you lose the benefit of ceph zero byte copies19:17
kplantand it's much slower19:17
cosmicsoundthanks for the explain19:18
cosmicsoundmakes sense19:18
cosmicsoundil make sure all are raw by default19:18
kplantthe other thing to be aware of is, if you do upload raw images into glance19:18
cosmicsoundits weird enough i cant delete images now19:18
kplantif you haev images deployed with that image, you can't delete the image from glance19:18
cosmicsound:D19:18
kplantsince they'll depend on it19:18
cosmicsoundlol19:18
kplantwhat you can do, is inflate all of the instance volumes19:18
cosmicsoundjust as i wrote i cant delete images19:19
kplantthen you can delete the glance image19:19
cosmicsoundfunny you mentioned this19:19
kplantyeah it kind of sucks19:19
cosmicsoundinflate the volume?19:19
cosmicsoundhmm not sure i got to that chapter19:19
cosmicsoundis this ceph action or openstack19:19
kplantthat's a ceph specific thing19:19
cosmicsoundok19:19
cosmicsoundlooks like a pool stuff19:19
kplantso if you don't want to deal with that, you can upload qcow2s19:19
kplantbut it'll be less efficient19:19
*** k_mouza has quit IRC19:21
cosmicsoundi dont have any instaces deployed with an image19:22
cosmicsoundand i still cant delete it19:22
cosmicsoundwill look intot hat deflate actions19:22
openstackgerritDincer Celik proposed openstack/kolla master: Upgrade RabbitMQ to 3.8  https://review.opendev.org/69479919:30
kplantthat's odd19:40
kplantdo you have any volumes hanging around19:40
kplantyou can delete a raw image with a ceph backend if there are no deps19:40
openstackgerritMerged openstack/kolla master: add dpdk_telemetry and logparser, use centos-release-opstools for C8  https://review.opendev.org/71592319:48
*** TrevorV has quit IRC19:54
cosmicsoundi think there are volumes hanging19:59
cosmicsoundnot sure19:59
cosmicsoundif they are they arent visible in horizon19:59
openstackgerritMerged openstack/kolla master: Update hacking for Python3  https://review.opendev.org/71669120:03
*** sorin-mihai has quit IRC20:03
*** sorin-mihai has joined #openstack-kolla20:04
openstackgerritAndreas Jaeger proposed openstack/kolla-ansible master: Update hacking for Python3  https://review.opendev.org/71738220:13
*** jimcrowleyibm has quit IRC20:13
*** sorin-mihai_ has joined #openstack-kolla20:36
*** sorin-mihai has quit IRC20:38
*** e0ne has quit IRC20:38
*** sorin-mihai__ has joined #openstack-kolla20:43
*** sorin-mihai_ has quit IRC20:46
*** sorin-mihai has joined #openstack-kolla20:49
*** sorin-mihai has quit IRC20:50
*** sorin-mihai has joined #openstack-kolla20:51
*** sorin-mihai__ has quit IRC20:51
*** sorin-mihai_ has joined #openstack-kolla20:53
*** sorin-mihai has quit IRC20:55
*** EvilienM is now known as EmilienM21:07
*** ktibi has joined #openstack-kolla21:07
born2bakemnasiadka Hi, I still feel like you migh help me with that one :) - https://bugs.launchpad.net/kolla-ansible/+bug/1846476 ; I ve added heat resource-list/show output here - https://bugs.launchpad.net/kolla-ansible/+bug/1846476/comments/821:08
openstackLaunchpad bug 1846476 in kolla-ansible "K8s cluster deployment is stuck at kube_cluster_deploy OS::Heat::SoftwareDeployment" [Undecided,New]21:08
*** ktibi has quit IRC21:11
*** KeithMnemonic has quit IRC21:34
*** k_mouza has joined #openstack-kolla21:43
*** k_mouza has quit IRC21:47
*** k_mouza has joined #openstack-kolla21:48
*** JamesBenson has quit IRC21:50
*** gfidente|afk has quit IRC21:53
*** born2bake has quit IRC22:04
*** vedup19 has joined #openstack-kolla22:16
*** pbing19 has quit IRC22:18
*** vedup19 has quit IRC22:31
*** chrizl has quit IRC22:32
*** igordc has quit IRC22:36
*** igordc has joined #openstack-kolla22:36
*** chrizl has joined #openstack-kolla22:43
*** negronjl has quit IRC23:17
*** negronjl has joined #openstack-kolla23:19
hrwcdearborn: INFO:kolla.common.utils.ironic-conductor:Successfully installed ansible-2.9.6 ironic-staging-drivers-0.12.0 proliantutils-2.9.4 pyasn1-0.4.8 pycryptodomex-3.9.7 pyghmi-1.5.13 pysmi-0.3.4 pysnmp-4.4.12 python-dracclient-3.2.0 python-ibmcclient-0.1.0 python-scciclient-0.8.1 python-xclarityclient-0.1.7 sushy-3.2.0 sushy-oem-idrac-0.0.223:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!