| *** ivve has joined #openstack-kolla | 00:24 | |
| *** schwicht has joined #openstack-kolla | 00:28 | |
| *** Sravan has joined #openstack-kolla | 00:32 | |
| *** schwicht has quit IRC | 00:32 | |
| *** cah_link has joined #openstack-kolla | 00:55 | |
| *** Sravan has quit IRC | 00:55 | |
| *** Sravan has joined #openstack-kolla | 00:57 | |
| *** guozijn has joined #openstack-kolla | 01:16 | |
| *** cah_link has quit IRC | 01:27 | |
| *** unicell has joined #openstack-kolla | 01:32 | |
| *** openstackgerrit has joined #openstack-kolla | 01:47 | |
| openstackgerrit | sunguangning proposed openstack/kolla-ansible stable/stein: Spelling mistake in manila.conf.j2 https://review.opendev.org/662661 | 01:47 |
|---|---|---|
| *** owalsh has quit IRC | 01:49 | |
| *** whoami-rajat has quit IRC | 01:54 | |
| *** owalsh has joined #openstack-kolla | 01:57 | |
| *** Sravan has quit IRC | 02:05 | |
| *** Sravan has joined #openstack-kolla | 02:15 | |
| *** owalsh_ has joined #openstack-kolla | 02:18 | |
| *** owalsh has quit IRC | 02:20 | |
| *** owalsh_ is now known as owalsh | 02:56 | |
| guozijn | morning | 03:27 |
| guozijn | Because the variable ‘is_ceph’ still in stein, upgrate jobs failed. | 03:30 |
| *** Sravan has quit IRC | 03:32 | |
| guozijn | Why this patch only in stable/stein: https://opendev.org/openstack/kolla-ansible/commit/f50e331e1a3b377b0530fa3f99b542abef0a6005 | 03:47 |
| *** Sravan has joined #openstack-kolla | 03:50 | |
| *** Sravan has quit IRC | 04:03 | |
| *** guozijn has quit IRC | 04:13 | |
| *** Sravan has joined #openstack-kolla | 04:14 | |
| *** whoami-rajat has joined #openstack-kolla | 04:19 | |
| *** Sravan has quit IRC | 04:26 | |
| *** altlogbot_2 has quit IRC | 04:36 | |
| *** altlogbot_0 has joined #openstack-kolla | 04:37 | |
| *** janki has joined #openstack-kolla | 04:38 | |
| *** Sravan has joined #openstack-kolla | 04:52 | |
| *** sshnaidm is now known as sshnaidm|off | 05:03 | |
| *** radeks has joined #openstack-kolla | 05:05 | |
| *** Sravan has quit IRC | 05:06 | |
| *** Sravan has joined #openstack-kolla | 05:08 | |
| *** pcaruana has joined #openstack-kolla | 05:08 | |
| *** Sravan has quit IRC | 05:12 | |
| *** guozijn has joined #openstack-kolla | 05:48 | |
| *** skramaja has joined #openstack-kolla | 05:57 | |
| *** jbadiapa has joined #openstack-kolla | 06:02 | |
| *** dpawlik has joined #openstack-kolla | 06:16 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Add CentOS/source aarch64 experimental job https://review.opendev.org/662456 | 06:19 |
| *** cah_link has joined #openstack-kolla | 06:20 | |
| *** ivve has quit IRC | 06:28 | |
| *** cah_link has quit IRC | 06:28 | |
| *** unicell has quit IRC | 06:43 | |
| *** luksky has joined #openstack-kolla | 07:12 | |
| *** guozijn has quit IRC | 07:13 | |
| *** luksky has quit IRC | 07:23 | |
| *** ivve has joined #openstack-kolla | 07:36 | |
| *** luksky has joined #openstack-kolla | 07:39 | |
| *** _Cyclone_ has quit IRC | 07:41 | |
| *** hamzy has quit IRC | 07:41 | |
| *** rm_work has quit IRC | 07:41 | |
| *** evrardjp has quit IRC | 07:41 | |
| *** irclogbot_0 has quit IRC | 07:44 | |
| *** irclogbot_1 has joined #openstack-kolla | 07:45 | |
| *** dasp has quit IRC | 07:47 | |
| *** sm806 has quit IRC | 07:47 | |
| *** portdirect has quit IRC | 07:47 | |
| *** randyl has quit IRC | 07:47 | |
| *** ab-a has quit IRC | 07:47 | |
| *** egonzalez has quit IRC | 07:47 | |
| *** dasp has joined #openstack-kolla | 07:52 | |
| *** sm806 has joined #openstack-kolla | 07:52 | |
| *** portdirect has joined #openstack-kolla | 07:52 | |
| *** randyl has joined #openstack-kolla | 07:52 | |
| *** ab-a has joined #openstack-kolla | 07:52 | |
| *** egonzalez has joined #openstack-kolla | 07:52 | |
| *** _Cyclone_ has joined #openstack-kolla | 07:53 | |
| *** evrardjp has joined #openstack-kolla | 07:53 | |
| *** hamzy has joined #openstack-kolla | 07:53 | |
| *** rm_work has joined #openstack-kolla | 07:53 | |
| *** ktibi has joined #openstack-kolla | 07:56 | |
| *** yoctozepto has quit IRC | 07:59 | |
| *** yoctozepto has joined #openstack-kolla | 07:59 | |
| *** yoctozepto has quit IRC | 08:03 | |
| *** dougsz has joined #openstack-kolla | 08:08 | |
| *** yoctozepto has joined #openstack-kolla | 08:09 | |
| *** priteau has joined #openstack-kolla | 08:15 | |
| *** jchhatbar has joined #openstack-kolla | 08:20 | |
| *** janki has quit IRC | 08:20 | |
| *** k_mouza has joined #openstack-kolla | 08:22 | |
| *** chason has joined #openstack-kolla | 08:22 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Test Ceph upgrade in CI https://review.opendev.org/658132 | 08:24 |
| *** k_mouza has quit IRC | 08:27 | |
| *** k_mouza has joined #openstack-kolla | 08:27 | |
| mgoddard | bline: I think the qdrouterd image is for AMQP? | 08:28 |
| mgoddard | bline: set om_rpc_transport='amqp'? | 08:29 |
| hrw | morning | 08:30 |
| *** yoctozepto_ has joined #openstack-kolla | 08:30 | |
| hrw | discovery of the week: http://logs.openstack.org/56/662456/ like urls - easy access to previous logs for job | 08:30 |
| *** yoctozepto has quit IRC | 08:30 | |
| *** happyhemant has joined #openstack-kolla | 08:31 | |
| *** jchhatbar has quit IRC | 08:40 | |
| *** janki has joined #openstack-kolla | 08:40 | |
| mgoddard | morning hrw | 08:50 |
| priteau | mgoddard: Did you see the gate failures due to `is_ceph` being undefined in globals-default.j2? | 08:50 |
| mgoddard | priteau: yeah, https://review.opendev.org/658132 should fix it but was blocked by a Depends-On which I have now removed | 08:51 |
| priteau | Great, thanks | 08:52 |
| openstackgerrit | Krzysztof Klimonda proposed openstack/kolla-ansible master: Allow disabling insecure API endpoints https://review.opendev.org/548407 | 08:55 |
| *** gfidente has joined #openstack-kolla | 09:01 | |
| *** cah_link has joined #openstack-kolla | 09:06 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Add CI job for ironic https://review.opendev.org/568829 | 09:08 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: create missing apache2 directory on Debian/Ubuntu https://review.opendev.org/661713 | 09:09 |
| hrw | now with sudo | 09:09 |
| yankcrime | bah, i just hit https://bugs.launchpad.net/kolla/+bug/1823989 | 09:09 |
| openstack | Launchpad bug 1823989 in kolla stein "Ubuntu binary deploys are failing on master and stable/stein" [High,Fix committed] - Assigned to Mark Goddard (mgoddard) | 09:09 |
| yankcrime | that's in the RC1 for the stein release of k-a | 09:09 |
| yankcrime | the bug i mean, not the fix | 09:09 |
| yankcrime | serves me right for installing via pip i guess | 09:16 |
| *** luksky has quit IRC | 09:27 | |
| *** guozijn has joined #openstack-kolla | 09:34 | |
| openstackgerrit | Krzysztof Klimonda proposed openstack/kolla-ansible master: Allow disabling insecure API endpoints https://review.opendev.org/548407 | 09:36 |
| openstackgerrit | Krzysztof Klimonda proposed openstack/kolla-ansible master: Allow disabling insecure API endpoints https://review.opendev.org/548407 | 09:45 |
| *** ktibi has quit IRC | 10:15 | |
| *** guozijn has quit IRC | 10:16 | |
| mgoddard | yankcrime: doh. Is this on sausage? | 10:29 |
| yankcrime | yeah | 10:29 |
| yankcrime | also just hit basically the same problem with placement | 10:29 |
| mgoddard | stable/rocky should work | 10:30 |
| mgoddard | did you upgrade? | 10:30 |
| yankcrime | i wanted to upgrade to stein | 10:30 |
| yankcrime | so yeah, too late! | 10:30 |
| mgoddard | ah. We haven't released yet | 10:30 |
| yankcrime | aye, i realise | 10:30 |
| mgoddard | although it's unlikely to change too much from stable/stein for ubuntu binary | 10:30 |
| yankcrime | i thought the rc would be good enough | 10:30 |
| mgoddard | it's quite old now | 10:31 |
| yankcrime | my fault, i should be using a git checkout of that branch as you say instead of pip packages if i want to be a bit closer to the bleeding edge | 10:32 |
| *** dpawlik has quit IRC | 10:54 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Spelling mistake in manila.conf.j2 https://review.opendev.org/662123 | 10:55 |
| *** janki has quit IRC | 10:57 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Add become for watcher config copying over https://review.opendev.org/662136 | 10:57 |
| *** janki has joined #openstack-kolla | 10:58 | |
| hrw | mgoddard: did you asked RDO about py3 plans? | 11:01 |
| *** guozijn has joined #openstack-kolla | 11:01 | |
| mgoddard | hrw: not yet, still working through ptg actions | 11:01 |
| hrw | ok | 11:01 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Fix quickstart for virtual environments https://review.opendev.org/662732 | 11:03 |
| *** schwicht has joined #openstack-kolla | 11:05 | |
| goldyfruit | I still the same error even after the rebase, https://review.opendev.org/#/c/660789 | 11:07 |
| goldyfruit | Anyone has an idea ? | 11:07 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: there is no easy_install in Debian 'buster' https://review.opendev.org/662733 | 11:07 |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Improve registration of Monasca InfluxDB database https://review.opendev.org/656325 | 11:10 |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Disable InfluxDB reporting https://review.opendev.org/656326 | 11:10 |
| *** dpawlik has joined #openstack-kolla | 11:16 | |
| *** kplant has joined #openstack-kolla | 11:18 | |
| hrw | mgoddard: RDO/Train is Centos8 only | 11:20 |
| hrw | https://review.rdoproject.org/etherpad/p/moving-rdo-to-centos8 | 11:21 |
| mnasiadka | so then it's clear | 11:22 |
| hrw | so we will be able to forget about py2 during train ;D | 11:22 |
| *** luksky has joined #openstack-kolla | 11:24 | |
| mnasiadka | mgoddard: any idea what's wrong here? http://logs.openstack.org/89/657989/23/check/kolla-ansible-ubuntu-source-upgrade/75e73bd/job-output.txt.gz#_2019-05-31_18_32_48_151130 | 11:25 |
| mgoddard | hrw: I thought that was the case but wasn't sure why. Thanks for digging it up | 11:26 |
| mgoddard | hrw: at least that simplifies things (sort of) | 11:26 |
| kplant | stein->train will be a fun upgrade | 11:26 |
| hrw | mgoddard: kill centos7, kill oraclelinux. | 11:26 |
| mgoddard | mnasiadka: yeah, it's blocking the gate | 11:26 |
| hrw | kplant: go away ;D | 11:27 |
| mnasiadka | kplant: especially with the fact that Redhat doesn't plan to support upgrade between 7 and 8 | 11:27 |
| mgoddard | mnasiadka: patch working its way through zuul: http://logs.openstack.org/89/657989/23/check/kolla-ansible-ubuntu-source-upgrade/75e73bd/job-o | 11:27 |
| mgoddard | mnasiadka: :o | 11:27 |
| mnasiadka | mgoddard: ok, so I'll kindly wait | 11:27 |
| mgoddard | no upgrade! | 11:27 |
| mgoddard | when you say no support, you mean they won't support it, or its not possible? | 11:28 |
| mgoddard | I guess we can deploy centos8 containers on centos7 host or vice versa | 11:28 |
| mgoddard | then upgrade host OS with a rebuild | 11:28 |
| mgoddard | still, sucks | 11:28 |
| mnasiadka | mgoddard: well, they didn't plan to support it - now I see they support it a bit, but the limitations are interesting: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/upgrading_to_rhel_8/index | 11:29 |
| *** janki has quit IRC | 11:33 | |
| *** schwicht has quit IRC | 11:42 | |
| yankcrime | mgoddard: we're back in business | 11:43 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Set log message field name in Monasca Fluentd plugin https://review.opendev.org/661018 | 11:47 |
| *** dave-mccowan has joined #openstack-kolla | 11:48 | |
| mnasiadka | mgoddard: the kernel difference between C7 and C8 is what bothers me... what if some features are dependent on new kernel? | 11:48 |
| *** ktibi has joined #openstack-kolla | 11:49 | |
| kplant | mnasiadka: if you did it in reverse you avoid that problem. upgrade host from c7->c8, deploy stein c7 containers on c8 host | 11:50 |
| kplant | then upgrade stein->train | 11:50 |
| mnasiadka | kplant: I understand, what if I don't want to upgrade the host? :D | 11:51 |
| kplant | enjoy patching 3.10? :) | 11:52 |
| hrw | mnasiadka: 'on new kernel' or 'on new kernel features'? | 11:53 |
| hrw | mnasiadka: remember that 3.10-rhel != 3.10-mainline | 11:53 |
| mgoddard | yankcrime: \o/ | 11:54 |
| *** dave-mccowan has quit IRC | 11:55 | |
| mnasiadka | hrw: yeah, still 3.10 vs 4.18, even if it's rhel-style - should differ ;) | 11:56 |
| hrw | mnasiadka: you are complaining like someone from previous decade... | 11:57 |
| mnasiadka | hrw: if it comes to me - I don't have a problem, but already my customers are asking those questions... | 11:57 |
| hrw | mnasiadka: you add rack of hw, deploy rhel/centos8 there, deploy train, merge two installations, migrate from old one to new one and redeploy old hosts | 11:57 |
| mnasiadka | hrw: I'm R&D, I haven't ever seen a field deployment or a field upgrade :) | 11:58 |
| hrw | mnasiadka: I am build engineer. no field deployments/upgrade. but that's how it is done nowadays | 11:58 |
| mnasiadka | hrw: yeah - btw nice thread on centos-devel - I think it will be interesting discussion ;) | 11:58 |
| *** janki has joined #openstack-kolla | 11:59 | |
| hrw | mnasiadka: look at all those in-cloud services. you do not upgrade system on cloud-host but add upgraded one to farm | 11:59 |
| hrw | mnasiadka: have you seen my 'centos 9 or 10' note? | 11:59 |
| mnasiadka | hrw: yeah, pets vs cattle kind of a thing | 11:59 |
| mnasiadka | hrw: yeah :) | 11:59 |
| *** guozijn has quit IRC | 12:00 | |
| mnasiadka | hrw: I never liked ,,pets'' approach, but I'm just an engineer ;) | 12:00 |
| hrw | mnasiadka: I am so far from devops that I had to check 'pets or cattle' thing ;D | 12:01 |
| hrw | mnasiadka: my server is a pet. | 12:01 |
| hrw | mnasiadka: called malenstwo (Roo from Winnie the Pooh) as it is small atom machine somewhere | 12:01 |
| hrw | mnasiadka: how goes ansible upgrade? | 12:11 |
| *** janki has quit IRC | 12:13 | |
| *** priteau has quit IRC | 12:15 | |
| *** guozijn has joined #openstack-kolla | 12:15 | |
| *** schwicht has joined #openstack-kolla | 12:18 | |
| *** ktibi has quit IRC | 12:24 | |
| *** schwicht has quit IRC | 12:30 | |
| *** schwicht has joined #openstack-kolla | 12:32 | |
| *** rm_work has quit IRC | 12:43 | |
| *** jschlueter has quit IRC | 12:43 | |
| *** jschlueter has joined #openstack-kolla | 12:43 | |
| *** portdirect has quit IRC | 12:44 | |
| *** priteau has joined #openstack-kolla | 12:44 | |
| *** edleafe has joined #openstack-kolla | 12:46 | |
| *** rm_work has joined #openstack-kolla | 12:47 | |
| mnasiadka | mnasiadka: I'll look into that today, was busy with other things | 12:49 |
| *** schwicht has quit IRC | 12:49 | |
| mnasiadka | ehh | 12:52 |
| mnasiadka | hrw: ^^ | 12:52 |
| mnasiadka | ;) | 12:52 |
| hrw | mnasiadka: thx | 12:53 |
| *** portdirect has joined #openstack-kolla | 12:58 | |
| *** priteau has quit IRC | 13:03 | |
| *** owalsh_ has joined #openstack-kolla | 13:04 | |
| *** rpittau has joined #openstack-kolla | 13:06 | |
| *** owalsh has quit IRC | 13:06 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Add CI job for ironic https://review.opendev.org/568829 | 13:07 |
| *** zul has joined #openstack-kolla | 13:07 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: create missing apache2 directory on Debian/Ubuntu https://review.opendev.org/661713 | 13:11 |
| *** mbanszel_ has joined #openstack-kolla | 13:17 | |
| mbanszel_ | Hello, I wonder if anyone has hit an issue, when deploying kolla with kolla-ansible, during starting keystone apache threads. Sometimes I get this error message: SystemExit: /etc/keystone/fernet-keys/ does not contain keys, use keystone-manage fernet_setup to create Fernet keys. This is reported by some of the apache threads and then all requests to those particular threads end with "ArgsAlreadyParsedError: arguments alrea | 13:20 |
| mbanszel_ | The client then gets 500 response. | 13:20 |
| *** guozijn has quit IRC | 13:21 | |
| mbanszel_ | When I check the keys, they are there and it is enough just to restart the keystone container. It looks like a race condition in starting kolla containers. | 13:22 |
| mnasiadka | interesting | 13:23 |
| mnasiadka | mbanszel_: can you raise a bug and attach some logs there? | 13:24 |
| *** schwicht has joined #openstack-kolla | 13:28 | |
| *** happyhemant has quit IRC | 13:33 | |
| *** randyl has quit IRC | 13:33 | |
| *** randyl has joined #openstack-kolla | 13:34 | |
| *** happyhemant has joined #openstack-kolla | 13:35 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Add CI job for ironic https://review.opendev.org/568829 | 13:35 |
| goldyfruit | The failures I got seems to be related to this commit: https://github.com/openstack/kolla-ansible/commit/f50e331e1a3b377b0530fa3f99b542abef0a6005 | 13:36 |
| *** cah_link has quit IRC | 13:39 | |
| jroll | mgoddard: good PTG summary, thanks for that :) | 13:47 |
| mgoddard | yankcrime: \o/ | 13:47 |
| mgoddard | jroll: seems mostly ironic people that have read it - I got two reviews on the ironic CI job patch :) | 13:48 |
| jroll | heh | 13:48 |
| goldyfruit | I guess ` is_ceph: "{{ 'ceph' in scenario }}" ` requires a default value | 13:57 |
| goldyfruit | Ok, few other reviews are in the same status, I guess the CI has a little glitch. I'll wait :) | 14:05 |
| *** skramaja has quit IRC | 14:07 | |
| *** iclon_ has joined #openstack-kolla | 14:25 | |
| hrw | hm. that 'is_ceph' gets to other jobs too... | 14:26 |
| *** iclon__ has quit IRC | 14:27 | |
| goldyfruit | hrw, yep | 14:28 |
| hrw | where it came from? | 14:44 |
| goldyfruit | https://github.com/openstack/kolla-ansible/commit/f50e331e1a3b377b0530fa3f99b542abef0a6005 | 14:46 |
| goldyfruit | Not in master, in stable/stein | 14:47 |
| *** igordc has joined #openstack-kolla | 14:50 | |
| *** owalsh has joined #openstack-kolla | 14:52 | |
| *** owalsh_ has quit IRC | 14:54 | |
| hrw | mnasiadka: alive? | 15:02 |
| hrw | - is_ceph: "{{ 'ceph' in scenario }}" | 15:02 |
| hrw | + is_ceph: "{{ true if 'ceph' in scenario else false }}" | 15:02 |
| hrw | will it work? | 15:02 |
| mnasiadka | hrw: alive, should work, as long as my brain yaml parser works ;) | 15:02 |
| hrw | ;d | 15:03 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla-ansible stable/stein: Fix 'is_ceph' failure in CI jobs https://review.opendev.org/662783 | 15:04 |
| hrw | mnasiadka: please take a look | 15:04 |
| mnasiadka | hrw: I assumed mgoddard is fixing that, but let’s look :) | 15:05 |
| goldyfruit | hrw, looks good to me, thx :) | 15:06 |
| mgoddard | hrw: yeah, it's fixed in master when https://review.opendev.org/658132 merges | 15:07 |
| mgoddard | it keeps failing, CI seems flaky today :( | 15:07 |
| goldyfruit | \o/ | 15:07 |
| mgoddard | it's in the gate currently | 15:07 |
| hrw | ok | 15:08 |
| *** igordc has quit IRC | 15:08 | |
| *** igordc has joined #openstack-kolla | 15:08 | |
| goldyfruit | hrw, mgoddard I guess when the fix will be merged I'll have to rebase, right ? | 15:09 |
| mgoddard | goldyfruit: not necessarily - zuul merges with HEAD | 15:09 |
| goldyfruit | mgoddard, oh! | 15:09 |
| hrw | goldyfruit: check is on your branch, gate is on HEAD. | 15:09 |
| goldyfruit | :p | 15:10 |
| hrw | 16:20 < hrw> goldyfruit: final zuul is 'rebase on HEAD and test' and some changes were merged which broke your patch | 15:10 |
| hrw | 16:20 < hrw> goldyfruit: you send patch, 'check' is started with branch as you sent. then we vote. then zuul rebases and checks again. | 15:10 |
| *** irclogbot_1 has quit IRC | 15:10 | |
| hrw | friday ^^ :D | 15:10 |
| goldyfruit | Damn it, I skipped this part I guess | 15:10 |
| goldyfruit | Sorry :D | 15:10 |
| *** altlogbot_0 has quit IRC | 15:11 | |
| *** irclogbot_3 has joined #openstack-kolla | 15:11 | |
| goldyfruit | It was test, to see if you were consistent hrw :p | 15:12 |
| *** altlogbot_2 has joined #openstack-kolla | 15:12 | |
| goldyfruit | And you are! | 15:12 |
| goldyfruit | ^^ | 15:12 |
| mgoddard | hrw: I thought check was also tested using the latest HEAD? | 15:12 |
| hrw | goldyfruit: trying to not be pure asshole all the time | 15:12 |
| mgoddard | otherwise is_ceph would not have broken so many jobs | 15:13 |
| hrw | mgoddard: then rebase button would be useless | 15:13 |
| mgoddard | hrw: sort of, but it means you can pull down the patch with current changes | 15:14 |
| mgoddard | hrw: and if we didn't have recheck it would be useful | 15:14 |
| hrw | mgoddard: https://review.opendev.org/#/c/643467/ is from March. it's parent is still from March. | 15:15 |
| mgoddard | hrw: right, but that isn't what zuul uses to test | 15:16 |
| *** priteau has joined #openstack-kolla | 15:17 | |
| mgoddard | hrw: see https://review.opendev.org/#/c/660617/ | 15:17 |
| mgoddard | patch set 4 has check pass, gate fail, check fail | 15:18 |
| mgoddard | what changed is HEAD | 15:18 |
| hrw | ok | 15:18 |
| hrw | I may be wrong | 15:18 |
| hrw | mgoddard: gate check does rebase. GATE. | 15:19 |
| hrw | and in 660617 you have: check +1, gate -2, gate -1 | 15:19 |
| hrw | check, gate, check sorry | 15:20 |
| mgoddard | gate won't change your commit ID - it rebases then throws it away if it fails | 15:20 |
| *** whoami-rajat has quit IRC | 15:20 | |
| hrw | and second check could failed due to is_ceph change in stein but not in master | 15:21 |
| mgoddard | true | 15:22 |
| hrw | which confirms 'check=branch as is, gate=HEAD rebase' | 15:22 |
| mgoddard | not necessarily | 15:22 |
| hrw | will not argue ;D | 15:23 |
| mgoddard | when this fix finally merges, let's run recheck on a patch and find out | 15:23 |
| *** michaelbarkdoll has quit IRC | 15:24 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Install pymysql for python3 in Ubuntu binary https://review.opendev.org/662788 | 15:27 |
| openstackgerrit | Merged openstack/kolla-ansible master: Test Ceph upgrade in CI https://review.opendev.org/658132 | 15:28 |
| *** mbanszel_ has quit IRC | 15:29 | |
| hrw | uf. | 15:29 |
| mgoddard | We're back in the game. | 15:29 |
| *** ivve has quit IRC | 15:30 | |
| goldyfruit | Yeah ! | 15:32 |
| *** priteau has quit IRC | 15:32 | |
| *** schwicht has quit IRC | 15:33 | |
| *** dasp has quit IRC | 15:35 | |
| hrw | hm. py3 patch is at 24th edition ;D | 15:36 |
| goldyfruit | mnasiadka, could you please +2 this review https://review.opendev.org/#/c/661975/ ? I already got one, it could be nice to have this merged in the same time as the Qinling Ansible role | 15:36 |
| hrw | goldyfruit: let it flow | 15:38 |
| goldyfruit | hrw, cool! | 15:38 |
| hrw | https://review.opendev.org/#/c/661713 -> https://review.opendev.org/#/c/642375/ -> py3 in debian/ubuntu done | 15:47 |
| *** zbr has quit IRC | 15:47 | |
| hrw | both got rechecks | 15:47 |
| *** dasp has joined #openstack-kolla | 15:53 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Add CI job for ironic https://review.opendev.org/568829 | 15:56 |
| *** zbr has joined #openstack-kolla | 15:56 | |
| *** rpittau is now known as rpittau|afk | 16:28 | |
| *** rouk has joined #openstack-kolla | 16:33 | |
| *** dosaboy has joined #openstack-kolla | 16:36 | |
| *** ivve has joined #openstack-kolla | 16:37 | |
| *** k_mouza has quit IRC | 16:46 | |
| *** unicell has joined #openstack-kolla | 16:49 | |
| *** luksky has quit IRC | 16:50 | |
| *** unicell has quit IRC | 16:53 | |
| *** gfidente is now known as gfidente|afk | 16:55 | |
| *** dougsz has quit IRC | 17:03 | |
| *** whoami-rajat has joined #openstack-kolla | 17:03 | |
| *** schwicht has joined #openstack-kolla | 17:24 | |
| rouk | anyone have any ideas why a kolla deployment would have timeouts to all ports on haproxy until haproxy is restarted, then times out after another 5 mins | 17:52 |
| kplant | O_O | 17:53 |
| goldyfruit | The VIP is ok ? | 17:53 |
| kplant | do you see your vip assigned? | 17:53 |
| rouk | i really dont know what to do, i updated all my containers on my repo, then did an upgrade, just going up container versions within stein | 17:54 |
| rouk | vip working as intended | 17:54 |
| rouk | backend services are reachable directly, bounce the vip to a different node, same issue. | 17:54 |
| goldyfruit | it is moving across the nodes randomly ? | 17:54 |
| rouk | nope | 17:54 |
| goldyfruit | No issue with VRRP packets ? | 17:54 |
| rouk | seems fine, i only need to restart haproxy to bring things back up, but horizon can never get past login, /projects 504s | 17:55 |
| rouk | api works perfectly for 5 minutes after every haproxy restart | 17:55 |
| kplant | if you do a netstat -ntpl do you still see haproxy listening on those ports? | 17:56 |
| goldyfruit | rouk, what is the max_connections on MySQL ? | 17:57 |
| goldyfruit | How many connections currently opened on MySQL ? | 17:57 |
| rouk | hey, even horizon loaded for a moment. i have the default mysql setting for kolla... gotta check | 17:57 |
| rouk | 10k max connections | 17:59 |
| goldyfruit | how many open ? | 17:59 |
| rouk | 638 | 18:01 |
| goldyfruit | did you just restart ? | 18:01 |
| rouk | its been alive for.... about 10 minutes now without failing. No, last haproxy restart was 10 mins ago, last mariadb restart was over an hour ago | 18:02 |
| goldyfruit | ok | 18:02 |
| goldyfruit | nothing in haproxy logs ? | 18:02 |
| rouk | just haproxy was breaking, all services behind it seemed fine, only a haproxy restart "fixed" the issue. haproxy status all green the entire time. | 18:03 |
| goldyfruit | nothing in dmesg ? | 18:03 |
| rouk | dmesg on the controllers empty for the past 11 days | 18:05 |
| rouk | quiet boxes | 18:05 |
| rouk | but... everything is healthy now, and i have no idea why, all i did was restart haproxy every 10 minutes for an hour. | 18:06 |
| goldyfruit | Really nothing in /var/lib/docker/volumes/kolla_logs/_data/haproxy/haproxy_latest.*.log ? | 18:06 |
| openstackgerrit | Merged openstack/kolla-ansible master: Enable kolla-ansible to use a custom metrics files with Cloudkitty https://review.opendev.org/658422 | 18:06 |
| goldyfruit | Nothing without the components about the database ? | 18:06 |
| goldyfruit | within* | 18:07 |
| rouk | components all complained about db loss, but only due to vip issues | 18:07 |
| rouk | just reachability complaints | 18:07 |
| rouk | 0 error level errors in haproxy logs | 18:08 |
| *** kplant has quit IRC | 18:08 | |
| rouk | just healthy infos | 18:08 |
| goldyfruit | and about the keepalived log ? | 18:09 |
| rouk | where is that? doesnt seem to be exposed in kolla_logs | 18:09 |
| goldyfruit | docker logs keepalived | 18:09 |
| rouk | guess that works | 18:10 |
| rouk | dont see any stealing or flapping in vrrp | 18:11 |
| rouk | openstack is still healthy now, after 2 hours of prod downtime due to... sorcery i guess? not sure how to explain this one. | 18:12 |
| goldyfruit | There are no server/device on the network with the same IP as your VIP ? | 18:12 |
| rouk | nope, the network has been healthy for many weeks without issue, i ran an upgrade to new container versions, everything was happy, but haproxy started dropping packets, failed the vip onto another controller, fixed the issue for 5 mins, dropped packets again, then restarted haproxy every 5 minutes for 35 mins, now its happy. | 18:14 |
| rouk | if it was an ip conflict dropping http traffic... it was very coincidental | 18:14 |
| goldyfruit | :/ | 18:15 |
| rouk | i just want to know what i did wrong... heh | 18:15 |
| rouk | perhaps there was still a transient haproxy container on the nodes that didnt actually shut down? didnt check the process list. | 18:16 |
| goldyfruit | Maybe | 18:16 |
| rouk | well, thanks for the rubber ducking, im going to stare into a corner and think about what happened. | 18:18 |
| goldyfruit | Good luck :) | 18:19 |
| *** radeks_ has joined #openstack-kolla | 18:19 | |
| *** radeks has quit IRC | 18:22 | |
| bline | hello, is there a way to have a custom file synced with config? Like to add a CA.pem to barbican | 18:23 |
| bline | or maybe a way to override config.json.j2 and add the file there? | 18:23 |
| goldyfruit | bline, you could override /etc/kolla/barbican-*/config.json on the controller nodes | 18:28 |
| goldyfruit | But when a reconfigure will be triggered you will erase your manual changes | 18:28 |
| openstackgerrit | Merged openstack/kolla-ansible master: Add support for ceilometer custom event yaml https://review.opendev.org/660702 | 18:29 |
| bline | yeah, waa hopping to keep the whole update process (reconfigure) in tact | 18:35 |
| bline | it is a nice system | 18:36 |
| goldyfruit | bline, why do you need to add the CA.pem ? | 18:36 |
| goldyfruit | For a barbican option ? | 18:37 |
| bline | was trying to get ssl working for swarm; guess it uses barbican to store the cert; when I setup barbican (everything is setup to run ssl) it just dies trying to make some api call with ssl errors | 18:38 |
| rouk | your users would be loading certs into barbican, you wouldnt be seeding them during deploy... | 18:38 |
| bline | looked into it and it seems to have a CAfile option or some such but there doesn't appear to be a way to add a ca file to sync even though I can add the config option | 18:39 |
| bline | rouk, I was just trying to fix the SSL api call errors | 18:39 |
| bline | guess i could look for an insecure option | 18:40 |
| rouk | well, if you want ssl to the openstack vip, thats not a barbican thing | 18:40 |
| *** luksky has joined #openstack-kolla | 18:43 | |
| *** altlogbot_2 has quit IRC | 18:43 | |
| *** altlogbot_1 has joined #openstack-kolla | 18:44 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Make kolla-ansible support extra volumes https://review.opendev.org/651143 | 18:51 |
| *** radeks__ has joined #openstack-kolla | 19:01 | |
| *** radeks_ has quit IRC | 19:04 | |
| *** altlogbot_1 has quit IRC | 19:21 | |
| *** my_nd has joined #openstack-kolla | 19:22 | |
| my_nd | Hi there, do you know if there are limitations or specials configuration for installing Openstack Rocky using kolla-ansible in Ubuntu Server 18.04 LTS with processors AMD Opteron(tm) 6380 working in compute nodes and KVM as a hypervisor. | 19:24 |
| openstackgerrit | Merged openstack/kolla master: Use the default policy.json for Qinling https://review.opendev.org/661975 | 19:25 |
| *** altlogbot_1 has joined #openstack-kolla | 19:26 | |
| goldyfruit | Opteron 63xx supports KVM | 19:28 |
| goldyfruit | my_nd, it should work | 19:29 |
| rouk | what branch are the stein containers built on? patches went out to stable/stein for octavia last week, but are not in the most recent images built in the last day? | 19:34 |
| rouk | currently loadbalancers are non-functional via horizon in stein under kolla | 19:35 |
| openstackgerrit | Merged openstack/kolla-ansible master: Add support for ceilometer custom pipeline.yaml https://review.opendev.org/660617 | 19:35 |
| goldyfruit | rouk, octavia-dashboard-3.0.0.tar.gz and octavia-4.0.0.tar.gz | 19:37 |
| goldyfruit | https://github.com/openstack/kolla/blob/stable/stein/kolla/common/config.py#L669-L672 | 19:37 |
| rouk | what container are they packed into? | 19:37 |
| goldyfruit | docker ps -a -f name=octavia | 19:41 |
| rouk | yeah, theres worker, housekeeping, health-manager, api, no dashboard, must be packed in with the horizon container | 19:42 |
| rouk | seems that octavia doesnt have a versioned release with the breakages fixed yet... cant make LBs in stein via horizon | 19:43 |
| goldyfruit | Yes the extensions are in Horizon container | 19:43 |
| my_nd | thanks goldyfruit | 19:54 |
| openstackgerrit | Alex Schultz proposed openstack/kolla master: Remove tripleo-ui https://review.opendev.org/662846 | 19:58 |
| *** k_mouza has joined #openstack-kolla | 20:03 | |
| *** hamzy_ has joined #openstack-kolla | 20:06 | |
| *** radeks__ has quit IRC | 20:07 | |
| *** k_mouza has quit IRC | 20:07 | |
| *** evrardjp_ has joined #openstack-kolla | 20:09 | |
| rouk | when does kolla bump the minor version on components on dockerhub? i can build my own, just wondering though | 20:12 |
| goldyfruit | I don't know when it's bumbped | 20:13 |
| goldyfruit | But yes you can build your own | 20:13 |
| goldyfruit | https://docs.openstack.org/kolla/latest/admin/image-building.html | 20:13 |
| rouk | yep, i build my own neutron, but i prefer to keep the list short. | 20:14 |
| goldyfruit | If you git clone kolla master then automatically, you will get the master tarbal | 20:14 |
| goldyfruit | http://tarballs.openstack.org/octavia/ | 20:14 |
| goldyfruit | http://tarballs.openstack.org/octavia/octavia-master.tar.gz | 20:14 |
| *** _Cyclone_ has quit IRC | 20:15 | |
| *** hamzy has quit IRC | 20:15 | |
| *** evrardjp has quit IRC | 20:15 | |
| *** _Cyclone_ has joined #openstack-kolla | 20:15 | |
| rouk | http://tarballs.openstack.org/octavia-dashboard/octavia-dashboard-3.0.1.tar.gz contains the fixes that make LB operation via gui not dead | 20:15 |
| *** radeks has joined #openstack-kolla | 20:17 | |
| goldyfruit | In kolla-build.conf you should have a section horizon-plugin-octavia-dashboard | 20:17 |
| *** schwicht has quit IRC | 20:18 | |
| *** dpawlik has quit IRC | 20:21 | |
| rouk | yeah, ill build it and get it going. | 20:23 |
| *** spiette_ has quit IRC | 20:24 | |
| *** spiette has joined #openstack-kolla | 20:26 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Adds Qinling Ansible role https://review.opendev.org/660789 | 20:29 |
| goldyfruit | Qinling merged \o/ | 20:30 |
| *** schwicht has joined #openstack-kolla | 20:37 | |
| *** altlogbot_1 has quit IRC | 20:45 | |
| *** jbadiapa has quit IRC | 20:52 | |
| *** radeks has quit IRC | 21:03 | |
| *** pcaruana has quit IRC | 21:07 | |
| *** whoami-rajat has quit IRC | 21:22 | |
| *** sshnaidm|off has quit IRC | 21:34 | |
| *** sshnaidm has joined #openstack-kolla | 21:40 | |
| *** sshnaidm is now known as sshnaidm|off | 21:41 | |
| *** kplant has joined #openstack-kolla | 21:50 | |
| *** schwicht has quit IRC | 21:52 | |
| *** dcapone2004 has joined #openstack-kolla | 22:02 | |
| *** dcapone2004 has left #openstack-kolla | 22:02 | |
| *** dcapone2004 has joined #openstack-kolla | 22:04 | |
| *** dcapone2004_ has joined #openstack-kolla | 22:05 | |
| *** dcapone2004_ has quit IRC | 22:05 | |
| *** schwicht has joined #openstack-kolla | 22:24 | |
| *** iclon__ has joined #openstack-kolla | 22:25 | |
| *** iclon_ has quit IRC | 22:28 | |
| *** my_nd has quit IRC | 22:28 | |
| *** luksky has quit IRC | 22:28 | |
| *** schwicht has quit IRC | 22:41 | |
| *** zbr has quit IRC | 23:00 | |
| *** Sravan has joined #openstack-kolla | 23:30 | |
| *** Sravan has quit IRC | 23:37 | |
| *** Sravan has joined #openstack-kolla | 23:55 | |
| *** Sravan has quit IRC | 23:56 | |
| *** Sravan has joined #openstack-kolla | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!