| *** rcernin has joined #tripleo | 00:02 | |
| *** d0ugal has joined #tripleo | 00:06 | |
| *** dsneddon has joined #tripleo | 00:34 | |
| *** dsneddon has quit IRC | 00:39 | |
| *** dsneddon has joined #tripleo | 01:07 | |
| *** spsurya has joined #tripleo | 01:09 | |
| *** dsneddon has quit IRC | 01:12 | |
| *** pkopec has quit IRC | 01:25 | |
| openstackgerrit | pengyuesheng proposed openstack/paunch master: Blacklist sphinx 2.1.0 (autodoc bug) https://review.opendev.org/673762 | 01:29 |
|---|---|---|
| openstackgerrit | Steve Baker proposed openstack/python-tripleoclient master: Move enable_nova after enable_ironic https://review.opendev.org/677099 | 01:31 |
| *** dsneddon has joined #tripleo | 01:40 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Also disable undercloud glance with nova https://review.opendev.org/677100 | 01:41 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-quickstart-extras master: DNM: undercloud_enable_nova: false by default https://review.opendev.org/664170 | 01:42 |
| *** dsneddon has quit IRC | 01:44 | |
| *** bhagyashris has joined #tripleo | 01:48 | |
| *** dsneddon has joined #tripleo | 02:13 | |
| *** dsneddon has quit IRC | 02:18 | |
| *** redrobot has quit IRC | 02:23 | |
| *** Guest90568 has joined #tripleo | 02:29 | |
| *** Guest90568 is now known as redrobot | 02:32 | |
| *** ramishra has joined #tripleo | 02:45 | |
| *** dsneddon has joined #tripleo | 02:51 | |
| *** dsneddon has quit IRC | 02:56 | |
| *** surpatil has joined #tripleo | 03:10 | |
| *** dsneddon has joined #tripleo | 03:27 | |
| *** dsneddon has quit IRC | 03:31 | |
| *** ykarel has joined #tripleo | 03:51 | |
| *** dsneddon has joined #tripleo | 04:04 | |
| *** dsneddon has quit IRC | 04:09 | |
| *** saneax has quit IRC | 04:09 | |
| *** ratailor has joined #tripleo | 04:14 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Don't mount /etc/nova into placement migrate script https://review.opendev.org/677106 | 04:16 |
| ykarel | i too faced it few days back ^^ | 04:23 |
| *** udesale has joined #tripleo | 04:30 | |
| openstackgerrit | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Use /var/tmp on host to store temporal files for image upload via Horizon https://review.opendev.org/677107 | 04:37 |
| *** dsneddon has joined #tripleo | 04:41 | |
| *** dsneddon has quit IRC | 04:46 | |
| openstackgerrit | Merged openstack/tripleo-ci master: Use proxy when retrieving delorean file from RDO https://review.opendev.org/677077 | 05:06 |
| *** ykarel has quit IRC | 05:09 | |
| *** bhagyashris has quit IRC | 05:09 | |
| *** skramaja has joined #tripleo | 05:09 | |
| *** ykarel has joined #tripleo | 05:10 | |
| *** suraj_ has joined #tripleo | 05:12 | |
| *** surpatil has quit IRC | 05:12 | |
| *** ykarel is now known as ykarel|afk | 05:14 | |
| *** suraj_ is now known as surpatil | 05:14 | |
| *** surpatil has joined #tripleo | 05:15 | |
| *** surpatil has quit IRC | 05:15 | |
| *** surpatil has joined #tripleo | 05:17 | |
| *** surpatil has quit IRC | 05:17 | |
| *** kopecmartin|off is now known as kopecmartin | 05:18 | |
| *** surpatil has joined #tripleo | 05:18 | |
| *** dsneddon has joined #tripleo | 05:20 | |
| *** raukadah is now known as chkumar|ruck | 05:23 | |
| *** janki has joined #tripleo | 05:24 | |
| *** dsneddon has quit IRC | 05:25 | |
| *** dsneddon has joined #tripleo | 05:45 | |
| *** ykarel|afk is now known as ykarel | 05:49 | |
| *** jaosorior has joined #tripleo | 05:51 | |
| *** mrunge has quit IRC | 05:52 | |
| *** marios has joined #tripleo | 05:52 | |
| *** mrunge_ has joined #tripleo | 05:52 | |
| *** mrunge_ is now known as mrunge | 05:56 | |
| openstackgerrit | Takashi Kajinami proposed openstack/tripleo-heat-templates master: Use the special user role 'service' as service token role https://review.opendev.org/674516 | 05:56 |
| *** yprokule has joined #tripleo | 06:03 | |
| *** jbadiapa has joined #tripleo | 06:05 | |
| Tengu | hello there :) | 06:05 |
| *** numans has quit IRC | 06:06 | |
| *** skramaja_ has joined #tripleo | 06:07 | |
| *** skramaja has quit IRC | 06:07 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: Make the rabbitmq-ready exec more stringent https://review.opendev.org/676431 | 06:10 |
| ykarel | bandini, too quick :) | 06:15 |
| bandini | morning | 06:17 |
| bandini | ykarel: ;) I *think* the above should fix that race. let's cross fingers. did you happen to see some failures with the debug patch? | 06:17 |
| ykarel | bandini, i seen it two times | 06:18 |
| ykarel | after that didn't rechecked | 06:18 |
| *** marios has quit IRC | 06:18 | |
| *** matbu has joined #tripleo | 06:19 | |
| *** ykarel is now known as ykarel|afk | 06:21 | |
| *** mschuppert has joined #tripleo | 06:22 | |
| openstackgerrit | yatin proposed openstack/tripleo-heat-templates master: [DNM] Debug puppet apply rabbitmq #1835615 https://review.opendev.org/676364 | 06:24 |
| *** marios has joined #tripleo | 06:25 | |
| ykarel|afk | bandini, ack okk so it's green for 5 runs then it should be good, running once with debug too | 06:25 |
| bandini | acl | 06:26 |
| *** saneax has joined #tripleo | 06:27 | |
| *** threestrands has quit IRC | 06:34 | |
| *** threestrands has joined #tripleo | 06:35 | |
| openstackgerrit | Karthik S proposed openstack/os-net-config stable/rocky: Numvfs setting during update/upgrade https://review.opendev.org/676328 | 06:35 |
| *** ykarel|afk is now known as ykarel | 06:38 | |
| *** paramite has joined #tripleo | 06:39 | |
| ykarel | chkumar|ruck, had u filed bug for the intermittent timeout in HA jobs? | 06:39 |
| ykarel | i seen it twice in my debug patch | 06:39 |
| openstackgerrit | Karthik S proposed openstack/os-net-config stable/queens: Numvfs setting during update/upgrade https://review.opendev.org/676329 | 06:39 |
| ykarel | chkumar|ruck, i seen it in https://review.opendev.org/#/c/676364/, so it's likely a real issue | 06:42 |
| ykarel | most likely infra | 06:42 |
| *** dsneddon has quit IRC | 06:48 | |
| *** ksambor has quit IRC | 06:50 | |
| *** ksambor has joined #tripleo | 06:51 | |
| *** pkopec has joined #tripleo | 06:51 | |
| *** shyamb has joined #tripleo | 06:55 | |
| *** jtomasek has joined #tripleo | 06:58 | |
| *** shyamb has quit IRC | 07:00 | |
| *** shyam89 has joined #tripleo | 07:00 | |
| *** amoralej|off is now known as amoralej | 07:05 | |
| *** udesale has quit IRC | 07:05 | |
| *** shyam89 has quit IRC | 07:06 | |
| *** shyamb has joined #tripleo | 07:06 | |
| *** udesale has joined #tripleo | 07:06 | |
| chkumar|ruck | ykarel: https://bugs.launchpad.net/tripleo/+bug/1840616 | 07:12 |
| openstack | Launchpad bug 1840616 in tripleo "Master check and promotion jobs are giving ansible time out while overcloud deploy in fs01/02/35" [Critical,Triaged] | 07:12 |
| *** jaosorior has quit IRC | 07:14 | |
| ykarel | chkumar|ruck, Thanks, i will add what i found what i found in my patch | 07:16 |
| ykarel | in some time | 07:16 |
| *** ykarel is now known as ykarel|afk | 07:16 | |
| *** rcernin has quit IRC | 07:17 | |
| *** apetrich has joined #tripleo | 07:17 | |
| *** threestrands has quit IRC | 07:19 | |
| *** jtomasek has quit IRC | 07:21 | |
| *** dsneddon has joined #tripleo | 07:21 | |
| *** ykarel|afk has quit IRC | 07:23 | |
| *** dsneddon has quit IRC | 07:26 | |
| *** cylopez has joined #tripleo | 07:27 | |
| *** rpittau|afk is now known as rpittau | 07:35 | |
| *** yolanda has quit IRC | 07:42 | |
| *** yolanda__ has joined #tripleo | 07:43 | |
| *** udesale has quit IRC | 07:43 | |
| *** jpena|off is now known as jpena | 07:44 | |
| *** udesale has joined #tripleo | 07:44 | |
| *** ykarel|afk has joined #tripleo | 07:46 | |
| *** ykarel|afk is now known as ykarel | 07:46 | |
| *** lucasagomes has joined #tripleo | 07:48 | |
| *** waleedm has joined #tripleo | 07:51 | |
| openstackgerrit | Natal Ngétal proposed openstack/paunch master: [Configuration] Switch to stestr. https://review.opendev.org/629421 | 07:52 |
| *** dsneddon has joined #tripleo | 07:59 | |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates master: Fix NovaResumeGuestsStateOnHostBoot when using podman https://review.opendev.org/677135 | 08:01 |
| *** dsneddon has quit IRC | 08:04 | |
| *** jaosorior has joined #tripleo | 08:05 | |
| *** chem has joined #tripleo | 08:06 | |
| *** suuuper has joined #tripleo | 08:06 | |
| ykarel | ChanServ, commented https://bugs.launchpad.net/tripleo/+bug/1840616/comments/1 | 08:11 |
| openstack | Launchpad bug 1840616 in tripleo "Master check and promotion OVB jobs are randomly giving ansible time out while overcloud deploy" [Critical,Triaged] | 08:11 |
| ykarel | chkumar|ruck, ^^ | 08:11 |
| *** matbu has quit IRC | 08:12 | |
| *** xek has joined #tripleo | 08:12 | |
| *** skramaja has joined #tripleo | 08:12 | |
| *** skramaja_ has quit IRC | 08:12 | |
| *** tkajinam has quit IRC | 08:18 | |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates stable/stein: Allow combining system_upgrade_prepare and system_upgrade_run into system_upgrade https://review.opendev.org/677140 | 08:19 |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates stable/stein: Fix NovaResumeGuestsStateOnHostBoot when using podman https://review.opendev.org/677141 | 08:23 |
| *** ykarel is now known as ykarel|afk | 08:23 | |
| *** shyamb has quit IRC | 08:24 | |
| *** yolanda__ is now known as yolanda | 08:28 | |
| *** derekh has joined #tripleo | 08:34 | |
| *** dsneddon has joined #tripleo | 08:35 | |
| *** dsneddon has quit IRC | 08:40 | |
| *** mkisielewski has joined #tripleo | 08:41 | |
| *** ykarel|afk is now known as ykarel | 08:51 | |
| *** avivgta has quit IRC | 08:55 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations stable/stein: Linting hardening with pre-commit https://review.opendev.org/675616 | 08:58 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations stable/stein: Convert tox to native zuul https://review.opendev.org/677150 | 08:58 |
| *** matbu has joined #tripleo | 09:02 | |
| *** surpatil has quit IRC | 09:04 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/stein: Force re-run of pacemaker bundle init containers during upgrade-scaleup https://review.opendev.org/677057 | 09:04 |
| *** dsneddon has joined #tripleo | 09:10 | |
| *** dsneddon has quit IRC | 09:14 | |
| *** bhagyashris has joined #tripleo | 09:17 | |
| *** janki has quit IRC | 09:17 | |
| mkisielewski | Dear all, has anyone experienced problems with Manila deployment + Ceph NFS Ganesha on Stein (current-tripleo-rdo)? It looks like during ceph-ansible execution `cephfs_data_pool.name` is set correctly on ceph-mds tasks, which creates `manila_data` and `manila_metadata`, but ceph-nfs tasks fails as they use default value `cephfs_data`. | 09:23 |
| *** surpatil has joined #tripleo | 09:30 | |
| *** rascasoft has quit IRC | 09:31 | |
| *** rascasoft has joined #tripleo | 09:34 | |
| *** surpatil has quit IRC | 09:39 | |
| *** surpatil has joined #tripleo | 09:40 | |
| *** pierreprinetti has joined #tripleo | 09:43 | |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-docs master: Enhancement to cell v2 doc with split for stein/train https://review.opendev.org/672744 | 09:43 |
| openstackgerrit | Chandan Kumar (raukadah) proposed openstack/tripleo-quickstart master: [DNM]Use config_drive for network basicops tests https://review.opendev.org/676439 | 09:45 |
| *** dsneddon has joined #tripleo | 09:46 | |
| *** pierreprinetti has quit IRC | 09:48 | |
| *** pierreprinetti has joined #tripleo | 09:48 | |
| *** dsneddon has quit IRC | 09:50 | |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-ansible master: Remove temporary workaround for hieradata_files https://review.opendev.org/677164 | 09:51 |
| *** apetrich has quit IRC | 09:56 | |
| tbarron | mkisielewski: what version of ceph-ansible do you have? looking at the ceph-ansible code | 10:06 |
| tbarron | both ceph-mds tasks and | 10:06 |
| tbarron | ceph-nfs tasks use the *value* of the variable cephfs_data, which should be 'manila_data' | 10:07 |
| tbarron | mkisielewski: but i'm looking at a downstream osp13 deployment atm | 10:08 |
| tbarron | mkisielewski: at at a git repo | 10:08 |
| bandini | ykarel: seen this one Evaluation Error: Error while evaluating a Function Call, Could not find class ::rabbitmq::params already? (spotted it on https://zuul.opendev.org/t/openstack/build/e767db93fe214514bebc065a64288548/log/job-output.txt) | 10:10 |
| tbarron | mkisielewski: so my overcloud deploy used ceph-ansible 3.2.15-1.el7cp which doesn't have | 10:10 |
| mkisielewski | tbarron: 4.0.0.0.rc13.1.el7 I've looked on code history in git repo and also tasks in installed role and it uses `cephfs_data_pool.name` variable. On ceph-mds it also uses those variables, but they're defined in playbook group vars - mdss, but not for nfss. | 10:10 |
| *** shyamb has joined #tripleo | 10:11 | |
| tbarron | mkisielewski: is the end result that you have the wrong pool in /etc/ganesha/ganesha.conf RADOS_KV section? | 10:13 |
| tbarron | mkisielewski: or that it breaks building that config file? | 10:13 |
| *** dsneddon has joined #tripleo | 10:17 | |
| mkisielewski | tbarron: It breaks during overcloud deployment - [ceph-nfs : create an empty rados index object] - 'error opening pool cephfs_data: (2) No such file or directory' Now I'm trying to redeploy it with defined variables: `CephAnsibleExtraConfig: cephfs_data_pool: ...` | 10:17 |
| bandini | ykarel: I filed https://bugs.launchpad.net/tripleo/+bug/1840641 | 10:17 |
| openstack | Launchpad bug 1840641 in tripleo "puppet-tripleo unit tests are failing due to too new puppet-rabbitmq" [High,New] | 10:17 |
| tbarron | mkisielewski: oh i need another cup of coffee, i'm using queens, sorry bout that | 10:18 |
| *** morazi has quit IRC | 10:19 | |
| tbarron | mkisielewski: explains the older ceph-ansible that I have | 10:19 |
| ykarel | bandini, /me looks | 10:19 |
| mkisielewski | tbarron: After inspecting logs and group_vars in /var/lib/mistral/ it looks like those variables are set only for mdss and not for nfss. And I figured it now, that my config is using seperate machines for mon and mds, so it might not fail on standard all-in-controller deployment because mds is the same machine as nfs. | 10:20 |
| openstackgerrit | Gael Chamoulaud proposed openstack/python-tripleoclient stable/stein: Run Validations with ThreadPoolExecutor https://review.opendev.org/677170 | 10:21 |
| *** dsneddon has quit IRC | 10:22 | |
| *** hjensas has joined #tripleo | 10:24 | |
| ykarel | bandini, ack then it seems u need to pin in Puppetfile too | 10:24 |
| bandini | ykarel: ack, why only now though? | 10:24 |
| ykarel | bandini, https://review.opendev.org/#/c/677082/1/Puppetfile | 10:25 |
| ykarel | merged today | 10:25 |
| bandini | ah boom | 10:25 |
| *** rascasoft has quit IRC | 10:27 | |
| *** rascasoft has joined #tripleo | 10:27 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations stable/stein: Adding molecule tests for no-op-firewall-nova-driver validation https://review.opendev.org/677171 | 10:31 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations stable/stein: Adds molecule tests for the nova-status validation https://review.opendev.org/677172 | 10:31 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: Pin puppet-rabbitmq https://review.opendev.org/677173 | 10:31 |
| *** holser has joined #tripleo | 10:35 | |
| *** shyamb has quit IRC | 10:36 | |
| *** skramaja has quit IRC | 10:39 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-common master: Add "rhel_containers" variable to skip containers for RHEL https://review.opendev.org/676497 | 10:43 |
| *** bhagyashris has quit IRC | 10:50 | |
| *** pierreprinetti has quit IRC | 10:53 | |
| *** dsneddon has joined #tripleo | 10:55 | |
| *** florianf has joined #tripleo | 10:58 | |
| *** shyamb has joined #tripleo | 10:58 | |
| *** dsneddon has quit IRC | 11:00 | |
| *** dprince has joined #tripleo | 11:00 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart-extras master: Add rhel_containers variable https://review.opendev.org/676474 | 11:01 |
| *** waleedm has quit IRC | 11:01 | |
| *** hjensas has quit IRC | 11:08 | |
| *** gfidente has joined #tripleo | 11:10 | |
| *** marios has quit IRC | 11:12 | |
| *** udesale has quit IRC | 11:15 | |
| *** tesseract has joined #tripleo | 11:15 | |
| *** hjensas has joined #tripleo | 11:16 | |
| bandini | ykarel: ideas as to why https://review.opendev.org/#/c/677173/ breaks only on unit-6.0 ? it's not entirely obvious to me atm | 11:19 |
| ykarel | bandini, checking | 11:19 |
| bandini | thanks! | 11:19 |
| *** pierreprinetti has joined #tripleo | 11:20 | |
| ykarel | bandini, WARN -> Remove the duplicates of the following modules: rabbitmq | 11:20 |
| ykarel | so possibly the check is added with puppet6 only | 11:20 |
| ykarel | i remember some warning to be treated as error were done recently | 11:21 |
| ykarel | bandini, so i think you can revert the orignal patch in puppet-openstack-integration, also pin it so it's not updated again with bot | 11:22 |
| openstackgerrit | Francesco Pantano proposed openstack/tripleo-heat-templates master: Add the certificate specs in ceph_grafana composable service https://review.opendev.org/674556 | 11:22 |
| ykarel | bandini, to pin need change https://github.com/openstack/puppet-openstack-integration/blob/master/external_modules.txt#L13 just in case u not aware | 11:24 |
| *** ansmith has quit IRC | 11:27 | |
| bandini | ykarel: but I assumed that reverting it in the common openstack-integration is a no no? | 11:29 |
| bandini | i.e. we (tripleo) need to fix our stuff | 11:29 |
| ykarel | bandini, yes better to fix it in TripleO side, as per https://review.rdoproject.org/r/#/c/20654/ looks like already we had a tech debt | 11:32 |
| ykarel | as per commit message | 11:33 |
| *** dsneddon has joined #tripleo | 11:33 | |
| openstackgerrit | Chandan Kumar (raukadah) proposed openstack/tripleo-quickstart-extras master: [DNM]gate testing https://review.opendev.org/677182 | 11:34 |
| bandini | ykarel: yeah my idea was it to pin it first and then spend some time looking into it | 11:34 |
| bandini | but if pinning it won't work then I guess there are not that many options.. | 11:34 |
| *** apetrich has joined #tripleo | 11:36 | |
| ykarel | bandini, ack, /me cant' think of an option other than pinning it in poi or solving it in Tripleo | 11:36 |
| ykarel | mwhahaha, may have some idea around it ^^ | 11:36 |
| bandini | got you. let me spend some time after lunch looking at it | 11:36 |
| *** dsneddon has quit IRC | 11:38 | |
| *** jpena is now known as jpena|lunch | 11:39 | |
| tbarron | gfidente: note mkisielewski remark above that in stein ceph-ansible | 11:40 |
| tbarron | uses `cephfs_data_pool.name` variable. On ceph-mds it also uses those variables, but they're defined in playbook group vars - mdss, but not for nfss. | 11:40 |
| tbarron | gfidente: maybe we need a change like | 11:41 |
| tbarron | https://review.opendev.org/#/c/673569/ | 11:42 |
| tbarron | for ceph_nfs | 11:42 |
| mkisielewski | tbarron: Current deployment is in progress, but I can that after defining `CephAnsibleExtraConfig: cephfs_data_pool: ...` it passed ceph-nfs tasks. Please note that I'm using separate machine for mon, mds, and nfs is located on controller. | 11:45 |
| gfidente | mkisielewski tbarron hi | 11:46 |
| tbarron | mkisielewski: my theory is that this is because of the change in the way variables are set for ceph-ansible in stein | 11:46 |
| tbarron | mkisielewski: but gfidente knows this stuff better than me, i'm just a manila guy | 11:47 |
| gfidente | mkisielewski can you dump somewhere the group vars from the failing attempt? | 11:47 |
| *** rlandy has joined #tripleo | 11:47 | |
| gfidente | mkisielewski or if you could open bug in LP that would help | 11:47 |
| openstackgerrit | Gauvain Pocentek proposed openstack/tripleo-heat-templates master: Add multi region support in nova_wait_for_compute_service.py https://review.opendev.org/677184 | 11:48 |
| gfidente | mkisielewski I mean describing in there what is failing and what is the workaround you're using, also what version of ceph-ansible | 11:48 |
| *** rlandy is now known as rlandy|rover | 11:48 | |
| *** chkumar|ruck is now known as chkumar|rover | 11:49 | |
| *** rlandy|rover is now known as rlandy|ruck | 11:49 | |
| mkisielewski | gfidente: I will let you know when the deployment ends and will to try describe it. | 11:49 |
| gfidente | mkisielewski ok thanks for pinging | 11:49 |
| gfidente | are you using some ceph-ansible build from centos/cbs nautilus? | 11:50 |
| *** janki has joined #tripleo | 11:50 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: zypper-minimal: install without recommends https://review.opendev.org/677188 | 11:56 |
| gfidente | mkisielewski or if you have the logs from the ceph-ansible run | 11:57 |
| gfidente | that would help too | 11:58 |
| *** weshay_pto is now known as weshay | 12:00 | |
| *** rh-jelabarre has joined #tripleo | 12:04 | |
| *** raildo has joined #tripleo | 12:05 | |
| *** marios has joined #tripleo | 12:05 | |
| *** ykarel is now known as ykarel|afk | 12:05 | |
| *** bfournie has quit IRC | 12:06 | |
| *** dprince has quit IRC | 12:06 | |
| *** dsneddon has joined #tripleo | 12:06 | |
| mwhahaha | bandini: propose a revert of the poi version bumo | 12:08 |
| bandini | are you calling me a bumo? | 12:09 |
| bandini | none calls me a bumo | 12:09 |
| bandini | mwhahaha: will do | 12:10 |
| *** avivgta has joined #tripleo | 12:10 | |
| *** dsneddon has quit IRC | 12:11 | |
| bandini | mwhahaha, ykarel|afk: https://review.opendev.org/#/c/677192/ | 12:13 |
| *** jaosorior has quit IRC | 12:14 | |
| ykarel|afk | bandini, you also need to pin it to external_modules.txt else it will be reprosed with bot | 12:14 |
| ykarel|afk | reproposed | 12:14 |
| *** amoralej is now known as amoralej|lunch | 12:16 | |
| *** ansmith has joined #tripleo | 12:16 | |
| *** lucasagomes has quit IRC | 12:17 | |
| bandini | ykarel|afk: should I do that in the same change or in a separate one? | 12:17 |
| *** lucasagomes has joined #tripleo | 12:18 | |
| ykarel|afk | bandini, single patch is better i think, as both are related | 12:19 |
| bandini | ykarel|afk: how often does the bot repropose this (i.e. with what frequency?) | 12:19 |
| ykarel|afk | bandini, iirc daily | 12:19 |
| bandini | ah ok then no we need to pin that | 12:19 |
| bandini | let me do that | 12:19 |
| *** apetrich has quit IRC | 12:20 | |
| bandini | ykarel|afk: ok updated | 12:20 |
| *** florianf has quit IRC | 12:21 | |
| *** ansmith_ has joined #tripleo | 12:22 | |
| ykarel|afk | bandini, ack, yes it's daily as that job runs in periodic pipeline, https://opendev.org/openstack/project-config/src/branch/master/zuul.d/pipelines.yaml#L191 | 12:22 |
| bandini | got you | 12:22 |
| bandini | thanks for the info | 12:22 |
| ykarel|afk | https://opendev.org/openstack/project-config/src/branch/master/zuul.d/jobs.yaml#L967 | 12:22 |
| *** fultonj has joined #tripleo | 12:23 | |
| bandini | interesting | 12:23 |
| *** morazi has joined #tripleo | 12:24 | |
| *** ansmith has quit IRC | 12:24 | |
| *** florianf has joined #tripleo | 12:26 | |
| *** apetrich has joined #tripleo | 12:28 | |
| *** ykarel|afk is now known as ykarel | 12:28 | |
| *** rfolco has joined #tripleo | 12:28 | |
| *** jpena|lunch is now known as jpena | 12:31 | |
| *** paramite has quit IRC | 12:32 | |
| *** numans has joined #tripleo | 12:34 | |
| *** jaosorior has joined #tripleo | 12:38 | |
| *** flaper87 has left #tripleo | 12:40 | |
| *** dsneddon has joined #tripleo | 12:42 | |
| *** spsurya has quit IRC | 12:43 | |
| chem | mwhahaha: hey, do you still encounter podman failing to start a puppet container because it believe it still exists somehow ( https://github.com/containers/libpod/issues/2240#issuecomment-484134257 ) ? | 12:44 |
| *** ykarel is now known as ykarel|away | 12:47 | |
| *** dsneddon has quit IRC | 12:47 | |
| *** shyamb has quit IRC | 12:48 | |
| openstackgerrit | Merged openstack/tripleo-validations master: Adds molecule tests to image-serve and correct validation https://review.opendev.org/672494 | 12:49 |
| cloudnull | mornings | 12:50 |
| *** ykarel|away has quit IRC | 12:52 | |
| *** mcornea has joined #tripleo | 12:55 | |
| *** bfournie has joined #tripleo | 12:56 | |
| *** pierreprinetti has quit IRC | 12:59 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations stable/stein: Convert tox to native zuul https://review.opendev.org/677150 | 12:59 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations stable/stein: Adding molecule tests for no-op-firewall-nova-driver validation https://review.opendev.org/677171 | 12:59 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations stable/stein: Adds molecule tests for the nova-status validation https://review.opendev.org/677172 | 12:59 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations stable/stein: Adds molecule tests to image-serve and correct validation https://review.opendev.org/677197 | 12:59 |
| *** pierreprinetti has joined #tripleo | 12:59 | |
| *** bfournie has quit IRC | 13:00 | |
| *** dsneddon has joined #tripleo | 13:00 | |
| *** pierreprinetti has quit IRC | 13:00 | |
| *** ratailor has quit IRC | 13:00 | |
| openstackgerrit | Rajesh Tailor proposed openstack/tripleo-heat-templates master: Add new role parameters for cpu/ram/disk allocation ratio https://review.opendev.org/675854 | 13:01 |
| openstackgerrit | Luca Miccini proposed openstack/tripleo-common stable/rocky: Adds redfish support to 'overcloud generate fencing'. https://review.opendev.org/677199 | 13:03 |
| *** pierreprinetti has joined #tripleo | 13:06 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations stable/stein: Convert tox to native zuul https://review.opendev.org/677150 | 13:08 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations stable/stein: Adding molecule tests for no-op-firewall-nova-driver validation https://review.opendev.org/677171 | 13:08 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations stable/stein: Adds molecule tests for the nova-status validation https://review.opendev.org/677172 | 13:08 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations stable/stein: Adds molecule tests to image-serve and correct validation https://review.opendev.org/677197 | 13:08 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations stable/stein: Add molecule tests for the undercloud-cpu role https://review.opendev.org/677200 | 13:08 |
| *** bfournie has joined #tripleo | 13:14 | |
| mkisielewski | gfidente: group_vars and extra_vars (as a workaround) looks like this: https://pastebin.com/rJe2LQqN ceph-ansible version 4.0.0.0.rc13.1.el7 | 13:16 |
| gfidente | mkisielewski awesome | 13:17 |
| gfidente | thanks | 13:17 |
| gfidente | mkisielewski I think tbarron nailed it down though https://bugs.launchpad.net/tripleo/+bug/1840651 | 13:17 |
| openstack | Launchpad bug 1840651 in tripleo "cephfs_data_pool should be into ceph-ansible all vars not mds only" [Medium,Triaged] | 13:17 |
| gfidente | I'll try to post a patch asap | 13:17 |
| *** aakarsh has quit IRC | 13:18 | |
| mkisielewski | Default setup will work in scenario with collocated mds and nfs on controller, but in my case with seperate mds and nfs it fails | 13:18 |
| gfidente | mkisielewski yeah makes sense | 13:18 |
| mkisielewski | gfidente exactly putting it in all vars will fix it | 13:18 |
| *** janki has quit IRC | 13:20 | |
| *** amoralej|lunch is now known as amoralej | 13:25 | |
| *** beekneemech is now known as bnemec | 13:26 | |
| *** surpatil has quit IRC | 13:36 | |
| mwhahaha | chem: i haven't but i think we added some randomization to work around that | 13:39 |
| *** ykarel|away has joined #tripleo | 13:43 | |
| xarlos | When tripleo is deploying a stack, does it need access to all the networks? Or just the provisioning network? | 13:43 |
| mwhahaha | chem: https://review.opendev.org/#/c/676984/ | 13:44 |
| mwhahaha | xarlos: depends on what you mean. the undercloud only has access to the provisioning network so I believe all actions occur over that. the overcloud nodes need to have access to their networs | 13:45 |
| *** ykarel|away is now known as ykarel | 13:45 | |
| openstackgerrit | Francesco Pantano proposed openstack/tripleo-heat-templates master: [WIP] Add a StorageDashboard network used by CephGrafana service https://review.opendev.org/674318 | 13:46 |
| xarlos | mwhahaha: That was my assumption. but based on a response someone gave to my pastebin, it was suggested that it may have been that tripleo was not able to get a response from an api endpoint, and therefor geerating my error. | 13:48 |
| mwhahaha | that's also possible | 13:48 |
| mwhahaha | it' | 13:48 |
| mwhahaha | it really depends on your configuration | 13:48 |
| mwhahaha | you can make it do silly things :D | 13:49 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: Move cephfs and cephfs_*_pool ceph-ansible parameters in -base https://review.opendev.org/677207 | 13:50 |
| xarlos | I don't want silly things. I have network isolation configured (I believe) to be in line with he default networks. | 13:50 |
| xarlos | https://pastebin.com/edJhFNWW | 13:50 |
| xarlos | This however is given, and I cannot figure out why. | 13:51 |
| mwhahaha | xarlos: what version is this? | 13:51 |
| xarlos | It does appear that the deployment is looking to check on a completion. Perhaps it's timing out ona deployment? Or perhaps it indeed iis not able to check an endpoint. | 13:52 |
| mwhahaha | it's timing out | 13:52 |
| xarlos | This is centos/stein latest. | 13:52 |
| mwhahaha | so it's stuck on step1 which shouldn't be doing anythign too crazy | 13:52 |
| mwhahaha | the no json object is from the client waiting on messages on the undercloud. now if you configure network isolation in a way that nukes the provisioning network, that'd be a problem. but I think that runs prior to step1 | 13:53 |
| mwhahaha | so it seems like it's stuck on step one, would need to look at the logs on the nodes to see what it's doing | 13:53 |
| xarlos | provisioning network seems to be intact. | 13:54 |
| *** aakarsh has joined #tripleo | 13:54 | |
| xarlos | Could this be an "unknown host" on the ssh keys from previous deployments? | 13:56 |
| xarlos | I assume deployments ignore these keys... hmm. | 13:56 |
| mwhahaha | i don't think that's related | 13:56 |
| mwhahaha | i think we ignore the host keys | 13:56 |
| *** dsneddon has quit IRC | 14:03 | |
| *** avivgta has quit IRC | 14:05 | |
| *** pkopec has quit IRC | 14:08 | |
| *** pkopec has joined #tripleo | 14:09 | |
| *** pierreprinetti has quit IRC | 14:13 | |
| *** pierreprinetti has joined #tripleo | 14:14 | |
| *** pkopec has quit IRC | 14:15 | |
| *** sshnaidm is now known as sshnaidm|bbl | 14:16 | |
| *** pkopec has joined #tripleo | 14:20 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: Pin puppet-rabbitmq https://review.opendev.org/677173 | 14:22 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: Fix spec tests to be compatible with latest puppet-rabbitmq https://review.opendev.org/677173 | 14:23 |
| *** artom has quit IRC | 14:29 | |
| *** yolanda has quit IRC | 14:31 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fix NovaResumeGuestsStateOnHostBoot when using podman https://review.opendev.org/677135 | 14:31 |
| *** dtrainor has joined #tripleo | 14:31 | |
| zbr | it seems that we still have a serious number of authetnication failures with dockerhub: http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Action%5C%5C%5C%22:%5C%5C%5C%22pull%5C%22%20AND%20%20%20message:%5C%22ERROR%20%2Fvar%2Flog%2Ftripleo-container-image-prepare.log%5C%22%20AND%20%20%20tags:%5C%22console%5C%22&from=7d | 14:35 |
| mwhahaha | yea we're trying to figure out what's happening via https://review.opendev.org/#/c/674919/ | 14:36 |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-heat-templates master: WIP Remove GroupVars from nested stacks https://review.opendev.org/677218 | 14:36 |
| *** yolanda has joined #tripleo | 14:36 | |
| *** dsneddon has joined #tripleo | 14:39 | |
| *** rascasoft has quit IRC | 14:40 | |
| *** rascasoft has joined #tripleo | 14:40 | |
| *** dsneddon has quit IRC | 14:43 | |
| *** saneax has quit IRC | 14:49 | |
| *** cfontain_ has quit IRC | 14:50 | |
| *** surpatil has joined #tripleo | 14:52 | |
| *** ykarel has quit IRC | 14:57 | |
| *** cdearborn has joined #tripleo | 14:57 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Add overcloud deploy parameter to set CinderLVMLoopDeviceSize https://review.opendev.org/677227 | 14:58 |
| openstackgerrit | Cédric Jeanneret (Tengu) proposed openstack/tripleo-ansible master: Creates tripleo-validations-package role https://review.opendev.org/677229 | 14:58 |
| *** pkopec has quit IRC | 15:02 | |
| *** jaosorior has quit IRC | 15:04 | |
| *** pkopec has joined #tripleo | 15:12 | |
| *** cylopez has quit IRC | 15:12 | |
| *** artom has joined #tripleo | 15:14 | |
| *** dsneddon has joined #tripleo | 15:14 | |
| openstackgerrit | Cédric Jeanneret (Tengu) proposed openstack/tripleo-ansible master: Creates tripleo-validations-package role https://review.opendev.org/677229 | 15:15 |
| *** saneax has joined #tripleo | 15:17 | |
| *** dsneddon has quit IRC | 15:19 | |
| *** chkumar|rover is now known as raukadah | 15:24 | |
| cloudnull | zbr mwhahaha if you either of you have an idea of why we're seeing the mismatched digests and want to push over that review, please feel free. | 15:33 |
| *** cfontain_ has joined #tripleo | 15:34 | |
| mwhahaha | might want to poke stevebaker this afternoon to see if he has any idea | 15:34 |
| openstackgerrit | Merged openstack/tripleo-validations master: Removed older version of python https://review.opendev.org/676112 | 15:35 |
| *** suuuper has quit IRC | 15:38 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: [WIP] Convert firewall rules to use TripleO-Ansible https://review.opendev.org/677237 | 15:40 |
| cloudnull | mwhahaha mind Tengu mind giving that a quick look to see if i'm on the right track ? | 15:40 |
| cloudnull | my yaql bits are here - https://review.opendev.org/#/c/677237/1/deployment/tripleo-firewall/tripleo-firewall-baremetal-ansible.yaml@55 | 15:41 |
| mwhahaha | yea i think that's about right | 15:42 |
| mwhahaha | we'll need to document the fact that the extraconfig no longer works | 15:42 |
| cloudnull | which "should" query the firewall_rules object as defined in the various templates, like so - https://review.opendev.org/#/c/677237/1/deployment/swift/swift-storage-container-puppet.yaml ? | 15:42 |
| mwhahaha | maybe still expose a way to inject random rules | 15:43 |
| cloudnull | +1 | 15:43 |
| cloudnull | will do | 15:43 |
| mwhahaha | in a specific param | 15:43 |
| mwhahaha | (role specific too) | 15:43 |
| mwhahaha | ExtraFirewallRules or something | 15:43 |
| cloudnull | ++ | 15:43 |
| cloudnull | would it be best to just add that option to the tripleo-firewall-baremetal-ansible template? or should I add ExtraFirewallRules to every template and map_merge with the template specific firewall_rules object? | 15:45 |
| *** shyamb has joined #tripleo | 15:47 | |
| mwhahaha | yes | 15:48 |
| mwhahaha | just add it to the firewall template | 15:48 |
| mwhahaha | and then you would jsut merge it with FirewallRules | 15:49 |
| *** dsneddon has joined #tripleo | 15:50 | |
| cloudnull | ++ | 15:50 |
| openstackgerrit | Francesco Pantano proposed openstack/puppet-tripleo master: Add certmonger-grafana-refresh script https://review.opendev.org/676395 | 15:52 |
| *** dsneddon has quit IRC | 15:55 | |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Remove get_overcloud_passwords, a unused function https://review.opendev.org/674011 | 15:56 |
| openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient master: Remove converge_nodes functions, unused package_update code https://review.opendev.org/674015 | 15:56 |
| *** yprokule has quit IRC | 15:57 | |
| xarlos | mwhahaha: Anythig obvious in here? https://pastebin.com/v5bCx486 | 16:02 |
| *** marios is now known as marios|out | 16:03 | |
| *** gfidente has quit IRC | 16:03 | |
| xarlos | Seems the container has been downloaded successfully. | 16:04 |
| *** rpittau is now known as rpittau|afk | 16:07 | |
| mwhahaha | looks like it's runnign teh ceph ansible bits | 16:07 |
| mwhahaha | i don't know much about that, i'd have to defer to fultonj. | 16:08 |
| *** Vorrtex has joined #tripleo | 16:08 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: Fix spec tests to be compatible with latest puppet-rabbitmq https://review.opendev.org/677173 | 16:10 |
| xarlos | I shall wait patiently and | 16:10 |
| xarlos | appreciatly :-) | 16:10 |
| *** jpena is now known as jpena|off | 16:11 | |
| fultonj | hi mwhahaha and xarlos | 16:11 |
| fultonj | xarlos: so your deployment which includes ceph is failing? | 16:12 |
| fultonj | xarlos: what version? | 16:13 |
| openstackgerrit | Rabi Mishra proposed openstack/tripleo-heat-templates master: Move GroupVars to overcloud.j2.yaml https://review.opendev.org/677218 | 16:16 |
| mwhahaha | stein | 16:17 |
| *** lucasagomes has quit IRC | 16:18 | |
| fultonj | xarlos: there should be a ceph-ansible log in undercloud:/var/lib/mistral/$stackname/ceph-ansible/ | 16:20 |
| fultonj | stackname defaults to overcloud | 16:21 |
| openstackgerrit | Francesco Pantano proposed openstack/puppet-tripleo master: Add certmonger-grafana-refresh script https://review.opendev.org/676395 | 16:22 |
| *** dsneddon has joined #tripleo | 16:23 | |
| *** mkisielewski has quit IRC | 16:23 | |
| *** surpatil has quit IRC | 16:25 | |
| xarlos | fultonj: New deployment, should be as up to date as a month or to ago. | 16:25 |
| xarlos | So, seems the last step listed in that log is waiting on the ceph monitors to form the quorum | 16:26 |
| *** dsneddon has quit IRC | 16:27 | |
| fultonj | https://docs.ceph.com/docs/nautilus/rados/troubleshooting/troubleshooting-mon/ | 16:29 |
| xarlos | fultonj: here's my ceph config that I assume you will call "stupid" :-) https://pastebin.com/hAGZxF2H | 16:30 |
| fultonj | xarlos: ^ use the ceph binary command in the container | 16:30 |
| xarlos | Oooh, okay, i'll read that - thank you! | 16:30 |
| fultonj | e.g. podman exec $mon_name "ceph -s" | 16:30 |
| fultonj | xarlos: in that file lines 1-11 look good. lines 13-26 should't be there | 16:31 |
| fultonj | xarlos: but none of those params have been used yet if the mons aren't in qurom | 16:32 |
| fultonj | lines 13-26 won't work for stein so you should get rid of them | 16:33 |
| *** marios|out has quit IRC | 16:33 | |
| *** mburned has quit IRC | 16:35 | |
| *** tesseract has quit IRC | 16:35 | |
| xarlos | I will, thanks. | 16:36 |
| xarlos | fultonj: Also, Hi :) | 16:36 |
| * fultonj waves at xarlos | 16:37 | |
| *** dsneddon has joined #tripleo | 16:39 | |
| xarlos | hmm. I do not seem to ave the ceph binary on these ceph servers. | 16:41 |
| *** ramishra has quit IRC | 16:42 | |
| *** shyamb has quit IRC | 16:43 | |
| xarlos | Oh, I see it. | 16:46 |
| xarlos | Please give me a few minutes to stop being stupid. | 16:46 |
| xarlos | health detail is just hanging. I'll let it time out or whatever. | 16:56 |
| xarlos | authentication timed out after 300 is all it says. | 16:57 |
| xarlos | (on the controller nodes) | 16:57 |
| *** cfontain_ has quit IRC | 16:58 | |
| *** cfontain has joined #tripleo | 16:59 | |
| openstackgerrit | Luca Miccini proposed openstack/tripleo-common stable/rocky: Adds redfish support to 'overcloud generate fencing'. https://review.opendev.org/677199 | 16:59 |
| *** mburned has joined #tripleo | 17:00 | |
| *** derekh has quit IRC | 17:03 | |
| *** cfontain has quit IRC | 17:03 | |
| fultonj | 12:57 xarlos: authentication timed out after 300 is all it says. | 17:05 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-ci master: Configure ovb jobs to use ansible 2.8 https://review.opendev.org/677256 | 17:05 |
| fultonj | https://docs.ceph.com/docs/nautilus/rados/troubleshooting/troubleshooting-mon/#initial-troubleshooting | 17:05 |
| fultonj | see What if ceph -s doesn’t finish? | 17:05 |
| fultonj | try using the monitors admin socket | 17:05 |
| fultonj | as per the next section | 17:05 |
| fultonj | Monitor nodes must be able to reach the others with IP, short hostname - hostname -s and via telnet on port 6789 | 17:06 |
| *** pkopec has quit IRC | 17:06 | |
| fultonj | verify that to be the case | 17:06 |
| *** amoralej is now known as amoralej|off | 17:08 | |
| xarlos | ASsuming they are using the %hostname%.storage. nomenclature, they are present in the hosts file, and manual checks with telnet seem to work. | 17:24 |
| xarlos | They will not exist in dns, but.. | 17:25 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: [WIP] Convert firewall rules to use TripleO-Ansible https://review.opendev.org/677237 | 17:29 |
| xarlos | quorum status yeilds some values. (continues reading) | 17:31 |
| *** kopecmartin is now known as kopecmartin|off | 17:33 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/stein: Support TLS priorities for pacemaker https://review.opendev.org/676586 | 17:39 |
| openstackgerrit | Merged openstack/os-net-config stable/rocky: Numvfs setting during update/upgrade https://review.opendev.org/676328 | 17:39 |
| openstackgerrit | Merged openstack/os-net-config stable/queens: Numvfs setting during update/upgrade https://review.opendev.org/676329 | 17:39 |
| openstackgerrit | Merged openstack/tripleo-validations stable/stein: Linting hardening with pre-commit https://review.opendev.org/675616 | 17:39 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/stein: Fix NovaResumeGuestsStateOnHostBoot when using podman https://review.opendev.org/677141 | 17:39 |
| xarlos | State is v2 and v2 endpoints seem to be accessible, but it looks like it's not getting pas the probe state. | 17:44 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: [WIP] Convert firewall rules to use TripleO-Ansible https://review.opendev.org/677237 | 17:49 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Configure time using tripleo-ansible https://review.opendev.org/671166 | 17:49 |
| *** florianf has quit IRC | 17:52 | |
| *** ekultails has joined #tripleo | 18:01 | |
| *** pierreprinetti has quit IRC | 18:15 | |
| *** Vorrtex has quit IRC | 18:15 | |
| xarlos | Not quite sure what it's problem is. Might attempt a stack delete and then redeploy, but not sure if a) i'll learn anything b) it'll help. | 18:21 |
| *** morazi has quit IRC | 18:23 | |
| xarlos | fultonj: If I plan on setting me journal on /dev/sdh - should I list it within my disk devices list? | 18:34 |
| *** aakarsh has quit IRC | 18:35 | |
| xarlos | I have a feeling that some of my repeated deploys may have messed someting up, so I am attempting a delete of the stack and redeploying. | 18:35 |
| xarlos | Connections appear fine, but nothing appears to be connecting to oneanother - even though via IP/names seems to be correct. | 18:36 |
| fultonj | xarlos: you'll want to clean your disks between deployments | 18:36 |
| xarlos | one of my controllers doesn't seem to be reading the container's files correctly, like there's a disk issue of some sort though there shouldn't be. | 18:37 |
| xarlos | fultonj: Yes, I have this set. | 18:37 |
| xarlos | I think in tinkering i've broken something. Will find out in circa 3 hours. | 18:37 |
| fultonj | stein uses nautilus which defaults to bluestore by default | 18:38 |
| fultonj | bluestore doesn't exactly use a journal | 18:38 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-ci master: New vexxhost CI settings https://review.opendev.org/677270 | 18:38 |
| sshnaidm|bbl | dmsimard, tristanC ^^ | 18:38 |
| fultonj | it has a bluestore DB device | 18:38 |
| fultonj | https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ceph_config.html#configure-osd-settings-with-ceph-ansible | 18:38 |
| xarlos | Oh really? Every time I blink, openstack has changed :-D | 18:38 |
| xarlos | oooh, I see. | 18:39 |
| xarlos | I have a lot to learn. | 18:39 |
| *** aakarsh has joined #tripleo | 18:39 | |
| xarlos | That's perfect | 18:39 |
| fultonj | xarlos: you have N HDDs and K SSDs per ceph storage server | 18:39 |
| fultonj | what are N and K? | 18:39 |
| xarlos | N=6K=1 | 18:40 |
| fultonj | xarlos: ok, so just list your SSD at the end of the list | 18:40 |
| xarlos | Doe. | 18:40 |
| xarlos | *domne | 18:40 |
| fultonj | ceph-volume should do the right thing | 18:41 |
| fultonj | if ironic cleans the osd before each deployment you should be fine | 18:41 |
| fultonj | sounds like you're still stuck on the mon issue though | 18:41 |
| fultonj | so the OSDs don't get set up until the mons are done | 18:41 |
| xarlos | Yes. There's sometihng not quite right. But the mon container on one of my controllers: I couldn't even read any logs. It's like it's got a disk issue of some sort (shouldn't be an actual physical problem). | 18:42 |
| fultonj | use journalctl to read the mon logs | 18:43 |
| fultonj | the ceph containers log differently than you might expect | 18:43 |
| xarlos | And I've deployed a few iterations over the time. The services for the mons suggested they have been up a few days. Wasn't just the ceph logs, messages couldn't be read either. | 18:43 |
| fultonj | are you using network isolation? | 18:43 |
| xarlos | yes. | 18:44 |
| xarlos | (That seemed to be a battle in itself :-)) | 18:44 |
| fultonj | if you do a deployment without network isolation (just don't -e certain files) then ceph will deploy on the provisioning network | 18:44 |
| fultonj | not a production way to do it, but it might help you rule out network issues | 18:44 |
| fultonj | xarlos: you'll want to find out where exactly the mon is failing to get qurom | 18:45 |
| xarlos | Yes, towards the beginning of this deployment, they did complete over the single provisioning network. | 18:45 |
| fultonj | xarlos: ceph did mons did get qurom on provisioning network? | 18:45 |
| fultonj | https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ceph_config.html#tuning-ceph-osd-cpu-and-memory | 18:46 |
| xarlos | It completed a deployment entirely. I tool that as good grace that things looked correct, and then started my woeful path of networking isolation. | 18:46 |
| *** saneax has quit IRC | 18:47 | |
| fultonj | xarlos: It completed a deployment entirely. (with ceph?) | 18:47 |
| fultonj | xarlos: point is, if ceph completed without network isolation and adding it back in, implies that something about your storage network has an issue | 18:48 |
| fultonj | the storage management network is only used by the OSDs to ballance data you're not yet at the point of using it | 18:49 |
| fultonj | in the doc i linked above it shows how to disable mon_host_v1 try that | 18:49 |
| fultonj | see "ceph-ansible 4.0 and newer" | 18:49 |
| xarlos | Mega! I'll add that in too. | 18:51 |
| xarlos | Many thanks for your time, i'll update you soon! | 18:51 |
| fultonj | ack | 18:53 |
| openstackgerrit | Francesco Pantano proposed openstack/puppet-tripleo master: Add certmonger-grafana-refresh script https://review.opendev.org/676395 | 18:55 |
| *** pierreprinetti has joined #tripleo | 18:55 | |
| *** cfontain has joined #tripleo | 18:58 | |
| hrybacki | weshay: can you help me push on https://review.opendev.org/#/c/662068/ and https://review.opendev.org/#/c/674398/ ? rdo failures are unrelated -- been trying to get these in for a minute | 18:59 |
| * weshay looks | 19:00 | |
| weshay | hrybacki +2 +w but note.. the gate is flaky due to the ovh cloud and their proxy.. we may need to recheck | 19:03 |
| hrybacki | weshay: ack, ty | 19:03 |
| hrybacki | weshay++ | 19:03 |
| hrybacki | need a karma bot | 19:03 |
| *** morazi has joined #tripleo | 19:04 | |
| *** cfontain has quit IRC | 19:06 | |
| *** cfontain has joined #tripleo | 19:09 | |
| openstackgerrit | Merged openstack/tripleo-validations stable/stein: Add molecule tests for the undercloud-cpu role https://review.opendev.org/677200 | 19:10 |
| hrybacki | slagle: o/ any chance I could get your eyes on https://review.opendev.org/#/c/675131/ ? | 19:12 |
| slagle | hrybacki: looking | 19:13 |
| hrybacki | thanks slagle! | 19:13 |
| weshay | hrybacki need anything w/ regards to John's work? | 19:15 |
| hrybacki | weshay: I need to circle back around with him and Nate in the morning tbh | 19:16 |
| hrybacki | probably similar -- pushing these patches beyond the gates | 19:16 |
| *** cfontain has quit IRC | 19:18 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Add rhel_containers variable https://review.opendev.org/676474 | 19:19 |
| *** sshnaidm|bbl is now known as sshnaidm | 19:19 | |
| *** dsneddon has quit IRC | 19:20 | |
| *** dtrainor has quit IRC | 19:21 | |
| *** dtrainor has joined #tripleo | 19:23 | |
| *** morazi has quit IRC | 19:24 | |
| *** dsneddon has joined #tripleo | 19:26 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: Make puppet-tripleo compatible with latest puppet-rabbitmq https://review.opendev.org/677173 | 19:27 |
| *** pierreprinetti has quit IRC | 19:33 | |
| *** saneax has joined #tripleo | 19:36 | |
| bandini | mwhahaha: I will -W https://review.opendev.org/#/c/677192/ since https://review.opendev.org/#/c/677173/ seems to be the better way anyhow | 19:37 |
| bandini | scream if you disagree | 19:37 |
| mwhahaha | sure | 19:37 |
| *** morazi has joined #tripleo | 19:37 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Convert firewall rules to use TripleO-Ansible https://review.opendev.org/677237 | 19:38 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Configure time using tripleo-ansible https://review.opendev.org/671166 | 19:38 |
| *** sanjayu_ has joined #tripleo | 19:53 | |
| *** saneax has quit IRC | 19:56 | |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo master: Make the rabbitmq-ready exec more stringent https://review.opendev.org/676431 | 19:56 |
| rlandy|ruck | https://review.opendev.org/#/c/677212/ anyone with +2 rights on openstack/kolla? | 19:56 |
| mwhahaha | nagative | 19:57 |
| *** sanjayu_ has quit IRC | 19:57 | |
| rlandy|ruck | mwhahaha: hmmm - thought you were all powerful | 19:57 |
| mwhahaha | not on kolla | 19:58 |
| *** pierreprinetti has joined #tripleo | 20:02 | |
| *** pierreprinetti has quit IRC | 20:09 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: update gentoo systemd profile to 17.1 from 17.0 https://review.opendev.org/677290 | 20:14 |
| weshay | hrybacki your puppet gate jobs look like they just died | 20:19 |
| weshay | on 2019-08-19 19:42:34.426952 | TASK [upload-logs-swift : Upload logs to swift] | 20:19 |
| *** morazi has quit IRC | 20:22 | |
| *** ansmith_ has quit IRC | 20:24 | |
| hrybacki | :| | 20:29 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-quickstart-extras master: DNM: undercloud_enable_nova: false by default https://review.opendev.org/664170 | 20:30 |
| *** morazi has joined #tripleo | 20:37 | |
| stevebaker | mwhahaha: cloudnull hey, one quirk of dockerhub is that the response for missing content is generally UNAUTHORIZED instead of 404 | 20:44 |
| cloudnull | ORLY!? | 20:44 |
| cloudnull | so how does one differentiate missing vs expired token? | 20:46 |
| cloudnull | there's an error header that's returned, I guess we could try and interpret the string value ? | 20:46 |
| *** mmethot has quit IRC | 20:47 | |
| *** mmethot has joined #tripleo | 20:47 | |
| stevebaker | cloudnull: yeah, good question | 20:49 |
| openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient master: Use reset to fix cmdline https://review.opendev.org/677296 | 20:49 |
| stevebaker | cloudnull: it looks like this change is just for debugging right now, but let me know when you want reviews for landing it https://review.opendev.org/#/c/674919 | 20:52 |
| cloudnull | https://pasted.tech/pastes/ba2569eb91c386c4c84386577729f80191cbb076.raw - these are some of the headers being returned | 20:54 |
| weshay | cloudnull++ stevebaker++ | 20:55 |
| weshay | sshnaidm rlandy|ruck fyi ^ | 20:55 |
| cloudnull | I guess group on the the error key and search for "invalid_token" | 20:55 |
| cloudnull | stevebaker review away, and if you want to push a change, feel free. | 20:56 |
| weshay | cloudnull are you thinking we just log that or take some additional action? | 20:56 |
| sshnaidm | stevebaker, it's proxy actually that returns 401, right? not dockerhub itself | 20:56 |
| weshay | like... send an extra bit of cheese to the ovh proxy gerbel | 20:57 |
| cloudnull | sshnaidm https://pasted.tech/pastes/ba2569eb91c386c4c84386577729f80191cbb076.raw - that's a return from the docker itself | 20:57 |
| stevebaker | cloudnull: is max-age=31536000 for token expiry? maybe we could keep track of the token age and reauth before its needed | 20:57 |
| mwhahaha | 31536000 i slike a year | 20:58 |
| *** raildo has quit IRC | 20:58 | |
| cloudnull | stevebaker its 300 | 20:58 |
| mwhahaha | i think that's the cache expiry | 20:58 |
| stevebaker | sshnaidm: I think the proxy is only for layer requests, and the manifest request goes to the source registry | 20:58 |
| sshnaidm | stevebaker, shouldn't proxy check the cache before? | 20:59 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-heat-templates master: Convert firewall rules to use TripleO-Ansible https://review.opendev.org/677237 | 21:00 |
| stevebaker | sshnaidm: its not a generic http proxy as we know it. Its more like an explicitly configured CDN to offload bulky layer transfers | 21:02 |
| *** mcornea has quit IRC | 21:03 | |
| *** cfontain has joined #tripleo | 21:04 | |
| weshay | stevebaker any idea why this only happens w/ the ovh cloud? | 21:05 |
| sshnaidm | stevebaker, I think it's a regular apache http cache reverse proxy: https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/templates/mirror.vhost.erb#L288 | 21:06 |
| stevebaker | weshay: maybe requests from ovh are sent to a different dockerhub CDN which is less... consistent | 21:06 |
| sshnaidm | stevebaker, at least according to config, if nothing changed | 21:06 |
| weshay | maybe we're using a datacenter geographically far from docker.io? https://www.ovh.com/world/about-us/datacenters.xml | 21:08 |
| weshay | https://www.ovh.com/world/images/about-us/dcTab/bhs1.jpg looks kind of run down lolz | 21:10 |
| *** aakarsh has quit IRC | 21:10 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-common master: Log exceptions when checking status https://review.opendev.org/674919 | 21:12 |
| *** holser has quit IRC | 21:13 | |
| cloudnull | weshay stevebaker ^ updated to look at the returned headers to try and only reauth if we see "invalid_token" | 21:14 |
| *** pkopec has joined #tripleo | 21:14 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/tripleo-common master: Log exceptions when checking status https://review.opendev.org/674919 | 21:14 |
| weshay | sshnaidm based on cloudnull's patch we should update collect logs to parse the debug so we can easily spot wtf is going on | 21:19 |
| cloudnull | lots more logs now | 21:20 |
| *** xek has quit IRC | 21:21 | |
| sshnaidm | weshay, to parse? | 21:21 |
| *** ansmith_ has joined #tripleo | 21:22 | |
| weshay | sshnaidm meh.. we'll chat tomorrow.. don't want to bug you now | 21:22 |
| sshnaidm | ok | 21:24 |
| *** sshnaidm is now known as sshnaidm|afk | 21:24 | |
| *** pkopec has quit IRC | 21:30 | |
| *** morazi has quit IRC | 21:31 | |
| *** morazi has joined #tripleo | 21:33 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Update supplemental role to paramaterize networking https://review.opendev.org/662068 | 21:43 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Add local development with FreeIPA focused featureset043 https://review.opendev.org/674398 | 21:43 |
| weshay | hrybacki ^ | 21:44 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Fixes for deploying nova-less undercloud https://review.opendev.org/677100 | 21:45 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-quickstart-extras master: DNM: undercloud_enable_nova: false by default https://review.opendev.org/664170 | 21:47 |
| *** bfournie has quit IRC | 21:50 | |
| hrybacki | weshay+++++ | 21:54 |
| *** rh-jelabarre has quit IRC | 22:05 | |
| openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient master: Use reset to fix cmdline https://review.opendev.org/677296 | 22:11 |
| openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient master: Use reset to fix cmdline https://review.opendev.org/677296 | 22:14 |
| *** aakarsh has joined #tripleo | 22:35 | |
| *** rlandy|ruck is now known as rlandy|ruck|bbl | 22:52 | |
| *** EmilienM|pto has quit IRC | 22:54 | |
| *** EmilienM has joined #tripleo | 22:56 | |
| *** ChanServ sets mode: +v EmilienM | 22:56 | |
| *** rcernin has joined #tripleo | 22:57 | |
| *** cdearborn has quit IRC | 23:07 | |
| *** morazi has quit IRC | 23:21 | |
| *** d0ugal has quit IRC | 23:25 | |
| *** tkajinam has joined #tripleo | 23:27 | |
| *** morazi has joined #tripleo | 23:38 | |
| *** d0ugal has joined #tripleo | 23:41 | |
| cloudnull | https://pasted.tech/pastes/2e1320c2bada2c9b756e975cecb473f17746a673.raw - weshay sshnaidm|afk mwhahaha stevebaker - still seeing the 401 and a return from the docker api indicating that there's an "invalid_token". | 23:54 |
| cloudnull | https://37ff822ebdc11824c352-ffc80d196410a18186442d9badd30b78.ssl.cf5.rackcdn.com/674919/20/check/tripleo-ci-centos-7-standalone/ca7c796/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 23:54 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
