| *** chem has quit IRC | 00:47 | |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 01:00 |
|---|---|---|
| *** agopi has joined #oooq | 02:11 | |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 03:01 |
| *** skramaja has joined #oooq | 03:02 | |
| *** brault has joined #oooq | 03:09 | |
| *** brault has quit IRC | 03:14 | |
| *** sanjayu has joined #oooq | 03:50 | |
| *** udesale has joined #oooq | 03:55 | |
| *** ykarel has joined #oooq | 04:44 | |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ (1 more message) | 05:01 |
| *** links has joined #oooq | 05:03 | |
| *** ratailor has joined #oooq | 05:20 | |
| *** sanjayu has quit IRC | 05:33 | |
| *** quiquell|off is now known as quiquell | 05:42 | |
| *** bogdando has joined #oooq | 05:55 | |
| *** d0ugal_ has joined #oooq | 05:55 | |
| *** chandankumar is now known as chkumar|ruck | 05:56 | |
| *** d0ugal has quit IRC | 05:56 | |
| chkumar|ruck | %gatestatus | 05:59 |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ (1 more message) | 05:59 |
| *** quiquell has quit IRC | 06:01 | |
| *** quiquell has joined #oooq | 06:01 | |
| *** jtomasek has joined #oooq | 06:18 | |
| *** sanjayu_ has joined #oooq | 06:36 | |
| *** jfrancoa has joined #oooq | 06:40 | |
| *** matbu has quit IRC | 06:41 | |
| *** matbu has joined #oooq | 06:41 | |
| *** chem has joined #oooq | 06:42 | |
| *** skramaja has quit IRC | 06:57 | |
| *** sanjayu_ is now known as saneax | 07:00 | |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ (1 more message) | 07:01 |
| *** amoralej|off is now known as amoralej | 07:02 | |
| *** gkadam has joined #oooq | 07:14 | |
| *** kopecmartin has joined #oooq | 07:23 | |
| *** holser_ has joined #oooq | 07:29 | |
| ssbarnea | simple fix https://review.openstack.org/#/c/571180/ | 08:00 |
| *** d0ugal_ has quit IRC | 08:03 | |
| *** d0ugal has joined #oooq | 08:03 | |
| *** d0ugal has quit IRC | 08:03 | |
| *** d0ugal has joined #oooq | 08:03 | |
| *** tosky has joined #oooq | 08:03 | |
| *** ccamacho has joined #oooq | 08:17 | |
| *** links has quit IRC | 08:24 | |
| *** florianf has joined #oooq | 08:26 | |
| *** links has joined #oooq | 08:26 | |
| *** sshnaidm|off has quit IRC | 08:27 | |
| *** saneax has quit IRC | 08:27 | |
| ssbarnea | a new tox bug that is likely to affect lots of us https://github.com/tox-dev/tox/issues/899#issuecomment-404427845 | 08:29 |
| *** saneax has joined #oooq | 08:32 | |
| quiquell | ssbarnea: it will appear at CI, don't know what tox version we are running on | 08:35 |
| ssbarnea | already found it on ara builds yesterday, expect more to appear on CI. | 08:35 |
| ssbarnea | it was introduced by very new tox release, i hope they will take it down as it woudl minimize the "damage" :) | 08:36 |
| *** tesseract has joined #oooq | 08:37 | |
| quiquell | ssbarnea: We will have to pin all the tox requirements.txt :-) | 08:37 |
| quiquell | fun | 08:37 |
| ssbarnea | even worse I got a very weird behaviour on my machine: it attempted to test using py34 when the tox.ini was mentioning only pe35. | 08:37 |
| arxcruz | marios: hey, can you +2+w https://review.openstack.org/#/c/580384/ ? | 08:38 |
| ssbarnea | is even worse, tox is not in requirement.txt for obvious reasons. | 08:38 |
| *** jaganathan has quit IRC | 08:38 | |
| *** jaganathan has joined #oooq | 08:39 | |
| quiquell | ssbarnea: Humm you are right | 08:40 |
| marios | arxcruz: ack | 08:40 |
| arxcruz | marios: thanks | 08:40 |
| arxcruz | chkumar|ruck: https://review.rdoproject.org/r/#/c/14554/ is it still needed, from amoralej comments | 08:41 |
| chkumar|ruck | arxcruz: nope | 08:42 |
| chkumar|ruck | arxcruz: I need to abandon all those patches | 08:42 |
| chkumar|ruck | arxcruz: I need to take a different route all together | 08:42 |
| arxcruz | chkumar|ruck: https://review.rdoproject.org/r/#/c/14553/ as well | 08:42 |
| ssbarnea | glat to hear the "-e VAR" works on docker, makes cleaner to passenv. | 08:43 |
| chkumar|ruck | arxcruz: all abandoned on rdo side | 08:44 |
| arxcruz | chkumar|ruck: thanks :) | 08:44 |
| chkumar|ruck | quiquell: seen this error http://paste.openstack.org/show/725664/ while creating env using reproducer script | 08:47 |
| chkumar|ruck | ? | 08:47 |
| *** sshnaidm has joined #oooq | 08:48 | |
| *** sshnaidm is now known as sshnaidm|rover | 08:48 | |
| quiquell | chkumar|ruck: Could be a problem with your tenant | 08:48 |
| quiquell | chkumar|ruck: Or that RDO is out of capacity | 08:49 |
| chkumar|ruck | quiquell: https://github.com/openstack-infra/tripleo-ci/blob/master/toci_quickstart.sh#L96 failing at this part | 08:50 |
| arxcruz | marios: done | 08:50 |
| quiquell | chkumar|ruck: Humm reproducer is broken, ENVIRONMENT is not set | 08:51 |
| quiquell | chkumar|ruck: find the TOCI_JOBTYPE env var at the top of the reproducer | 08:51 |
| quiquell | paste it here | 08:51 |
| chkumar|ruck | quiquell: ${TOCI_JOBTYPE:="multinode-1ctlr-featureset033"} | 08:51 |
| chkumar|ruck | quiquell: http://logs.openstack.org/43/579843/13/experimental/tripleo-ci-centos-7-scenario009-multinode-oooq/31c1bec/logs/reproducer-quickstart.sh | 08:51 |
| *** chem has quit IRC | 08:52 | |
| marios | thanks arxcruz revoted | 08:52 |
| arxcruz | sshnaidm|rover: can you take a look on https://review.openstack.org/#/c/580384/ when you have time ? | 08:53 |
| sshnaidm|rover | arxcruz, sure | 08:53 |
| quiquell | chkumar|ruck: How did you executed the reproducer ? | 08:54 |
| quiquell | chkumar|ruck: Did you sources env_vars_to_src.sh ? | 08:54 |
| chkumar|ruck | quiquell: so I found the problem | 08:55 |
| chkumar|ruck | quiquell: so i source env_vars_to_src.sh first | 08:55 |
| chkumar|ruck | then installed tmux | 08:55 |
| sshnaidm|rover | arxcruz, don't you need to specify value when set "-e" for docker? | 08:55 |
| chkumar|ruck | then runned the script that's why env is not exported | 08:55 |
| arxcruz | sshnaidm|rover: according panda|off no | 08:56 |
| quiquell | chkumar|ruck: Yep, you have to source after tmux | 08:56 |
| quiquell | chkumar|ruck: Let's see if we can put tmux on reproducer | 08:56 |
| chkumar|ruck | quiquell: I will propose a patch for tmux switch | 08:56 |
| sshnaidm|rover | arxcruz, and why? | 08:56 |
| arxcruz | sshnaidm|rover: https://review.openstack.org/#/c/580384/4/roles/validate-tempest/templates/run-tempest.sh.j2@49 | 08:57 |
| quiquell | chkumar|ruck: btw, this is known ? http://logs.openstack.org/23/578023/5/gate/tripleo-ci-centos-7-scenario002-multinode-oooq-container/3ce1249/logs/undercloud/home/zuul/tempest/tempest.html.gz | 09:00 |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 09:01 |
| sshnaidm|rover | arxcruz, oh, so you set pythonwarnings before? | 09:01 |
| *** skramaja has joined #oooq | 09:01 | |
| *** saneax has quit IRC | 09:01 | |
| chkumar|ruck | quiquell: yup | 09:01 |
| arxcruz | sshnaidm|rover: yes | 09:01 |
| chkumar|ruck | quiquell: it comes sometimes and sometimes not | 09:01 |
| chkumar|ruck | quiquell: https://bugs.launchpad.net/tripleo/+bug/1773325 | 09:02 |
| openstack | Launchpad bug 1773325 in tripleo "tempest.api.object_storage.test_object_services is failing on scenario002" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 09:02 |
| arxcruz | sshnaidm|rover: since this script has some includes it's hard on the review to understand that :) | 09:02 |
| sshnaidm|rover | arxcruz, in which line do you set it? | 09:03 |
| *** saneax has joined #oooq | 09:03 | |
| arxcruz | sshnaidm|rover: when we source overcloudrc file, it's on the overcloudrc | 09:04 |
| *** links has quit IRC | 09:10 | |
| *** links has joined #oooq | 09:12 | |
| *** panda|off is now known as panda | 09:20 | |
| panda | arxcruz: more according to https://docs.docker.com/engine/reference/run/#env-environment-variables than to me :) | 09:22 |
| arxcruz | sshnaidm|rover: ^ | 09:22 |
| *** kopecmartin is now known as mkopec|lunch | 09:32 | |
| marios | arxcruz: any idea about this tempest error we get on ci-centos-7-scenario002-multinode-oooq-container at https://review.openstack.org/#/c/578023/5 (see my latest comment from just now there) please | 09:52 |
| quiquell | panda: Looks like cleanup dependencies works https://review.openstack.org/#/c/581668/ | 09:56 |
| quiquell | Don't know if I have to test it with a patch at TQE to exercise all CI | 09:57 |
| panda | quiquell: I think it's pretty straightforward patch | 09:59 |
| quiquell | panda: ack | 09:59 |
| sshnaidm|rover | quiquell, hi | 10:10 |
| sshnaidm|rover | quiquell, I saw you talked with rdo folks about moving grafana, did you reach some conclusions..? | 10:11 |
| quiquell | sshnaidm|rover: Let's wait for software-factory.org to upgrade | 10:13 |
| sshnaidm|rover | ok | 10:13 |
| quiquell | sshnaidm|rover: They don't support changes in grafana for commits yet | 10:13 |
| quiquell | sshnaidm|rover: sf at RDO is old | 10:13 |
| sshnaidm|rover | quiquell, and then they'll support grafyaml? | 10:13 |
| quiquell | sshnaidm|rover: Yep, but I am changing grafyaml, it misses a lot of features we need | 10:14 |
| quiquell | sshnaidm|rover: Also trying to port to it, we can do this while wait for sf upgrade | 10:14 |
| sshnaidm|rover | quiquell, where is its sources? | 10:14 |
| sshnaidm|rover | quiquell, yeah, I know they don't have influxdb support.. | 10:15 |
| quiquell | sshnaidm|rover: openstack/grafyaml | 10:15 |
| quiquell | sshnaidm|rover: I was going to start with table panels | 10:15 |
| sshnaidm|rover | great | 10:15 |
| quiquell | sshnaidm|rover: very easy to add new stuff, also pabelenger is quick merged | 10:16 |
| quiquell | sshnaidm|rover: It' doesn't need 2 +2s | 10:16 |
| quiquell | sshnaidm|rover: Example or adding adhoc variables https://review.openstack.org/#/c/581368/ | 10:16 |
| sshnaidm|rover | quiquell, nice | 10:18 |
| sshnaidm|rover | quiquell, do you have a list of features to implement? | 10:19 |
| quiquell | sshnaidm|rover: nop, just porting little by little and implementing the missing one | 10:19 |
| quiquell | sshnaidm|rover: Let me push what I have so far | 10:20 |
| quiquell | sshnaidm|rover: https://review.rdoproject.org/r/14804 | 10:22 |
| arxcruz | marios: concurrency :( | 10:23 |
| *** mkopec|lunch is now known as kopecmartin | 10:27 | |
| quiquell | sshnaidm|rover: Have concerns with grafyaml, it's going to be always a subset of the latest grafana | 10:27 |
| *** kopecmartin has quit IRC | 10:28 | |
| quiquell | sshnaidm|rover: And also I don't know how easy is to changes big dashboards with commits | 10:28 |
| *** kopecmartin has joined #oooq | 10:28 | |
| *** florianf has quit IRC | 10:30 | |
| *** florianf has joined #oooq | 10:32 | |
| sshnaidm|rover | quiquell, well, I don't know too.. maybe need something for testing | 10:34 |
| sshnaidm|rover | quiquell, like testing dashboard or kind of.. | 10:34 |
| *** udesale has quit IRC | 10:35 | |
| quiquell | sshnaidm|rover: meanwhile we can store into yaml what we have now, to simplify editing | 10:37 |
| quiquell | sshnaidm|rover: It's good enough and it's not a subset | 10:37 |
| quiquell | sshnaidm|rover: testing dashboard... we can use some kind of GUI testing | 10:38 |
| sshnaidm|rover | quiquell, but we can't store in yaml everything we have now, right? | 10:39 |
| quiquell | sshnaidm|rover: Yep, and change grafana-export to read yaml or json | 10:40 |
| quiquell | sshnaidm|rover: Depending on the extension or a command line arg | 10:40 |
| quiquell | sshnaidm|rover: Also there are some stuff that we can clean tooo | 10:40 |
| quiquell | sshnaidm|rover: It's a poor man grafyaml but more flexible | 10:40 |
| sshnaidm|rover | quiquell, yeah | 10:41 |
| sshnaidm|rover | quiquell, so we need another instance to use grafyaml in it | 10:41 |
| sshnaidm|rover | and test patches | 10:42 |
| quiquell | sshnaidm|rover: Kind of, or we don't use grafyaml all together | 10:42 |
| quiquell | sshnaidm|rover: We are going to update their schema forever | 10:42 |
| sshnaidm|rover | yeah... | 10:43 |
| quiquell | sshnaidm|rover: Maybe it's better to have a custom grafana with all the plugins that we want | 10:44 |
| quiquell | sshnaidm|rover: And the version that we want, it doesn't requires to much of maintenance | 10:44 |
| chkumar|ruck | sshnaidm|rover: https://bugs.launchpad.net/tripleo/+bug/1780224 we are hitting again in master noop job scenario 1 | 10:54 |
| openstack | Launchpad bug 1780224 in tripleo "ERROR configuring keystone_init_tasks in tripleo-ci-centos-7-scenario001-multinode-oooq-container job" [High,Triaged] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm) | 10:54 |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 11:01 |
| quiquell | sshnaidm|rover: What's the best way to store secrets at openstack ? | 11:15 |
| sshnaidm|rover | quiquell, in openstack..? | 11:15 |
| *** atoth has joined #oooq | 11:17 | |
| quiquell | sshnaidm|rover: Found how we do at tripleo-infra | 11:17 |
| *** amoralej is now known as amoralej|lunch | 11:18 | |
| ssbarnea | quick one https://review.openstack.org/#/c/581788/ please, already passed. | 11:20 |
| sshnaidm|rover | quiquell, you mean ansible vault? | 11:22 |
| quiquell | sshnaidm|rover: Humm was replicating what we do with promoter at ci-scripts/infra-setup/tenant_vars | 11:23 |
| quiquell | will look into ansible vault | 11:23 |
| sshnaidm|rover | quiquell, we don't have vault yet in tripleo-infra | 11:23 |
| sshnaidm|rover | quiquell, all secrets are just deployed manually | 11:24 |
| quiquell | sshnaidm|rover: Then I will go to same as promoter | 11:24 |
| quiquell | ok | 11:24 |
| sshnaidm|rover | quiquell, yeah, that's fine, the ansible-vault is the plan for future | 11:24 |
| sshnaidm|rover | ssbarnea, it's not tested in jobs | 11:27 |
| sshnaidm|rover | ssbarnea, otherwise it would fail on syntax error | 11:27 |
| sshnaidm|rover | ssbarnea, this kind of patches we need to test locally unfortunately | 11:27 |
| sshnaidm|rover | ssbarnea, I don't know how even ansible lint missed this, need to check it.. | 11:28 |
| panda | marios: do you know what's happening in http://logs.openstack.org/76/581376/5/check/tripleo-ci-centos-7-3nodes-multinode/b141c5e/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-11_22_34_43 ? | 11:28 |
| panda | marios: what's that file that is missing ? | 11:29 |
| ssbarnea | quiquell: exactly, I did had a good impression about ansilbe-linter, not to mention that I had the impression that the gate was testing the code,...or at least most of it. | 11:29 |
| ssbarnea | this kind of error should have being catched by either linter or the gate, but this time human did the best work :D | 11:29 |
| marios | panda: no looks like a new one ... seems to be about the upgradeinit but not sure what is giving this error ? mistral?ansible? looking more | 11:30 |
| panda | marios: thanks, 3 nodes is the only job missing to migrate at this point, standalone works | 11:31 |
| sshnaidm|rover | ssbarnea, I'm sshnaidm|rover :) I know it's confusing.. | 11:32 |
| sshnaidm|rover | ssbarnea, linter doesn't check this playbook: http://logs.openstack.org/88/581788/1/check/openstack-tox-linters/0b8f715/job-output.txt.gz#_2018-07-12_08_13_04_646931 | 11:33 |
| ssbarnea | no worry, before fixing the typo i will try to test if I can make the linter catch it | 11:33 |
| marios | panda: trying to find where that came from still, but it cant be upgrade related anyway since there is no upgrade here in this job (though the error is indeed about the upgrade init file...) | 11:40 |
| panda | marios: that's what I imagined. | 11:40 |
| bogdando | folks, shall I dup these https://review.openstack.org/#/c/465047/5/roles/collect-logs/defaults/main.yml for tripleo-ci collect logs? | 11:43 |
| bogdando | or is it enough to only change either place of two? | 11:43 |
| bogdando | PTAL btw | 11:43 |
| bogdando | the patch is wanted by upgrades team to diverge tht for undercloud vs overcloud and mixed upgrades | 11:44 |
| sshnaidm|rover | bogdando, is it used in CI? /home/*/undercloud_custom_tht_script.log | 11:45 |
| bogdando | sshnaidm|rover: it becomes in the patch | 11:46 |
| sshnaidm|rover | bogdando, it's very controversial patch | 11:47 |
| sshnaidm|rover | bogdando, firstly, please don't change overcloud_templates_path | 11:47 |
| *** quiquell is now known as quiquell|lunch | 11:47 | |
| bogdando | sshnaidm|rover: I believe it has to | 11:48 |
| bogdando | otherwise we have a mess un the tht script | 11:48 |
| bogdando | in | 11:48 |
| sshnaidm|rover | bogdando, it's default and used so in all featuresets | 11:48 |
| *** skramaja_ has joined #oooq | 11:49 | |
| sshnaidm|rover | bogdando, that's fine if we need to override it in some of tasks | 11:49 |
| bogdando | look folks, there is a real mess for mixed updates | 11:49 |
| *** skramaja has quit IRC | 11:49 | |
| bogdando | you can only install tht from rpm to /usr/share | 11:49 |
| bogdando | then, you have to diverge for mixed undercloud vs overcloud upgrades, which technically may be using 4 versions | 11:49 |
| *** skramaja_ is now known as skramaja | 11:50 | |
| bogdando | pre-update UC, pre-update OC, post-update UC, post-update OOC | 11:50 |
| sshnaidm|rover | panda, I believe there is a lot of design questions, can you look: https://review.openstack.org/#/c/465047 | 11:50 |
| bogdando | so we really should not mess in /usr/share IMO, but it's up to DFG upgrades how they would want thatpatch | 11:50 |
| bogdando | so yeah, I can improve that default changing not globally prolly | 11:51 |
| sshnaidm|rover | bogdando, who from upgrades is working on that? | 11:51 |
| bogdando | jfrancoa: ^^ | 11:51 |
| sshnaidm|rover | jfrancoa, bogdando I think we need a proper problem definition before.. | 11:51 |
| bogdando | right, IIUC jfrancoa wanted exactly what I described - a smooth way to diverge t-h-t used for mixed upgrades | 11:52 |
| panda | OMG this is huge. | 11:52 |
| bogdando | given that now we have not only overcloud deployed from it, but undercloud as well, and each has its upgrade path | 11:52 |
| bogdando | so there is 2x2 matrix or so | 11:53 |
| bogdando | I let upgrade folks to comment on that | 11:53 |
| bogdando | panda: just cuz of huge shell scripts moved from role to another one ;) | 11:54 |
| bogdando | to be shared for OC and Uc | 11:54 |
| bogdando | marios: ^^ | 11:54 |
| *** agopi has quit IRC | 11:55 | |
| panda | bogdando: so the problem is "I want to be able to inject two sets of custom THT in a way that I can test upgrades too" ? | 11:55 |
| bogdando | panda: yes, as I can see the request | 11:55 |
| *** udesale has joined #oooq | 11:57 | |
| *** saneax has quit IRC | 11:58 | |
| bogdando | sshnaidm|rover: the problem definition https://bugs.launchpad.net/tripleo/+bug/1781227 | 11:58 |
| openstack | Launchpad bug 1781227 in tripleo "Overcloud upgrade CI job doesn't load templates from /usr/share/openstack-tripleo-heat-templates" [High,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 11:58 |
| bogdando | basically, that's why I wanted to ditch that path | 11:59 |
| bogdando | and rely on working dir | 11:59 |
| marios | bogdando: which patch? reading back but agree on not messing with /usr/share for tht | 12:00 |
| bogdando | marios: see https://bugs.launchpad.net/tripleo/+bug/1781227 | 12:00 |
| openstack | Launchpad bug 1781227 in tripleo "Overcloud upgrade CI job doesn't load templates from /usr/share/openstack-tripleo-heat-templates" [High,In progress] - Assigned to Bogdan Dobrelya (bogdando) | 12:00 |
| bogdando | https://review.openstack.org/#/c/465047/ | 12:00 |
| marios | panda: re that multinode job. fwiw the error is coming from this https://github.com/openstack/tripleo-common/blob/master/tripleo_common/templates/deployments.yaml#L3 however i also see the file it complains about, is there at http://logs.openstack.org/76/581376/5/check/tripleo-ci-centos-7-3nodes-multinode/b141c5e/logs/undercloud/var/lib/mistral/d9b8772a-af39-427f-b9dd-6f53baa26ccc/ControllerApi/centos-7-ovh-bhs1-0000668818/UpgradeInitDeployment.g | 12:01 |
| marios | bogdando: ack checking | 12:03 |
| marios | bogdando: ack i'll have a look at the review its a bigun ;) but probably tomorrow | 12:04 |
| bogdando | marios: I think that patch may also help for https://trello.com/c/vzjMRcHt/28-upgrades-ci-with-the-standalone-installer | 12:08 |
| bogdando | if just thinking of undercloud_* as standalone_* | 12:08 |
| bogdando | ack, waiting for feedback tomorrow then :) | 12:09 |
| marios | bogdando: cool thanks for pointer to the card | 12:09 |
| *** trown|outtypewww is now known as trown | 12:10 | |
| *** ratailor has quit IRC | 12:11 | |
| *** quiquell|lunch is now known as quiquell | 12:12 | |
| panda | marios: the missing file ends with a 6 | 12:15 |
| panda | marios: centos-7-ovh-bhs1-0000668818 vs centos-7-ovh-bhs1-0000668816 | 12:16 |
| marios | panda: ah weird | 12:16 |
| panda | probably the file for the 3rd node is missing | 12:16 |
| marios | panda: well spotted ;) | 12:17 |
| *** amoralej|lunch is now known as amoralej | 12:20 | |
| *** saneax has joined #oooq | 12:20 | |
| *** atoth has quit IRC | 12:22 | |
| *** ykarel is now known as ykarel|afk | 12:25 | |
| *** ykarel|afk has quit IRC | 12:30 | |
| *** apetrich has quit IRC | 12:32 | |
| jfrancoa | bogdando: sshnaidm|rover: I just came from lunch | 12:33 |
| * jfrancoa having a look at https://review.openstack.org/#/c/465047 | 12:34 | |
| bogdando | yes, please go for it, I've added enough comments to have it looking not so big change in fact | 12:34 |
| bogdando | the only change is basically for overcloud_templates_path defaults, but that's totally discussible | 12:35 |
| bogdando | other changes should not be breaking for any release (hopefully) or config matrix :) | 12:35 |
| *** skramaja has quit IRC | 12:37 | |
| *** rlandy has joined #oooq | 12:38 | |
| *** ykarel|afk has joined #oooq | 12:38 | |
| weshay | sshnaidm|rover, chkumar|ruck upload is still failing I take it? | 12:44 |
| sshnaidm|rover | weshay, yeah, last time it failed, fixed by tristan: https://review.rdoproject.org/r/#/c/14799/ | 12:45 |
| sshnaidm|rover | weshay, looking now | 12:45 |
| sshnaidm|rover | weshay, I have images, so I'm ready to push them for master.. | 12:45 |
| *** agopi has joined #oooq | 12:46 | |
| weshay | sshnaidm|rover, perfect! thank you | 12:46 |
| weshay | chkumar|ruck, I was looking for the trello card I created re: tempest w/o a skip list | 12:46 |
| weshay | couldn't find it :( | 12:46 |
| weshay | trello-- | 12:47 |
| rlandy | panda: hi - I need to add a sprint card for the work I am doing with the perf team | 12:47 |
| sshnaidm|rover | weshay, let's wait 30 min and wee what happens with new upload job, if it fails - I'll push | 12:47 |
| weshay | sshnaidm|rover, aye | 12:47 |
| chkumar|ruck | weshay: https://trello.com/c/sIF5LG9K/857-add-24hr-periodic-tempest-job-w-o-skip-list | 12:48 |
| rlandy | sshnaidm|rover: hi ... wrt zuulv3 migration in rdocloud, have you come across this, | 12:48 |
| rlandy | The project "git.openstack.org/openstack/browbeat" was not found. All | 12:48 |
| rlandy | projects referenced within a Zuul configuration must first be added to | 12:48 |
| rlandy | the main configuration file by the Zuul administrator. | 12:48 |
| weshay | panda, anything we need to cover before the scrum? | 12:48 |
| weshay | chkumar|ruck, thanks | 12:49 |
| rlandy | not on that repo exactly but with referenced projects | 12:49 |
| panda | rlandy: ok | 12:49 |
| sshnaidm|rover | rlandy, hmm... which job is it? | 12:49 |
| weshay | ssbarnea, doing well? | 12:49 |
| rlandy | sshnaidm|rover: I am adding a new project in our testing ... https://review.rdoproject.org/r/#/c/14772/ | 12:50 |
| rlandy | clearly I missed a step | 12:50 |
| rlandy | I guess I don;t know what the main configuration file is | 12:51 |
| ssbarnea | weshay: yep well, regot acces to RHU labs and also learnign about how to bring oooq changes to meargeable state. | 12:51 |
| *** atoth has joined #oooq | 12:52 | |
| ssbarnea | also discovered an interesting tox change this morning. | 12:52 |
| rlandy | ykarel|afk: ^^ do you know where I find this main configuration file? | 12:53 |
| rlandy | panda: do I just add the card? | 12:53 |
| ykarel|afk | rlandy, yes, let me grab a link | 12:54 |
| rlandy | https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/projects.yaml? | 12:54 |
| *** ykarel|afk is now known as ykarel | 12:54 | |
| ykarel | rlandy, you need to submit a patch like:- https://softwarefactory-project.io/r/#/c/12800/ | 12:55 |
| ykarel | in softwarefactory config | 12:55 |
| panda | rlandy: yes, we can discuss it in the meeteing | 12:56 |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ (1 more message) | 13:01 |
| panda | rlandy: meeeting ? | 13:01 |
| panda | ssbarnea: meeting ? | 13:01 |
| panda | rfolco: meeting ? | 13:01 |
| ssbarnea | oops, sorry, calendar didn't popup. | 13:02 |
| *** udesale has quit IRC | 13:15 | |
| rlandy | ykarel: stupid question - but where is that repo? | 13:19 |
| chkumar|ruck | weshay: sshnaidm|rover what to od with scenario002 job having swift tempest tests failing with 412 error code while creating swift container tests, there is nothing in swift container logs also? | 13:20 |
| ykarel | rlandy, git clone https://softwarefactory-project.io/r/config | 13:20 |
| sshnaidm|rover | chkumar|ruck, does it fail always? | 13:20 |
| rlandy | ykarel: ty :) | 13:20 |
| chkumar|ruck | sshnaidm|rover: it fails one time, then do recheck it passes, for most of the patches, http://zuul.openstack.org/builds.html?job_name=tripleo-ci-centos-7-scenario002-multinode-oooq-container | 13:21 |
| chkumar|ruck | the job is more than 8k sec, it has failed with same reason | 13:22 |
| chkumar|ruck | it is started poping up from yesterday | 13:22 |
| chkumar|ruck | sshnaidm|rover: https://bugs.launchpad.net/tripleo/+bug/1773325 , I am not sure what to do in this case | 13:23 |
| openstack | Launchpad bug 1773325 in tripleo "tempest.api.object_storage.test_object_services is failing on scenario002" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 13:23 |
| *** anande has joined #oooq | 13:25 | |
| chkumar|ruck | sshnaidm|rover: ykarel post failure with periodic job https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-queens-upload/60b1009/job-output.txt.gz#_2018-07-12_13_07_44_402929 | 13:26 |
| ykarel | chkumar|ruck, https://review.rdoproject.org/r/#/c/14807 | 13:32 |
| chkumar|ruck | ykarel: I think we need to reschedule the periodic job then after merging this? | 13:33 |
| ykarel | chkumar|ruck, if all jobs pass sshnaidm|rover will promote manually | 13:34 |
| ykarel | sshnaidm|rover, from where u will get the image? will u build that? | 13:36 |
| sshnaidm|rover | ykarel, yeah, building locally | 13:36 |
| ykarel | sshnaidm|rover, okk | 13:36 |
| * chkumar|ruck headed home | 13:37 | |
| sshnaidm|rover | chkumar|ruck, I'm off tomorrow, I think weshay will cover me | 13:40 |
| weshay | aye | 13:40 |
| rlandy | ykarel: do I need to be added to that project? rlandy@softwarefactory-project.io: Permission denied | 13:41 |
| weshay | sshnaidm|rover, https://images.rdoproject.org/master/rdo_trunk/e0c5c24e3bca25349894b8055589d88a16f4b894_0d0d3bed/ | 13:41 |
| weshay | sshnaidm|rover, is that manual or from the job? | 13:42 |
| ykarel | rlandy, your pub keys to gerrit https://softwarefactory-project.io/r/#/settings/ssh-keys | 13:42 |
| sshnaidm|rover | weshay, being uploaded now | 13:43 |
| sshnaidm|rover | weshay, from my laptop | 13:43 |
| weshay | k | 13:43 |
| weshay | sshnaidm|rover, please hand off anything I need to fix the upload issue before you go :) | 13:44 |
| sshnaidm|rover | weshay, which issue? | 13:44 |
| weshay | image upload | 13:44 |
| sshnaidm|rover | weshay, ?? | 13:44 |
| weshay | assuming it did not work again.. or did the latest patch not land in time | 13:44 |
| sshnaidm|rover | weshay, I've fixed it | 13:45 |
| weshay | sshnaidm|rover, ok.. so it's fixed, but have not yet see a job upload images successfully, should happen $next run | 13:45 |
| weshay | is that correct? | 13:45 |
| sshnaidm|rover | weshay, yeah, sure, we'll see | 13:45 |
| weshay | k. .thanks | 13:46 |
| sshnaidm|rover | weshay, oh, sorry, my fault, read you message incorrectly | 13:46 |
| rlandy | ykarel: https://softwarefactory-project.io/r/#/c/13019/ | 13:47 |
| sshnaidm|rover | weshay, stopped for now promoter service.. | 13:47 |
| weshay | sshnaidm|rover, ? | 13:47 |
| weshay | I'm not following.. | 13:47 |
| ykarel | rlandy, need to ask tristanC or paul to get it merged | 13:48 |
| rlandy | ykarel: ok - will do - thanks | 13:48 |
| sshnaidm|rover | weshay, I stopped promotion service while images are being uploaded | 13:48 |
| weshay | ok | 13:48 |
| weshay | sshnaidm|rover, do we have images for queens and master? | 13:48 |
| sshnaidm|rover | weshay, for master, queens is in the progress | 13:48 |
| weshay | rock | 13:49 |
| *** links has quit IRC | 13:50 | |
| ykarel | rlandy, also you need to add job definition to rdo-jobs and project config to config, like https://review.rdoproject.org/r/#/c/14516/ and https://review.rdoproject.org/r/#/c/14508/ | 13:51 |
| ykarel | config/zuul and config/jobs directories were for zuul v2, which are not used now | 13:53 |
| ykarel | refering https://review.rdoproject.org/r/#/c/14772 | 13:53 |
| radez | rlandy: the env template you gave the me other day, do you have one that deploys ironic in the overcloud? | 13:53 |
| *** anande has quit IRC | 13:58 | |
| rlandy | radez: I don't but that's a good idea | 13:59 |
| rlandy | weshay: ^^ is this something we want to add a featureset for? | 13:59 |
| radez | rlandy: ok, I'm actually working on that so i'll try and pass it back to you when I get it working | 13:59 |
| weshay | rlandy, for deploying ironic in the overcloud? | 14:04 |
| *** apetrich has joined #oooq | 14:05 | |
| weshay | rlandy, I guess that should show up in the ci/environment scenarios | 14:05 |
| rlandy | weshay: yeah - I don;t think we have/test that | 14:08 |
| radez | I'll get back to you what I come up with incase you want to add a job for it | 14:09 |
| rfolco | panda, would you please w+ https://review.openstack.org/#/c/581814/ | 14:14 |
| panda | rfolco: you're late | 14:15 |
| rfolco | meh | 14:15 |
| panda | rfolco: needed to chat ? | 14:15 |
| *** ccamacho has quit IRC | 14:16 | |
| arxcruz | sshnaidm|rover: panda rlandy https://review.openstack.org/#/c/570884/ | 14:16 |
| *** quiquell is now known as quiquell|off | 14:16 | |
| rfolco | panda, we can chat tomorrow, I'll play a bit more with the code. My question was about 'periodic' jobs... and dryrun cases.... | 14:16 |
| arxcruz | when you guys have time | 14:16 |
| arxcruz | weshay: ^ | 14:16 |
| panda | rfolco: ok | 14:17 |
| weshay | arxcruz, chkumar|ruck https://review.openstack.org/#/c/574731/ | 14:20 |
| *** ccamacho has joined #oooq | 14:21 | |
| weshay | sshnaidm|rover, on Sunday / Monday can you please work w/ bogdando on https://review.openstack.org/#/c/465047/ | 14:23 |
| bogdando | on Sunday? :O | 14:24 |
| weshay | sshnaidm|rover, help him bring the design to the team | 14:24 |
| sshnaidm|rover | weshay, if Ci will be ok | 14:24 |
| bogdando | :D | 14:24 |
| weshay | sshnaidm|rover, right.. as time permits of course | 14:24 |
| weshay | bogdando, this is a fairly large patch | 14:24 |
| bogdando | do you guys work on Sundays?.. og gush | 14:24 |
| sshnaidm|rover | weshay, I think it could be just review session like we did sometimes | 14:24 |
| * weshay reads through the bugs | 14:24 | |
| sshnaidm|rover | bogdando, yeah, who breaks CI remains to work in Sunday | 14:25 |
| weshay | bogdando, we may need your assistance in review | 14:25 |
| bogdando | okkkay | 14:25 |
| bogdando | :) | 14:27 |
| *** ykarel is now known as ykarel|away | 14:27 | |
| bogdando | do you chain the duties for the most guiltiest? | 14:27 |
| bogdando | like in army :D | 14:28 |
| bogdando | but one can't take more than 3 in a raw IIRC | 14:28 |
| *** quiquell|off is now known as quiquell | 14:32 | |
| weshay | sshnaidm|rover, what was the name of the tool dci was coming up w/ ? | 14:32 |
| quiquell | panda, rfolco: Maybe we can try to merge https://review.openstack.org/#/c/581331/ | 14:33 |
| quiquell | panda, rfolco: Does it needs a extensive CI run ? | 14:33 |
| panda | quiquell: change this https://review.openstack.org/#/c/579653/ to depend on your patch | 14:34 |
| panda | quiquell: so we can test scenarios too | 14:34 |
| panda | at least | 14:34 |
| panda | quiquell: you can replace the existing depends on, that change is merged | 14:35 |
| quiquell | panda: Will do, tomorrow we decide | 14:35 |
| quiquell | created one with tqe too https://review.openstack.org/#/c/582198/ | 14:37 |
| sshnaidm|rover | bogdando, see you in Sunday, man | 14:37 |
| *** quiquell is now known as quiquell|off | 14:37 | |
| jrist | hey guys | 14:38 |
| jrist | how do we re-enable tempest on oooq undercloud install? | 14:38 |
| sshnaidm|rover | arxcruz, ^^ | 14:39 |
| jrist | asking for a friend... :) | 14:39 |
| arxcruz | jrist: run_tempest: true | 14:40 |
| jrist | pardon my ignorance, but where? :) | 14:40 |
| jrist | unrelated: getting lots of these | 14:40 |
| jrist | [DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|failed` instead use `result is failed`. This feature will be removed in version 2.9. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. | 14:40 |
| jrist | arxcruz: pardon my ignorance, but where? :) | 14:43 |
| arxcruz | jrist: in the featureset, or parse this as env in ansible | 14:44 |
| *** dtrainor has joined #oooq | 14:47 | |
| bogdando | sshnaidm|rover: ;) | 14:48 |
| bogdando | updated the Scarry Patch, to have less work on sunday | 14:48 |
| bogdando | no more global changes! | 14:48 |
| sshnaidm|rover | bogdando, ok, so we can work only day in Sun, and rest in night! | 14:50 |
| bogdando | yay | 14:50 |
| jrist | arxcruz: sorry you've lost me again. what do you mean by "in the featureset" | 14:51 |
| sshnaidm|rover | jrist, which job is it? | 14:54 |
| chkumar|ruck | jrist: for https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset003.yml#L7 | 15:00 |
| chkumar|ruck | for undercloud oooq | 15:00 |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ (1 more message) | 15:01 |
| weshay | arxcruz, ping | 15:01 |
| weshay | mtg | 15:01 |
| arxcruz | weshay: i'm on your bj | 15:01 |
| chkumar|ruck | arxcruz: bj link | 15:01 |
| *** ratailor has joined #oooq | 15:02 | |
| kopecmartin | arxcruz, chkumar|ruck https://bluejeans.com/7050859455/ | 15:02 |
| *** saneax has quit IRC | 15:04 | |
| chkumar|ruck | panda: sshnaidm|rover https://review.openstack.org/#/c/572347/ | 15:08 |
| *** bogdando has quit IRC | 15:11 | |
| sshnaidm|rover | chkumar|ruck, commented | 15:13 |
| jrist | chkumar|ruck, ssbarnea just trying to do oooq from my laptop to a machine with tempest installed and configured | 15:14 |
| rlandy | sshnaidm|rover: https://review.rdoproject.org/r/#/c/14808/ - did I do that right? | 15:19 |
| rlandy | oh I need playbooks | 15:20 |
| sshnaidm|rover | rlandy, yep, need to create playbooks/legacy/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053-master/run.yaml and post | 15:21 |
| rlandy | sshnaidm|rover: yep - just noticed that - adding | 15:21 |
| *** ccamacho has quit IRC | 15:26 | |
| sshnaidm|rover | rlandy, is timeout ok there? it's 3 hours, maybe need more? | 15:32 |
| rlandy | sshnaidm|rover: idk - I don;t want this to run forever | 15:32 |
| *** ykarel|away has quit IRC | 15:32 | |
| rlandy | we want minimal testing | 15:33 |
| rlandy | sshnaidm|rover: nothing longer than upgrades - what is that? | 15:33 |
| sshnaidm|rover | rlandy, I dunno.. anyway, we can always increase | 15:34 |
| rlandy | sshnaidm|rover: also not sure what to do about https://review.rdoproject.org/r/#/c/14772/4/jobs/tripleo-upstream.yml - do we just remove that now? | 15:34 |
| rlandy | that I have added the legacy job? | 15:35 |
| rlandy | or we keep both? | 15:35 |
| sshnaidm|rover | rlandy, it's not used anymore | 15:35 |
| rlandy | ok - will remove that | 15:35 |
| rlandy | sshnaidm|rover: so this right? https://review.rdoproject.org/r/#/c/14772/4/zuul.d/tripleo.yaml and I get rid of https://review.rdoproject.org/r/#/c/14772/4/zuul/upstream.yaml? | 15:36 |
| rlandy | or I define tripleo-browbeat-check-branchless to contain legacy? | 15:37 |
| sshnaidm|rover | rlandy, you configure jobs in one place | 15:38 |
| sshnaidm|rover | rlandy, in rdo-jobs | 15:38 |
| chkumar|ruck | weshay: https://bugs.launchpad.net/tripleo/+bug/1773325 | 15:38 |
| openstack | Launchpad bug 1773325 in tripleo "tempest.api.object_storage.test_object_services is failing on scenario002" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 15:38 |
| sshnaidm|rover | rlandy, so you can leave there project template only | 15:38 |
| rlandy | sshnaidm|rover: ok - what about triggering that test with tq/tqe? | 15:39 |
| sshnaidm|rover | rlandy, sorry? | 15:40 |
| rlandy | sshnaidm|rover: https://review.rdoproject.org/r/#/c/14772/4/zuul/upstream.yaml | 15:40 |
| rlandy | I get rid of the template here and add legacy jobs explicitly to https://review.rdoproject.org/r/#/c/14772/4/zuul.d/tripleo.yaml? | 15:41 |
| sshnaidm|rover | rlandy, sorry, I don't follow | 15:41 |
| sshnaidm|rover | rlandy, you configured jobs in rdo-jobs repo patch | 15:42 |
| sshnaidm|rover | rlandy, so now need to configure project template to trigger them, and you do it in right place | 15:42 |
| *** jfrancoa has quit IRC | 15:42 | |
| sshnaidm|rover | rlandy, but jobs definitions shouldn't be there, because you set them in rdo-jobs | 15:42 |
| rlandy | sshnaidm|rover: I am talking about the template ... | 15:43 |
| rlandy | tripleo-browbeat-check-branchless looks wrong | 15:43 |
| sshnaidm|rover | rlandy, template of what? | 15:44 |
| rlandy | sshnaidm|rover: I want to run the 053 legacy test with changes in browbeat, tq, tqe, tripleo-ci | 15:44 |
| rlandy | what do I need? | 15:45 |
| *** gkadam has quit IRC | 15:47 | |
| sshnaidm|rover | rlandy, you need to change this file only now: https://review.rdoproject.org/r/#/c/14772/4/zuul.d/tripleo.yaml | 15:49 |
| sshnaidm|rover | rlandy, maybe this too, not sure https://review.rdoproject.org/r/#/c/14772/4/sanity-check.py | 15:49 |
| sshnaidm|rover | but won't hurt | 15:49 |
| sshnaidm|rover | rlandy, lines 829-839 look correct | 15:50 |
| sshnaidm|rover | but tripleo-browbeat-check-branchless template doesn't exist | 15:51 |
| weshay | chkumar|ruck, http://tripleo.org/contributor/check_gates.html#troubleshooting-a-failed-job | 15:52 |
| weshay | https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/getthelogs | 15:52 |
| sshnaidm|rover | rlandy, if you want to create it, you need to define "project-templates:" like in https://review.rdoproject.org/r/#/c/14772/4/zuul/upstream.yaml | 15:52 |
| rlandy | sshnaidm|rover: how about this ... https://review.rdoproject.org/r/#/c/14772/ | 15:55 |
| rlandy | seem sane? | 15:55 |
| sshnaidm|rover | rlandy, yep, exactly, commented | 16:02 |
| sshnaidm|rover | rlandy, didn't you want it on tripleo-ci too? | 16:03 |
| *** hamzy_ has quit IRC | 16:04 | |
| *** hamzy_ has joined #oooq | 16:04 | |
| rlandy | sshnaidm|rover: yep - will add to tripleo-ci | 16:06 |
| *** vinaykns has joined #oooq | 16:06 | |
| *** sanjayu has joined #oooq | 16:06 | |
| rlandy | sshnaidm|rover: https://review.rdoproject.org/r/#/c/14772/ and https://review.rdoproject.org/r/#/c/14808/ (one more time, pls) | 16:08 |
| *** sanjayu has quit IRC | 16:11 | |
| *** panda is now known as panda|off | 16:12 | |
| *** ykarel|away has joined #oooq | 16:12 | |
| sshnaidm|rover | rlandy, last comment on https://review.rdoproject.org/r/#/c/14772 | 16:12 |
| rlandy | sshnaidm|rover: thanks - fixed | 16:15 |
| vinaykns | Hello channel...I am having issue with deploying 000q on a VM. | 16:17 |
| vinaykns | http://pastebin.test.redhat.com/617020 | 16:17 |
| vinaykns | At least what i can see is...it is unable to modify the disk image. | 16:18 |
| *** ratailor has quit IRC | 16:23 | |
| sshnaidm|rover | rlandy, thanks, +w | 16:27 |
| sshnaidm|rover | rlandy, you can try trigger them in 5 min | 16:27 |
| sshnaidm|rover | weshay, promoter SUCCESS promoting tripleo-ci-testing as current-tripleo | 16:30 |
| sshnaidm|rover | weshay, for master, queens is in progress | 16:30 |
| weshay | sshnaidm|rover++ | 16:31 |
| hubbot` | weshay: sshnaidm|rover's karma is now 1 | 16:31 |
| weshay | sshnaidm|rover++ | 16:31 |
| hubbot` | weshay: sshnaidm|rover's karma is now 2 | 16:31 |
| weshay | sshnaidm|rover++ | 16:31 |
| hubbot` | weshay: sshnaidm|rover's karma is now 3 | 16:31 |
| weshay | thank you | 16:32 |
| *** ykarel|away has quit IRC | 16:32 | |
| chkumar|ruck | sshnaidm|rover++ A great Rover and Friend :-) | 16:42 |
| hubbot` | chkumar|ruck: sshnaidm|rover's karma is now 4 | 16:42 |
| sshnaidm|rover | chkumar|ruck, :) | 16:42 |
| chkumar|ruck | does hubbot` stores comments also? | 16:43 |
| sshnaidm|rover | chkumar|ruck, I don't think so | 16:46 |
| sshnaidm|rover | rlandy, can you take a look please at https://review.openstack.org/#/c/576816/ ? I think it's better way to install additional roles than adding them to extras file | 16:47 |
| sshnaidm|rover | rlandy, like browbeat, etc | 16:48 |
| sshnaidm|rover | rlandy, in the scope of this sprint we can even configure it as different var for job | 16:48 |
| rlandy | sshnaidm|rover: weshay and I discussed that yesterday | 16:48 |
| rlandy | sshnaidm|rover: I think that review has a lot of merit for something we want to add occasionally | 16:49 |
| rlandy | weshay mentioned that we are going to need to start adding tempest related repos in requirements as well | 16:49 |
| rlandy | we can certainly discuss the best way to conquer that | 16:49 |
| sshnaidm|rover | rlandy, yeah | 16:50 |
| rlandy | but it is my understanding that if we just add it in quickstart, we leave out some use cases where people call ansible-playbook directly | 16:50 |
| rlandy | so I am +1 on the idea | 16:50 |
| rlandy | just not sure it fits in this case | 16:50 |
| rlandy | open to discussion on it | 16:50 |
| sshnaidm|rover | rlandy, in CI we still use quickstart.sh for bootstrap | 16:51 |
| sshnaidm|rover | rlandy, and in any other way we need to use something to bootstrap - installing venv, packages and roles | 16:51 |
| sshnaidm|rover | rlandy, ansible-playbook is already next step | 16:51 |
| sshnaidm|rover | rlandy, you can't just call ansible-playbook without setup environment with oooq | 16:52 |
| *** trown is now known as trown|lunch | 16:53 | |
| *** kopecmartin has quit IRC | 16:54 | |
| rlandy | sshnaidm|rover: so I would consider adding another separate requirements file but not only through quickstart | 16:54 |
| rlandy | sshnaidm|rover: I think this should be put to the team - esp considering the upcoming tempest work. Monday's meeting? | 16:56 |
| sshnaidm|rover | rlandy, yeah | 16:58 |
| *** agopi is now known as agopi|lunch | 16:59 | |
| rlandy | sshnaidm|rover: I +1'ed your patch in the mean time as I think it's a good idea in general | 16:59 |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 17:01 |
| rfolco | sshnaidm|rover, I thought we've agreed that the original script should not be modified by the j2 templates... rdo-cloud jobs won't follow the run-v3 workflow and use the original one ? https://review.openstack.org/#/c/581331/12/playbooks/tripleo-ci/run-v3.yaml | 17:03 |
| sshnaidm|rover | rfolco, well, that was an idea | 17:05 |
| rfolco | sshnaidm|rover, so let me put this way... do rdo-cloud jobs follow a different workflow (not run-v3) ? coz if they use the same run-v3, it will use the modified templates. | 17:09 |
| *** dtrainor has quit IRC | 17:10 | |
| *** dtrainor has joined #oooq | 17:14 | |
| sshnaidm|rover | rfolco, rdo cloud jobs are defined mostly here: https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/zuul-legacy-jobs.yaml and here https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/jobs.yaml | 17:15 |
| sshnaidm|rover | rfolco, as you see it needs to be configured separately.. until we know how to configure them in upstream repos | 17:16 |
| *** tesseract has quit IRC | 17:16 | |
| rfolco | sshnaidm|rover, ok so they do not follow run-v3, thats what I wanted to understand. Thanks | 17:17 |
| sshnaidm|rover | rfolco, yep | 17:17 |
| sshnaidm|rover | rfolco, but they use toci_* scripts in old way, that's what I warned about | 17:17 |
| rfolco | yep, we don't touch original toci_ scripts. Safe. | 17:18 |
| weshay | rlandy, nice https://review.rdoproject.org/r/#/c/14772/ | 17:19 |
| weshay | rlandy, bah.. but doesn't trigger the real job until merge | 17:20 |
| weshay | rdoproject-- | 17:20 |
| rlandy | weshay: we still need the playbook | 17:26 |
| weshay | k | 17:26 |
| rlandy | waiting for agopi|lunch to be back from lunch | 17:26 |
| rlandy | once we have that - we can try run something and see how long it takes | 17:27 |
| rlandy | weshay: hmmm ... probably should not have merged that - don't really want to add failing jobs :( | 17:30 |
| weshay | ? | 17:31 |
| weshay | rlandy, which? | 17:31 |
| weshay | the doc | 17:31 |
| *** sshnaidm|rover is now known as sshnaidm|afk | 17:31 | |
| weshay | rlandy, which one? | 17:32 |
| rlandy | no https://review.rdoproject.org/r/#/c/14772/ | 17:32 |
| rlandy | oh - maybe it didn't | 17:32 |
| weshay | meh | 17:32 |
| rlandy | ignore me | 17:32 |
| weshay | it's fine | 17:32 |
| rlandy | updating the tq review so we are ready to go there | 17:32 |
| *** sshnaidm|afk has quit IRC | 17:36 | |
| weshay | trown|lunch, so scen009 is non-containerized only? | 17:36 |
| *** holser_ has quit IRC | 17:36 | |
| rlandy | agopi|lunch: ping when you are back ... we need to chat about https://review.openstack.org/#/c/581488/1/playbooks/baremetal-quickstart-extras.yml and the playbook to add | 17:38 |
| rlandy | that is the missing piece | 17:38 |
| agopi|lunch | hey rlandy | 17:41 |
| agopi|lunch | sorry was out for lunch | 17:41 |
| *** agopi|lunch is now known as agopi | 17:41 | |
| rlandy | agopi: np | 17:51 |
| amoralej | hi, i'm confused about how are containers updated in oooq jobs | 17:51 |
| amoralej | when is container-check used? | 17:51 |
| rlandy | agopi: looks good ... https://gist.github.com/aakarshg/73b4a1a7a87b0867d816ce94bb83f76e | 17:51 |
| rlandy | how long will it run? | 17:51 |
| rlandy | agopi: we will need to enable this only when enable_minimal_browbeat: true | 17:52 |
| rlandy | per https://review.openstack.org/#/c/581484/3/config/general_config/featureset053.yml | 17:52 |
| rlandy | agopi: the current job time limit is 3 hrs | 17:55 |
| rlandy | we can extend if need be | 17:55 |
| agopi | hello rlandy im guessing it shouldnt take more than 30-50 mins just for the playbook i sent. | 17:55 |
| agopi | and also | 17:55 |
| weshay | amoralej, it's been moved into the undercloud install | 17:56 |
| weshay | amoralej, outside of ci | 17:56 |
| weshay | amoralej, it's also moving from overcloud-prep-containers --> tripleo-common | 17:56 |
| weshay | amoralej, the logic is still the same | 17:56 |
| weshay | of when it is used | 17:56 |
| rlandy | agopi: ok - let's try it and see how it goes | 17:56 |
| agopi | rlandy, how will we go about passing variables? | 17:57 |
| amoralej | weshay, so in master is done by https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/overcloud-prep-containers/templates/overcloud-prep-containers.sh.j2#L34-L37 ? | 17:57 |
| agopi | are we assuming same setup as previous jobs ? | 17:57 |
| rlandy | we'll have to do an include when | 17:57 |
| weshay | amoralej, ya | 17:57 |
| amoralej | weshay, so repo-name is always gating-repo https://github.com/openstack/tripleo-quickstart-extras/blob/1340ff513fc9e3f89cb0d1ec0ccde223e0b43ce3/roles/undercloud-deploy/templates/containers-prepare-parameter.yaml.j2#L17 | 17:58 |
| amoralej | right? | 17:58 |
| amoralej | it used to be configurable before using gating_repo_name | 17:58 |
| * rlandy thinking about where we should put the when: condition | 17:58 | |
| rlandy | agopi: I'll add then when | 17:59 |
| agopi | rlandy, | 18:00 |
| rlandy | updating review | 18:00 |
| agopi | https://github.com/openstack/browbeat/blob/master/browbeat-config.yaml | 18:00 |
| agopi | browbeat needs a config file like this | 18:00 |
| agopi | a minimal config file is here | 18:00 |
| agopi | https://github.com/openstack/browbeat/blob/master/ansible/oooq/roles/template-configs/templates/browbeat-basic.yaml.j2 | 18:00 |
| agopi | as you see we need browbeat_cloud_name, elastic_host, grafana_host | 18:01 |
| rlandy | agopi: https://github.com/openstack/tripleo-quickstart-extras/blob/master/config/environments/rdocloud.yml | 18:01 |
| rlandy | so we would need to edit that | 18:02 |
| rlandy | the values are stored above: ^^ | 18:02 |
| agopi | okay rlandy | 18:02 |
| *** trown|lunch is now known as trown | 18:03 | |
| rlandy | agopi: you'll need a grafana dashboard on our graphana, right? | 18:03 |
| trown | weshay: it is deploying openshift, so ya | 18:03 |
| rlandy | cloud_name: rdocloud | 18:03 |
| trown | weshay: it doesnt deploy openstack at all (other than undercloud) | 18:03 |
| rlandy | trown: hey - how's the new role? | 18:04 |
| trown | rlandy: so far so good :P... a ton to learn | 18:04 |
| agopi | we already have hosts for ELK including grafana | 18:04 |
| rlandy | https://github.com/openstack-infra/tripleo-ci/blob/master/toci-quickstart/config/testenv/ovb-rdocloud.yml | 18:04 |
| agopi | http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/jenkins/jobs/tripleo-quickstart/tripleo-quickstart-builders.yml#L203 | 18:05 |
| rlandy | agopi: ^^ sorry that has the grapaha values | 18:05 |
| rlandy | influxdb | 18:05 |
| * rlandy updates review | 18:05 | |
| agopi | sorry i dont follow rlandy | 18:05 |
| agopi | how are you passing the password? well need to do same for grafana_apikey | 18:06 |
| rlandy | agopi: hmm not sure - will need to ask/check with sshnaidm | 18:07 |
| rlandy | panda|off: rfolco: ^^ do you remember | 18:07 |
| rfolco | sorry don't recall handling keys for grafana | 18:09 |
| weshay | trown, ah k | 18:10 |
| rlandy | agopi: https://review.openstack.org/#/c/581488/ updated | 18:14 |
| rlandy | agopi: if you have the config file, we can add it | 18:15 |
| amoralej | may i get some eyes on https://review.openstack.org/582327 ? | 18:15 |
| amoralej | i'm not sure if it's the right way | 18:15 |
| trown | weshay: I am working on the other path... deploying openshift inside of an openstack tenant... but just getting an env up to work with is challenging ... that is pretty far off from a CI perspective though | 18:16 |
| trown | weshay: undercloud deployed openshift (scenario009) seems pretty much done | 18:16 |
| weshay | k | 18:17 |
| *** amoralej is now known as amoralej|off | 18:17 | |
| *** hamzy_ has quit IRC | 18:18 | |
| *** hamzy has joined #oooq | 18:20 | |
| *** vinaykns has quit IRC | 18:22 | |
| weshay | sagi, chkumar|ruck, rlandy fyi.. I have disabled osp jobs https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tq-gate-rhos-12-ci-rhos-ovb-minimal-pacemaker-public-bond/ | 18:23 |
| rlandy | weshay: ok | 18:23 |
| weshay | amoralej|off, https://review.openstack.org/#/q/topic:bp/container-prepare-workflow+(status:open+OR+status:merged) | 18:25 |
| rlandy | rfolco: I think we had doc on it ... checking ruck/rover | 18:26 |
| rfolco | rlandy, I remember I did not get access to grafana server | 18:27 |
| rlandy | was it rdo infra or ours? | 18:27 |
| rfolco | I believe it was rdo infra | 18:28 |
| weshay | trown, is that something I can poke at w/ the reproducer ? | 18:28 |
| weshay | scen009 | 18:28 |
| trown | weshay: ya, you may need https://review.openstack.org/#/c/582164/1 if 1.1.1.1 does not work for DNS for you... though that may be very specific to my home ISP | 18:29 |
| weshay | k | 18:29 |
| trown | weshay: but the above patch was all I needed to get it working via libvirt | 18:29 |
| trown | you can even run oc commands against the openshift and make apps and what not | 18:30 |
| trown | was a bit surprised it actually works given that part is not at all tested yet | 18:30 |
| *** ykarel|away has joined #oooq | 18:32 | |
| hubbot` | FAILING CHECK JOBS on stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades @ (1 more message) | 19:01 |
| *** ykarel|away has quit IRC | 19:17 | |
| rlandy | marios: still around? ping re: reparent failure | 19:17 |
| *** vinaykns has joined #oooq | 19:28 | |
| *** jaganathan has quit IRC | 19:38 | |
| rlandy | marios: panda|off: I think the error is unrelated to the change - rerunning | 19:56 |
| *** apevec has joined #oooq | 20:35 | |
| weshay | 0/ | 20:35 |
| apevec | weshay, here? | 20:35 |
| weshay | aye mate | 20:35 |
| weshay | <apevec> weshay, are network interface stats collected somewhere in oooq logs? | 20:36 |
| apevec | so, .*: are network interface stats collected somewhere in oooq logs? | 20:36 |
| weshay | <weshay> apevec, can you ask in #oooq | 20:36 |
| weshay | <apevec> weshay, I need some guesstimate how much network traffic CI jobs are generating | 20:36 |
| apevec | oh thanks for CPaaS :) | 20:36 |
| weshay | apevec, what ever we have there would be $node/var/log/extra | 20:36 |
| weshay | apevec, if you give me some sample commands we can get them into the logs for you | 20:36 |
| * weshay gets example | 20:37 | |
| weshay | apevec, http://logs.openstack.org/26/581026/3/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/7242759/logs/undercloud/var/log/extra/network.txt.gz | 20:38 |
| apevec | basic stats would be just e.g. ifconfig, that would show RX/TX packets | 20:38 |
| weshay | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/collect-logs/tasks/collect.yml#L47 | 20:38 |
| weshay | ya.. it's there | 20:38 |
| apevec | ah cool, /me looks | 20:39 |
| apevec | so assuming nodepool node is freshly created, we can assume all packets were generated by the CI job | 20:39 |
| apevec | ok, cool, so can do some mining, thanks! | 20:40 |
| weshay | np | 20:40 |
| *** holser_ has joined #oooq | 20:40 | |
| apevec | weshay, br-ex would be traffic to/from Internet? | 20:51 |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci- (1 more message) | 21:01 |
| weshay | apevec, to the internet .. no | 21:02 |
| weshay | eth0 | 21:02 |
| *** florianf has quit IRC | 21:03 | |
| weshay | 192.168.24.x is the public connection between the undercoud and overcloud | 21:04 |
| *** trown is now known as trown|outtypewww | 21:04 | |
| *** holser_ has quit IRC | 21:18 | |
| *** rlandy is now known as rlandy|afk | 21:55 | |
| *** sshnaidm|afk has joined #oooq | 21:57 | |
| *** apevec has quit IRC | 21:57 | |
| *** florianf has joined #oooq | 22:17 | |
| *** florianf has quit IRC | 22:43 | |
| hubbot` | FAILING CHECK JOBS on stable/queens: tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario008-multinode-oooq-container @ https://review.openstack.org/567224, stable/ocata: legacy-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-ocata @ https://review.openstack.org/564291, master: legacy-tripleo-ci-centos-7-container-to-container-upgrades-master, tripleo-ci- (1 more message) | 23:01 |
| *** tosky has quit IRC | 23:01 | |
| *** rlandy|afk is now known as rlandy | 23:18 | |
| rlandy | original message: could not locate file in lookup: ControllerApi/centos-7-inap-mtl01-0000693370/UpgradeInitDeployment" | 23:23 |
| rlandy | ^^ consistent | 23:23 |
| rlandy | real issue with 3 node | 23:23 |
| *** vinaykns has quit IRC | 23:27 | |
| *** vinaykns has joined #oooq | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!