| *** mjturek has quit IRC | 00:06 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, (3 more messages) | 00:27 |
|---|---|---|
| *** dsneddon has quit IRC | 00:40 | |
| *** dsneddon has joined #oooq | 00:57 | |
| *** jbadiapa has quit IRC | 01:23 | |
| *** jbadiapa has joined #oooq | 01:23 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, (3 more messages) | 02:27 |
| *** ykarel|away has joined #oooq | 02:52 | |
| *** apetrich has quit IRC | 03:15 | |
| *** ykarel|away has quit IRC | 03:32 | |
| *** dsneddon has quit IRC | 03:35 | |
| *** dsneddon has joined #oooq | 03:37 | |
| *** ykarel|away has joined #oooq | 03:46 | |
| *** udesale has joined #oooq | 03:46 | |
| *** ykarel|away is now known as ykarel | 03:57 | |
| *** raukadah is now known as chandankumar | 04:02 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci- (3 more messages) | 04:27 |
| *** udesale has quit IRC | 04:44 | |
| *** rh-jelabarre has quit IRC | 04:46 | |
| *** udesale has joined #oooq | 04:46 | |
| *** saneax has joined #oooq | 04:49 | |
| *** ratailor has joined #oooq | 05:41 | |
| *** skramaja has joined #oooq | 06:21 | |
| *** jaosorior has quit IRC | 06:26 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci- (3 more messages) | 06:27 |
| *** jaosorior has joined #oooq | 06:28 | |
| *** quiquell|off is now known as quiquell | 06:31 | |
| *** apetrich has joined #oooq | 06:52 | |
| quiquell | marios: o/ | 06:56 |
| marios_|ruck | quiquell: \o | 06:57 |
| quiquell | marios_|ruck: did it work the override-checkout ? | 06:59 |
| marios_|ruck | quiquell: stuff merged 3 hours ago https://review.openstack.org/#/c/644912/ | 07:00 |
| marios_|ruck | quiquell: ah | 07:00 |
| marios_|ruck | quiquell: no | 07:00 |
| marios_|ruck | quiquell: i mean we ddin't use it i abandoned that | 07:00 |
| marios_|ruck | quiquell: sshnaidm|rover fixd it so we get proper delorean repo with https://review.openstack.org/#/c/645562/ | 07:01 |
| quiquell | ahh we have the --exclude option now | 07:01 |
| marios_|ruck | quiquell: no no exclude | 07:01 |
| marios_|ruck | quiquell: we have image source stuff, so it will exclude grafana as its openshift in https://review.openstack.org/#/c/644912/1/tripleo_common/image/kolla_builder.py | 07:01 |
| marios_|ruck | quiquell: for the branch calculation, sshnaidm|rover added that here https://review.openstack.org/#/c/645562/5/roles/tripleo-repos/tasks/main.yml so when we call tripleo-repos we'll get stable/rocky tripleo-ci-testing delorean | 07:02 |
| marios_|ruck | quiquell: so should be good | 07:02 |
| marios_|ruck | quiquell: well, we find out today :D | 07:02 |
| marios_|ruck | quiquell: for --excludes "we'll see" | 07:03 |
| marios_|ruck | quiquell: no run yet today so next periodic run we'll see what happens there https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-7-rocky-containers-build-push | 07:05 |
| quiquell | marios_|ruck: ok, so tripleo-repos are now branch_override aware, ack | 07:05 |
| quiquell | marios_|ruck: will check jobs | 07:06 |
| ykarel | marios_|ruck, more work is needed, i commented on your abandoned patch | 07:06 |
| ykarel | can you check | 07:06 |
| quiquell | ykarel: what patch ? | 07:06 |
| quiquell | marios_|ruck: do you have super powers here https://review.rdoproject.org/r/#/c/19021/ ? | 07:07 |
| ykarel | marios_|ruck, quiquell https://review.rdoproject.org/r/#/c/19746/ | 07:07 |
| quiquell | ykarel: sshnaidm|rover bypassed it with https://review.openstack.org/#/c/645562/ | 07:08 |
| ykarel | quiquell, no ^^ is not enough | 07:09 |
| quiquell | ykarel: do we still need it ? | 07:09 |
| ykarel | as tripleo common is used from source | 07:09 |
| ykarel | in that job | 07:09 |
| ykarel | see kolla-build.conf | 07:09 |
| quiquell | ykarel: I see | 07:09 |
| ykarel | so u need to rely on zuul | 07:09 |
| quiquell | marios_|ruck: ^ | 07:09 |
| ykarel | i must have checked the testing review yesterday, i missed those | 07:10 |
| quiquell | ykarel: you are talking about this ? template_override={{ openstack_git_root }}/tripleo-common/container-images/tripleo_kolla_template_overrides.j2 | 07:11 |
| marios_|ruck | ykarel: o/ ack so you think we may need that still https://review.rdoproject.org/r/#/c/19746/ the override checkout? | 07:11 |
| ykarel | quiquell, yes | 07:11 |
| marios_|ruck | ykarel: do we have logs yet? | 07:11 |
| marios_|ruck | ykarel: i mean the c herrypick at 09:00 < marios_|ruck> quiquell: stuff merged 3 hours ago https://review.openstack.org/#/c/644912/ | 07:12 |
| marios_|ruck | ykarel: so i was waiting for next run | 07:12 |
| quiquell | marios_|ruck: looks like we need it, tripleo_kolla_template_overrides.j2 is taking from git clone not from RPMs | 07:12 |
| ykarel | marios_|ruck, master is checked out, we need to checkout stable/rocky | 07:12 |
| ykarel | see inventory.yml | 07:12 |
| ykarel | and kolla-build.conf | 07:12 |
| ykarel | ^^ will make things clear | 07:12 |
| marios_|ruck | quiquell: ykarel thanks (yeah i saw ykarel pointer). ok cool. well we need both then i mean rocky tripleo-ci-testing delorean repo for packages in https://review.openstack.org/#/c/645562/ and then also override for zuul reequired projects | 07:13 |
| marios_|ruck | sec restoring | 07:13 |
| ykarel | yes exactly | 07:13 |
| marios_|ruck | ykarel: quiquell so sec, sshnaidm|rover said he might post something to set those but upstream lemme see if he did | 07:14 |
| quiquell | marios_|ruck: well, is better to just checkout rocky | 07:14 |
| marios_|ruck | https://review.openstack.org/#/c/647719/ maybe here sshnaidm|rover ? cc ykarel quiquell see https://review.openstack.org/#/c/647719/4/zuul.d/build-containers.yaml | 07:15 |
| quiquell | ykarel: do you know when are we going to have stable/stein branch at projects ? | 07:15 |
| ykarel | quiquell, you mean tripleo projects? | 07:16 |
| ykarel | non tripleo already cut stable/stein | 07:16 |
| *** jfrancoa has joined #oooq | 07:16 | |
| quiquell | ykarel: yep | 07:16 |
| quiquell | marios_|ruck: that looks like a testing review to play with override-checkout | 07:17 |
| ykarel | quiquell, possibly next week: Apr 01 - Apr 05 R-1 | 07:17 |
| ykarel | Final RCs and intermediary releases | 07:17 |
| quiquell | ykarel: so master will be train ? | 07:17 |
| ykarel | quiquell, as per schedule ^^, but exact dates jaosorior emilien might know | 07:18 |
| quiquell | ykarel: ack thanks! | 07:18 |
| quiquell | marios_|ruck: btw did we take any decision with the zuul reproducer or something ? | 07:19 |
| ykarel | marios_|ruck, that job is different, there is override-checkout, so we need same in rdo jobs | 07:20 |
| ykarel | not sure if there is plan to run branchfull job upstream | 07:20 |
| marios_|ruck | ykarel: ack i restored it so lets merge it then please thanks | 07:22 |
| marios_|ruck | quiquell: like what kind of decision you mean | 07:22 |
| ykarel | marios_|ruck, +A | 07:23 |
| quiquell | ykarel: can you +A this https://review.rdoproject.org/r/#/c/19021/ ? | 07:24 |
| ykarel | marios_|ruck, override-checkout patch merged, can you fire a test job(rocky-container-build-push) in testproject master branch | 07:26 |
| marios_|ruck | ykarel: in master? why not rocky | 07:26 |
| ykarel | marios_|ruck, we need to same as done in periodic pipeline | 07:27 |
| marios_|ruck | ykarel: but we won't get the rocky exceptions | 07:27 |
| ykarel | exceptions? | 07:27 |
| marios_|ruck | ykarel: like that stuff https://review.openstack.org/#/c/645562/5/playbooks/tripleo-buildcontainers/templates/build.sh.j2 and https://review.openstack.org/#/c/645562/5/playbooks/tripleo-buildcontainers/run.yaml | 07:28 |
| marios_|ruck | ykarel: ok anyway doing master sec (we'll see rocy in a bit when periodic runs ) | 07:28 |
| quiquell | marios_|ruck: to emulate pariodic running testproject at master is the way | 07:28 |
| marios_|ruck | quiquell: kk | 07:28 |
| quiquell | marios_|ruck: periodic runs as master, if some of the code there is not running right it have to be fixed | 07:28 |
| ykarel | marios_|ruck, periodic already ran | 07:29 |
| ykarel | for rocky | 07:29 |
| ykarel | and failed | 07:29 |
| *** ykarel is now known as ykarel|lunch | 07:29 | |
| marios_|ruck | ykarel|lunch: oh i didn't see it in zuul builds earlier | 07:31 |
| ykarel|lunch | zuul builds are reported after periodic pipeline finishes, currently master is still running | 07:32 |
| ykarel|lunch | for current run you can see https://review.rdoproject.org/zuul/status | 07:32 |
| * ykarel|lunch leaving | 07:32 | |
| marios_|ruck | ykarel|lunch: thx enjoy lunch | 07:32 |
| quiquell | biab | 07:33 |
| *** quiquell is now known as quiquell|brb | 07:33 | |
| marios_|ruck | quiquell|brb: ykarel|lunch https://review.rdoproject.org/r/19768 | 07:36 |
| quiquell|brb | have to be run at master branch | 07:38 |
| quiquell|brb | to emulate periodics | 07:38 |
| quiquell|brb | so we see how override-checkout works | 07:39 |
| *** matbu has quit IRC | 07:39 | |
| marios_|ruck | quiquell|brb: its master test project but https://review.rdoproject.org/r/#/c/19768/1/.gitreview rock there i thought is what we wanted? | 07:41 |
| marios_|ruck | quiquell|brb: so even there master? ^ | 07:41 |
| marios_|ruck | quiquell|brb: cos then we run rocky job for master branch? | 07:41 |
| quiquell|brb | marios_|ruck: you have checkout stable/rocky brach in the review | 07:43 |
| quiquell|brb | marios_|ruck: I see "Branch stable/rocky" | 07:43 |
| quiquell|brb | have to be master | 07:43 |
| quiquell|brb | do a checkout on master and cherry pick it | 07:43 |
| marios_|ruck | quiquell|brb: k | 07:43 |
| quiquell|brb | at testproject | 07:43 |
| marios_|ruck | (updated) | 07:43 |
| quiquell|brb | so we verify that override-checkout is doing the right thing | 07:43 |
| *** kopecmartin|off is now known as kopecmartin | 07:43 | |
| quiquell|brb | ack, now it's | 07:44 |
| marios_|ruck | quiquell|brb: but now why is it stable/rocky | 07:44 |
| marios_|ruck | damn it | 07:44 |
| quiquell|brb | nope still rocky | 07:44 |
| marios_|ruck | quiquell|brb: sec | 07:44 |
| quiquell|brb | ok have to leave now | 07:44 |
| marios_|ruck | quiquell|brb: ack ttyl thanks | 07:44 |
| marios_|ruck | https://review.rdoproject.org/r/#/c/19769/ quiquell|brb ykarel|lunch | 07:46 |
| *** amoralej|off is now known as amoralej | 07:58 | |
| *** holser_ has joined #oooq | 08:05 | |
| *** quiquell|brb is now known as quiquell | 08:08 | |
| *** tosky has joined #oooq | 08:17 | |
| sshnaidm|rover | marios_|ruck, o/ | 08:23 |
| *** ykarel|lunch is now known as ykarel | 08:25 | |
| marios_|ruck | sshnaidm|rover: \o | 08:26 |
| quiquell | sshnaidm|rover: I am going to add a task about prometheus here https://tree.taiga.io/project/tripleo-ci-board/us/849?milestone=223049 | 08:26 |
| quiquell | sshnaidm|rover: And going to start with it, ack ? | 08:26 |
| sshnaidm|rover | quiquell, let's discuss it before | 08:27 |
| quiquell | sshnaidm|rover: blue ? | 08:27 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci- (3 more messages) | 08:27 |
| sshnaidm|rover | quiquell, yeah, in 10 min | 08:27 |
| marios_|ruck | sshnaidm|rover: did you check the grafana error with the promotion reporting yesterday? | 08:27 |
| marios_|ruck | sshnaidm|rover: can i do something there | 08:27 |
| sshnaidm|rover | marios_|ruck, oh, not yet, will look now | 08:27 |
| quiquell | ack | 08:27 |
| marios_|ruck | sshnaidm|rover: thanks | 08:28 |
| ykarel | marios_|ruck, ack | 08:29 |
| *** dtantsur|afk is now known as dtantsur | 08:32 | |
| ykarel | amoralej, can you look https://review.rdoproject.org/r/#/c/19744/ | 08:35 |
| ykarel | openstack/requirements is branched | 08:35 |
| amoralej | ok | 08:36 |
| amoralej | let me check | 08:36 |
| marios_|ruck | sshnaidm|rover: ah i think grafana is ok its the time filter | 08:37 |
| amoralej | ykarel, we need to coordinate that with the update of dlrn configs | 08:37 |
| amoralej | did you talk to jpena? | 08:37 |
| ykarel | amoralej, yes, jpena pushed patches | 08:37 |
| sshnaidm|rover | marios_|ruck, yeah, I see it's reporting master etc | 08:37 |
| marios_|ruck | sshnaidm|rover: i was confused cos i saw wes note on the etherpad but when i change the time to last 7 days for example it shows. i guess maybe the default time range changed? | 08:37 |
| amoralej | ok | 08:37 |
| marios_|ruck | sshnaidm|rover: k thanks | 08:37 |
| sshnaidm|rover | marios_|ruck, maybe yesterdays restart helped | 08:38 |
| marios_|ruck | sshnaidm|rover: k | 08:38 |
| *** ccamacho has joined #oooq | 08:38 | |
| marios_|ruck | sshnaidm|rover: so here it says master promoted 23rd https://dashboards.rdoproject.org/master but grafana says 21st | 08:41 |
| marios_|ruck | sshnaidm|rover: and phase 1 master on 23rd at ^^ but not in grafana | 08:42 |
| sshnaidm|rover | marios_|ruck, k, will look | 08:42 |
| marios_|ruck | sshnaidm|rover: thx | 08:42 |
| sshnaidm|rover | marios_|ruck, can you please create a bug about 030 failing in master promotions? http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-master/6f15af3/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 08:43 |
| marios_|ruck | sshnaidm|rover: ack looking | 08:43 |
| sshnaidm|rover | marios_|ruck, we need to look at master failures | 08:44 |
| marios_|ruck | sshnaidm|rover: added in etherpad | 08:51 |
| *** bogdando has joined #oooq | 08:53 | |
| quiquell | sshnaidm|rover: just tested the telegraf output for prometheus and it works out of the box | 08:55 |
| quiquell | sshnaidm|rover: without chaing the recollecting scripts :-) | 08:56 |
| sshnaidm|rover | quiquell, can't we execute scripts just from prometheus? | 08:56 |
| *** jpena|off is now known as jpena | 08:56 | |
| quiquell | sshnaidm|rover: nope | 08:56 |
| quiquell | sshnaidm|rover: it just listten to http stuff | 08:56 |
| quiquell | sshnaidm|rover: It does not start scripts, there is something like exporters but you have to run them yourself | 08:57 |
| quiquell | sshnaidm|rover: telegraf + prometheus is good for us | 08:57 |
| sshnaidm|rover | quiquell, blue? | 09:02 |
| quiquell | yep | 09:03 |
| marios_|ruck | errand biab | 09:18 |
| *** d0ugal has quit IRC | 09:18 | |
| quiquell | sshnaidm|rover: https://review.rdoproject.org/r/#/c/19776/ | 09:36 |
| quiquell | This is it | 09:36 |
| ykarel | marios_|ruck, are the CIX care created automatically or manually? | 09:36 |
| ykarel | actually asking because seeing new Labels | 09:36 |
| *** derekh has joined #oooq | 09:39 | |
| quiquell | zbr|ssbarnea: ping | 09:39 |
| zbr|ssbarnea | quiquell: hi | 09:40 |
| quiquell | zbr|ssbarnea: The stein release file is at workable state ? | 09:40 |
| zbr|ssbarnea | quiquell: yes, please help merging it https://review.openstack.org/#/c/643963/ | 09:40 |
| zbr|ssbarnea | is cannot progress without merging this first. | 09:40 |
| quiquell | zbr|ssbarnea: we have to wait for stable/stain branch to test it I supose | 09:41 |
| zbr|ssbarnea | quiquell: no need to wait. | 09:42 |
| quiquell | zbr|ssbarnea: or we can just merge to have it and fix next week | 09:42 |
| zbr|ssbarnea | quiquell: this is what i was trying to say for the last week, merge now so we can test periodic ones, and fix eventual issues. | 09:43 |
| zbr|ssbarnea | without these, we would never be able to push stein containers which are key for any deployment. we need it for bootstrapping stein | 09:44 |
| quiquell | zbr|ssbarnea: have some comments https://review.openstack.org/#/c/643963 at master.yaml | 09:44 |
| zbr|ssbarnea | quiquell: addressed, look again. | 09:51 |
| zbr|ssbarnea | @oooq: on f28 we need to be bure we install dnf-plugins-core rpm or we may get error about missing config-manager. make no assumption. | 09:53 |
| quiquell | Why you don't remove {# dnf-plugins-core provides config-manager command #} ? | 09:53 |
| quiquell | Still this things are not related to stein | 09:54 |
| quiquell | Add a new review for them | 09:54 |
| *** matbu has joined #oooq | 09:54 | |
| zbr|ssbarnea | because each extra patch adds one extra day delay (if lucky) | 09:55 |
| zbr|ssbarnea | stein is first f28 release, so it makes sense to be the same (to me). | 09:56 |
| quiquell | acke let's just merge it | 09:58 |
| quiquell | marios_|ruck, sshnaidm|rover: Can we just merge the stein release file ? https://review.openstack.org/#/c/643963 | 09:58 |
| sshnaidm|rover | quiquell, looking | 09:58 |
| sshnaidm|rover | quiquell, does ceph should be luminous? | 09:59 |
| sshnaidm|rover | fultonj, ^^ | 09:59 |
| zbr|ssbarnea | quiquell: also remember that stein is the only f28 release. so even if I touch the master file, is only for stein. | 10:00 |
| zbr|ssbarnea | fedora28 will be removed in from master when we have centos-8 | 10:01 |
| *** jtomasek has joined #oooq | 10:01 | |
| zbr|ssbarnea | but i will create a separate change, to make people happy. | 10:01 |
| quiquell | zbr|ssbarnea: f28 master will be train | 10:01 |
| quiquell | zbr|ssbarnea: But I see is good that stein/master have same things | 10:02 |
| quiquell | zbr|ssbarnea: So I am all good there | 10:02 |
| zbr|ssbarnea | quiquell: that is what i was trying to explain, is needed for both. anyway, I spoke with wes and we both want to refactor the release files so we remove some of the duplication between them. too much identical stuff is spread in multiple files. | 10:03 |
| zbr|ssbarnea | it makes very easy to fix a bug in only one place and having to redo it for the other cases ;) | 10:04 |
| quiquell | zbr|ssbarnea: yep, nah let's merge this and fix I think | 10:04 |
| quiquell | zbr|ssbarnea: well will add complexity too as always | 10:04 |
| sshnaidm|rover | zbr|ssbarnea, https://review.openstack.org/#/c/643963 commented | 10:09 |
| marios_|ruck | quiquell: looking | 10:09 |
| marios_|ruck | ykarel: i created thos manually | 10:10 |
| marios_|ruck | ykarel: i mean i added 2 today for the blockers | 10:10 |
| zbr|ssbarnea | sshnaidm|rover: debatable, comment is to prevent accidental removal from code, not from the output of the templated file. | 10:10 |
| ykarel | marios_|ruck, ack, and the labels there were added intentionally? | 10:10 |
| zbr|ssbarnea | sshnaidm|rover: not sure what is the value of getting a comment in a final file, the value is at source. | 10:10 |
| sshnaidm|rover | zbr|ssbarnea, not critical, I'm worried more about comments in file | 10:11 |
| zbr|ssbarnea | sshnaidm|rover: but i am not strong about it, i can easily make it a bash commend instead of jinja. | 10:11 |
| marios_|ruck | ykarel: yeah please fix if they are wrong or tell me and i'll fix | 10:11 |
| ykarel | marios_|ruck, actually as per bug, these are TripleO CI | 10:11 |
| ykarel | so not COmponent CI, RDO label | 10:12 |
| ykarel | so TripleO-stein label | 10:12 |
| marios_|ruck | ykarel: ack thanks fixing | 10:12 |
| ykarel | marios_|ruck, if it start failing apart from TripleO stein jobs, u can add others, for now as per bug these affecting just TripleO stein | 10:13 |
| marios_|ruck | ykarel: ack thanks | 10:13 |
| zbr|ssbarnea | sshnaidm|rover: now i see, 2nd set of comments. mainly they were copied from master. | 10:15 |
| quiquell | zbr|ssbarnea: found typo | 10:15 |
| zbr|ssbarnea | add now, as I am fixing the comments noted by sshnaidm|rover | 10:15 |
| *** dsneddon has quit IRC | 10:16 | |
| quiquell | zbr|ssbarnea: nah false alarm was at my code of the mitter :-) | 10:17 |
| quiquell | emitter | 10:17 |
| quiquell | zbr|ssbarnea: yep little typo at the stein release file https://review.openstack.org/#/c/643963 | 10:19 |
| quiquell | extra s there | 10:19 |
| quiquell | My standalone-upgrade job with stein emitter was failing is testing it | 10:19 |
| zbr|ssbarnea | quiquell: not minor. thanks | 10:20 |
| quiquell | zbr|ssbarnea: let me rerun the emitter review so we exercise the release file | 10:25 |
| zbr|ssbarnea | am i wrong to wonder if we need new dlrn urls now? | 10:25 |
| quiquell | the use release | 10:25 |
| quiquell | so they are different from master already | 10:25 |
| zbr|ssbarnea | i am looking at the values of rdo_dlrn and tripleo_dlrn and I see that for fedora we do not use os version | 10:26 |
| chandankumar | sshnaidm|rover: I am renaming the project from tripleo-collect-logs to ansible-role-collect-logs as it is going to be consumed by other openstack project as suggested by alex | 10:27 |
| sshnaidm|rover | chandankumar, cool | 10:27 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci- (3 more messages) | 10:27 |
| zbr|ssbarnea | even if I compare master c7 with f28 there are some naming discrepancies which worry me a bit, especially as we introduce stein now. | 10:27 |
| marios_|ruck | sshnaidm|rover: did you manage tolook at grafana (sorry for harras i am preparing the program call status). esp that i see phase 1 at https://dashboards.rdoproject.org/master but not grafana (from 23rd promotion) | 10:32 |
| zbr|ssbarnea | who can hint me about the purpose of docker_openshift_tag, seems to differ between c7/f28 | 10:34 |
| zbr|ssbarnea | rfolco: sshnaidm|rover : please look at https://review.openstack.org/#/c/648070/1 -- i started to split the change to make it easy to review. better if you can also diff the two files. | 10:36 |
| quiquell | zbr|ssbarnea: missing yatin's comments on latest ps aka s/fedora/fedora28/ | 10:39 |
| zbr|ssbarnea | quiquell: wow, glad to see that we have consistent urls now, https://trunk.rdoproject.org/fedora28-master/ :D updating it.... | 10:40 |
| zbr|ssbarnea | i am going to add a variable that would expand to centos7 or fedora28, any names? distro_ver sounds ok? | 10:43 |
| quiquell | zbr|ssbarnea: don't do the refactor there | 10:44 |
| quiquell | let's just merge the stein release file | 10:44 |
| quiquell | then we do the refactor | 10:44 |
| zbr|ssbarnea | quiquell: i will refactor few things before, to avoid extra work (not in the stein change) | 10:44 |
| *** holser_ is now known as holser|lunch | 10:47 | |
| zbr|ssbarnea | quiquell: sshnaidm|rover ykarel : please check https://review.openstack.org/#/c/648070/2 -- this should ease our stein branching. | 10:54 |
| marios_|ruck | zbr|ssbarnea: did you miss ykarel comments https://review.openstack.org/#/c/643963/11 | 10:55 |
| zbr|ssbarnea | marios_|ruck: not really, look at my review, i am doing the refactoring first, before stein. i will rebase stein change on top of refactoring. | 10:56 |
| quiquell | zbr|ssbarnea: then is going to take ages to merge man | 10:57 |
| zbr|ssbarnea | yeah. but we need to fix it before. it will be faster this way. | 10:57 |
| quiquell | zbr|ssbarnea: now we have to test that all jobs are working fine | 10:58 |
| quiquell | zbr|ssbarnea: not just new rlease for for stein | 10:58 |
| marios_|ruck | zbr|ssbarnea: ack | 11:03 |
| *** zbr|ssbarnea has quit IRC | 11:06 | |
| *** zbr has joined #oooq | 11:07 | |
| sshnaidm|rover | arxcruz, kopecmartin chandankumar any ideas about https://bugs.launchpad.net/tripleo/+bug/1821690 ? Should we exclude this test until it's fixed? | 11:16 |
| openstack | Launchpad bug 1821690 in tripleo "periodic-ovb-1ctlr_1comp-featureset020-master fails tempest test_auto_allocate_network tempest.api.compute.images.test_images" [Undecided,Triaged] - Assigned to Marios Andreou (marios-b) | 11:16 |
| arxcruz | sshnaidm|rover: yes, defintively skip for now | 11:17 |
| arxcruz | sshnaidm|rover: i'm still investigating it | 11:17 |
| sshnaidm|rover | arxcruz, can you add a patch please? | 11:17 |
| arxcruz | maybe the problem is with the micro version | 11:17 |
| arxcruz | sshnaidm|rover: sure | 11:17 |
| sshnaidm|rover | thanks | 11:17 |
| *** jtomasek has quit IRC | 11:18 | |
| sshnaidm|rover | marios_|ruck, we so close to master promotion, but 030 is failing :( | 11:18 |
| chandankumar | sshnaidm|rover: https://review.openstack.org/#/q/topic:add-ansible-role-collect-logs+(status:open+OR+status:merged) done | 11:18 |
| marios_|ruck | sshnaidm|rover: thats the new bug from today right. yeah i mean we need to fix but we are kinda yellow for program call since we at least had a promotion in last 7 days | 11:19 |
| arxcruz | chandankumar: why raukadah ? | 11:19 |
| ykarel | sshnaidm|rover, may be can promote by skipping fs020 result and promote hash from yesterdays, as only 1 test failing | 11:19 |
| marios_|ruck | sshnaidm|rover: so we'll fix that by next couple days i expect the bug i mean ... would love to get rocky working too :D | 11:19 |
| ykarel | marios_|ruck, i think there is a fix for fs030, i think i saw a related patch | 11:20 |
| sshnaidm|rover | marios_|ruck, I hope to have fix for 030 today already, called for people.. | 11:20 |
| ykarel | in tht | 11:20 |
| marios_|ruck | sshnaidm|rover: ack thanks | 11:20 |
| sshnaidm|rover | ykarel, oh, is it there? Seems like without bug mention | 11:21 |
| ykarel | yes | 11:21 |
| ykarel | let me find link | 11:21 |
| ykarel | sshnaidm|rover, possibly https://review.openstack.org/#/c/647738/ | 11:22 |
| ykarel | good to get confirmation from hjensas or ramishra | 11:22 |
| ykarel | if ^^ is the fix or not | 11:22 |
| chandankumar | arxcruz: /me is doing some testing | 11:22 |
| arxcruz | chandankumar: ? | 11:23 |
| arxcruz | sorry, ykarel https://review.openstack.org/#/c/647738/ fix https://bugs.launchpad.net/tripleo/+bug/1821690 ? | 11:24 |
| openstack | Launchpad bug 1821690 in tripleo "periodic-ovb-1ctlr_1comp-featureset020-master fails tempest test_auto_allocate_network tempest.api.compute.images.test_images" [Undecided,Triaged] - Assigned to Marios Andreou (marios-b) | 11:24 |
| ykarel | arxcruz, not ^^, that's related to fs030 | 11:24 |
| zbr | sshnaidm|rover: thanks! also do you happen to know how this bash construct is supposed to work? https://github.com/openstack/tripleo-quickstart/blob/4e9c605600711b89029168bca35f34e2f542f941/config/release/tripleo-ci/Fedora-28/master.yml#L51 | 11:28 |
| zbr | i never used this kind of explansion, and apparently it does render as the first variable ( fedora_stable value) | 11:29 |
| zbr | which is always defined, so not sure what is the purpose, looks more of a mistake to me | 11:29 |
| *** dsneddon has joined #oooq | 11:29 | |
| quiquell | zbr: +2 there | 11:33 |
| quiquell | zbr: let's wait for CI and we workflow it | 11:33 |
| *** ratailor has quit IRC | 11:37 | |
| quiquell | zbr: are we missing creating containers for stein ? | 11:39 |
| quiquell | zbr: same as https://tree.taiga.io/project/tripleo-ci-board/task/887?kanban-status=1538166 but for containers | 11:39 |
| sshnaidm|rover | zbr, supposed to work, usually, it substitutes https://trunk.rdoproject.org with $NODEPOOL_RDO_PROXY | 11:42 |
| arxcruz | ykarel: ack | 11:44 |
| chandankumar | arxcruz: just looking a new/unique nick name so created one! | 11:46 |
| chandankumar | *for a | 11:46 |
| arxcruz | chandankumar: :) | 11:48 |
| arxcruz | sshnaidm|rover: https://review.openstack.org/#/c/648099/ | 11:48 |
| sshnaidm|rover | arxcruz, thanks | 11:49 |
| fultonj | sshnaidm|rover: in stein ansible whould be 2.7 and ceph should be nautilus; we're trying to get that all working together as per https://review.rdoproject.org/r/#/c/18721/ | 11:52 |
| sshnaidm|rover | zbr, ^^ | 11:53 |
| chandankumar | arxcruz: I am coverting one of the scenario1 to os_tempest if it is ok? to see how much it break? | 11:54 |
| fultonj | if you want to support python3 + ansible2.7 + podman + ceph, then the only supported ceph is nautilus with ceph-ansible 4.x | 11:54 |
| fultonj | zbr: ^ | 11:54 |
| arxcruz | chandankumar: did you see the tasks i create ? | 11:55 |
| fultonj | luminous --> ansible 2.6 + ceph-ansible3 + python2 + cent + docker | 11:55 |
| fultonj | was mentioned yesterday at tripleo meeting | 11:56 |
| chandankumar | arxcruz: so many tasks, lookin each of them | 11:56 |
| chandankumar | arxcruz: thanks for doing that | 11:56 |
| arxcruz | chandankumar: i believe we are going to need more, sorry, with all tempest stuff happening last week, i haven't time to do it earlier | 11:58 |
| fultonj | fmount: ^ fyi | 11:59 |
| chandankumar | arxcruz: yes, for tempestconf profile part I think I got the idea, I donot to change anything on os_tempest side may be https://stackoverflow.com/questions/25129728/ansible-override-single-dictionary-key | 11:59 |
| chandankumar | this will do the magic | 11:59 |
| *** jtomasek has joined #oooq | 12:00 | |
| fmount | fultonj: ack | 12:00 |
| quiquell | sshnaidm|rover, marios_|ruck: To get changes at baremetal jobs https://review.openstack.org/#/c/647580/ | 12:00 |
| arxcruz | chandankumar: cool :) | 12:01 |
| *** quiquell is now known as quiquell|lunch | 12:01 | |
| *** jtomasek has quit IRC | 12:05 | |
| sshnaidm|rover | quiquell|lunch, "openstack/tripleo-ci-internal-jobs" ? | 12:07 |
| sshnaidm|rover | quiquell|lunch, where does it exist? | 12:07 |
| quiquell|lunch | sshnaidm|rover: internal gerrit | 12:11 |
| quiquell|lunch | code.engineering... | 12:11 |
| *** jtomasek has joined #oooq | 12:12 | |
| *** rlandy has joined #oooq | 12:13 | |
| *** jtomasek has quit IRC | 12:13 | |
| *** trown|outtypewww is now known as trown | 12:17 | |
| sshnaidm|rover | quiquell|lunch, and it's in openstack/ namespace? | 12:18 |
| marios_|ruck | quiquell|lunch: call will check after | 12:18 |
| quiquell|lunch | sshnaidm|rover: yep | 12:18 |
| sshnaidm|rover | quiquell|lunch, ok | 12:19 |
| *** jtomasek has joined #oooq | 12:25 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, (3 more messages) | 12:27 |
| zbr | ykarel: thanks for spotting the diff deps vs base-stable: here is the updated version https://review.openstack.org/#/c/648105/ | 12:28 |
| ykarel | zbr, ack | 12:29 |
| zbr | ykarel: i am trying to consolidate configs before adding stein ones, to avoid extra work after. | 12:30 |
| ykarel | zbr, ack, but remember to check those changes are tested in some job to avoid breakages | 12:31 |
| kopecmartin | sshnaidm|rover, arxcruz the tests was skipped even before it was failing, we just didn't now it was failing because tempest.conf wasn't configured to run that test .. so skipping is fine, at least now | 12:36 |
| rlandy | rfolco: ping re: https://tree.taiga.io/project/tripleo-ci-board/task/888?kanban-status=1447275 | 12:42 |
| rfolco | o/ rlandy | 12:43 |
| rlandy | rfolco: hi - the link you ask to review points to standalone doc. | 12:43 |
| rfolco | rlandy, really ? | 12:43 |
| rfolco | rlandy, please ignore and click on the attachment | 12:43 |
| rlandy | rfolco: ok - there are two attachments | 12:44 |
| rfolco | oh dammn | 12:44 |
| rlandy | one for bm and one for standalone | 12:44 |
| rfolco | wrong file | 12:44 |
| rfolco | I am sorry | 12:44 |
| rfolco | fizing | 12:44 |
| rfolco | fixing | 12:44 |
| rlandy | I see the b, png | 12:45 |
| *** jpena is now known as jpena|lunch | 12:46 | |
| rfolco | rlandy, please check again | 12:46 |
| rfolco | I did not hide any ips, just replaced mac addr with FF:FF... | 12:47 |
| *** holser|lunch is now known as holser_ | 12:48 | |
| *** udesale has quit IRC | 12:48 | |
| rlandy | rfolco: ok - I see it now | 12:49 |
| *** udesale has joined #oooq | 12:49 | |
| rfolco | rlandy, just do a quick review pls, and I'll submit first patch to tripleo-docs. I just don't want to expose things that may compromise us. | 12:50 |
| rlandy | k | 12:50 |
| rfolco | rlandy, coz once you submit a patchset upstream, there is no way to delete it | 12:50 |
| quiquell|lunch | rfolco: do we need to manually put some containers at stein? | 12:58 |
| quiquell|lunch | rfolco: need that to debug the emitter review | 12:58 |
| rfolco | quiquell|lunch, yes... we need to sync on that. zbr may get something maybe? | 12:59 |
| quiquell|lunch | Rfo | 12:59 |
| quiquell|lunch | rfolco: can't we just do a docker push manually from promoted stuff at master? | 12:59 |
| rfolco | quiquell|lunch, in theory yes. That's exactly what wes said we should do. | 13:00 |
| quiquell|lunch | Rfo | 13:00 |
| rfolco | quiquell|lunch, check with zbr if he is going to do this, otherwise please help with this task | 13:00 |
| rfolco | quiquell|lunch, wtf is Rfo | 13:00 |
| rfolco | :) | 13:00 |
| quiquell|lunch | rfolco: is there any task ? | 13:00 |
| quiquell|lunch | rfolco: that's me not hiring tab autocompletion | 13:01 |
| quiquell|lunch | Hiting | 13:01 |
| rfolco | zbr, you there? have you tried pushing master containers as stein? | 13:01 |
| *** quiquell|lunch is now known as quiquell | 13:02 | |
| rfolco | quiquell, good point, I don't see where this work would fit, it is worth to create a new user story IMO | 13:03 |
| rfolco | quiquell, creating | 13:03 |
| quiquell | Will take care of it | 13:03 |
| rfolco | quiquell, excellent!! Just talk/sync w/ zbr about it | 13:04 |
| quiquell | zbr: ping | 13:05 |
| *** amoralej is now known as amoralej|lunch | 13:06 | |
| quiquell | rfolco: have you created dthe user story or I create it ? | 13:06 |
| quiquell | rlandy: rocky running now at bm dlrn trigger | 13:07 |
| rfolco | quiquell, creating | 13:07 |
| rlandy | quiquell: ack - thanks for merging that | 13:07 |
| rfolco | someone knocking my door | 13:08 |
| rlandy | quiquell: just checking the hashes now on the master job | 13:08 |
| rfolco | quiquell, https://tree.taiga.io/project/tripleo-ci-board/us/915 | 13:08 |
| quiquell | rfolco: is missing the epic | 13:09 |
| quiquell | rfolco: is this ? #837 Branch master to Stein | 13:09 |
| rfolco | quiquell, yes | 13:10 |
| rfolco | correct | 13:10 |
| quiquell | rfolco: so we have manully do a promotion at stein and I have to take the containers from there | 13:11 |
| quiquell | rfolco: is that it ? | 13:11 |
| rfolco | quiquell, isn't the last promotion enough ? | 13:12 |
| rfolco | quiquell, pull master containers from last promotion and push them back to tripleostein namespace | 13:12 |
| rfolco | quiquell, would this work ? | 13:13 |
| *** vinaykns has joined #oooq | 13:13 | |
| quiquell | rfolco: depends on where current-tripleo is pointing to at stein | 13:13 |
| rfolco | quiquell, will be out for 15 min brb... and we sync when I get back | 13:14 |
| quiquell | sure | 13:14 |
| quiquell | thanks dude | 13:14 |
| rfolco | ty | 13:14 |
| rlandy | quiquell: we are getting failures on mater bm we do not see in ovb | 13:16 |
| rlandy | waiting for the last of the patch to merge and then we should see the same pass rate | 13:16 |
| rlandy | if not, we have a problem | 13:16 |
| rlandy | quiquell: moving your dlrn triggering card to done | 13:17 |
| *** aakarsh has quit IRC | 13:18 | |
| quiquell | rlandy: ack, you mean the build-test-packages patch ? | 13:18 |
| rlandy | quiquell: the bm one - it was in QE | 13:19 |
| rlandy | quiquell: looking at this error: https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/periodic-tripleo-ci-centos-7-bm_envD-3ctlr_1comp-featureset001-master/7df8cd7/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-03-27_08_40_39 | 13:21 |
| rlandy | ovb on the same hash passed | 13:21 |
| quiquell | TASK [container-registry : enable and start docker] | 13:23 |
| quiquell | this is the one failing | 13:23 |
| quiquell | python exception is just in the middle of ansible logs | 13:24 |
| quiquell | rlandy: ... aren't we supose to be using podman ? | 13:25 |
| quiquell | rlandy: there is no docker daemon there | 13:26 |
| rlandy | quiquell: comparing with ovb | 13:26 |
| quiquell | rlandy: what's the OVB counterpart ? | 13:28 |
| rlandy | quiquell: we should be per https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset001.yml#L162 | 13:28 |
| rlandy | quiquell: http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/4550981/logs/undercloud/home/zuul/ | 13:29 |
| rlandy | ^^ ovb equivalent | 13:29 |
| rlandy | quiquell; there is no container specific code in http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/hardware_environments/hp_dl360_envD/network_configs/single_nic_vlans/env_settings.yml | 13:31 |
| *** jpena|lunch is now known as jpena | 13:32 | |
| rlandy | http://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/4550981/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz has the same docker lines | 13:32 |
| quiquell | yep | 13:32 |
| rlandy | quiquell: looks like more an access issue - maybe because we are not in rdocloud | 13:32 |
| quiquell | what container specific do we need there | 13:32 |
| rlandy | missing registry access | 13:32 |
| quiquell | hummm | 13:32 |
| quiquell | but this is a daemon restart | 13:32 |
| quiquell | docker daemon restart does not try to access I think | 13:32 |
| rlandy | here is where periodic and check differ | 13:33 |
| *** mjturek has joined #oooq | 13:33 | |
| quiquell | Let's find error at docker restart | 13:33 |
| rlandy | https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml#L4 | 13:35 |
| rlandy | ^^ different in regular master | 13:35 |
| *** matbu has quit IRC | 13:35 | |
| quiquell | rlandy: controller-2 is doing a enable/start docker and working fine | 13:36 |
| quiquell | rlandy: ripleo-ci-internal-jobs/master/periodic-tripleo-ci-centos-7-bm_envD-3ctlr_1comp-featureset001-master/7df8cd7/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2019-03-27_08_40_06 | 13:37 |
| * rlandy check network masquerade | 13:37 | |
| rlandy | tripleo::masquerade_networks::masquerade_networks: {'10.12.150.0/26': ['10.12.150.0/26'], '10.12.150.64/26': ['10.12.150.64/26']} | 13:37 |
| rlandy | that is right | 13:37 |
| quiquell | rlandy: where are te logs for overcloud-novacompute-0 ? | 13:39 |
| quiquell | is the machine still up ? | 13:39 |
| rlandy | quiquell: hmm - there are none - my guess is it is not accessible | 13:39 |
| rlandy | machine should still be up | 13:40 |
| rlandy | check it | 13:40 |
| rlandy | checking it | 13:40 |
| quiquell | let's ssh to nova node | 13:40 |
| rlandy | yep doing that | 13:41 |
| quiquell | just try to do enable/start by hand | 13:41 |
| quiquell | of docker daemon and see what happend | 13:41 |
| rlandy | wow | 13:42 |
| rlandy | can't get on node | 13:42 |
| quiquell | ups... | 13:42 |
| quiquell | but ansible is able to access | 13:42 |
| rlandy | (undercloud) [zuul@undercloud ~]$ ssh heat-admin@10.12.150.21 | 13:42 |
| rlandy | Unable to get valid context for heat-admin | 13:42 |
| rlandy | Last login: Wed Mar 27 13:41:49 2019 from 10.12.150.1 | 13:42 |
| rlandy | Connection to 10.12.150.21 closed. | 13:42 |
| quiquell | maybe there were network issues | 13:42 |
| rlandy | same error | 13:42 |
| rlandy | something is up there | 13:42 |
| rlandy | we could before | 13:42 |
| Tengu | overloaded system? | 13:42 |
| *** matbu has joined #oooq | 13:42 | |
| rlandy | it's baremetal | 13:42 |
| * Tengu remembers of some overloaded baremetal system at his previous job | 13:43 | |
| rlandy | usually an selinux error, no? | 13:43 |
| quiquell | rlandy: hummm i see that at logs | 13:44 |
| Tengu | https://access.redhat.com/solutions/1982653 ? | 13:44 |
| rlandy | I have not seen that before | 13:44 |
| rlandy | (undercloud) [zuul@undercloud ~]$ getenforce | 13:44 |
| rlandy | Enforcing | 13:44 |
| rlandy | ^^ wonder if that is correct | 13:44 |
| *** aakarsh has joined #oooq | 13:45 | |
| quiquell | rlandy: but this is selinux of the undercloud | 13:45 |
| quiquell | rlandy: what about selinux of the nova node ? | 13:45 |
| rlandy | quiquell: it is | 13:45 |
| quiquell | rlandy: maybe you can go ingot one of the overcloud controllers | 13:46 |
| quiquell | and check selinux | 13:46 |
| rfolco | quiquell, zbr do we need to talk about stein containers or this is already clarified ? please update user story with whatever approach we'll be using. | 13:46 |
| quiquell | rfolco: didn't progress on that | 13:46 |
| rlandy | quiquell: do you want me to tmate you in? | 13:46 |
| rlandy | idk what the nova compute node ip is on that network | 13:47 |
| quiquell | rlandy: ok let's add a pair of eyes there | 13:47 |
| rfolco | quiquell, tmate w/ rlandy on bm issue, we bj later w/ some expert in dlrn hashes, maybe sshnaidm|rover | 13:48 |
| quiquell | rfolco: ok | 13:48 |
| rlandy | k - exiting and starting tmate | 13:48 |
| quiquell | rlandy: I am in | 13:50 |
| rlandy | quiquell: pls open your window more | 13:50 |
| *** ykarel is now known as ykarel|away | 13:52 | |
| marios_|ruck | sshnaidm|rover: any idea about that one https://bugs.launchpad.net/tripleo/+bug/1821744?comments=all /me staring at the deploy command and environment files | 13:52 |
| openstack | Launchpad bug 1821744 in tripleo "various jobs failing for overcloud deploy with missing cluster error: Could not connect to cluster (is it running?)", "" [Critical,Triaged] | 13:52 |
| marios_|ruck | http://logs.rdoproject.org/45/560445/291/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/126a766/logs/undercloud/home/zuul/overcloud-deploy.sh | 13:52 |
| sshnaidm|rover | marios_|ruck, asked bandini to look into it | 13:52 |
| marios_|ruck | sshnaidm|rover: cool thanks | 13:53 |
| quiquell | rlandy: ovb rdocloud -> overcloud_image_url: http://images.rdoproject.org/{{ release }}/rdo_trunk/{{ dlrn_hash|default(dlrn_hash_tag) }}/overcloud-full.tar | 13:57 |
| *** matbu has quit IRC | 13:57 | |
| rlandy | quiquell: yeah - we do not have that in virt deployment | 13:57 |
| rlandy | we should use the one from relaese file | 13:57 |
| rlandy | https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml#L24 | 13:58 |
| rlandy | quiquell: ^^ | 13:58 |
| *** matbu has joined #oooq | 13:59 | |
| *** ykarel|away has quit IRC | 13:59 | |
| rlandy | quiquell: we need to check which one take preference in ovb | 14:00 |
| rlandy | release or env | 14:00 |
| quiquell | rlandy: they don't point to the same ? | 14:01 |
| quiquell | rlandy: where is the env for bm ? | 14:01 |
| *** Vorrtex has joined #oooq | 14:01 | |
| quiquell | rlandy: or are we using the same as ovb ? | 14:01 |
| rlandy | quiquell: no - we replace with http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/hardware_environments/hp_dl360_envD/network_configs/single_nic_vlans/env_settings.yml | 14:01 |
| rlandy | says nothing about images | 14:01 |
| rlandy | the bm one is blank | 14:01 |
| rfolco | rlandy, quiquell may be a stupid suggestion... but... # guestfish -i --rw -d <image> touch /.autorelabel | 14:01 |
| quiquell | so we use release | 14:02 |
| quiquell | rfolco: it's working at OVB | 14:02 |
| rlandy | 2019-03-27 07:23:17.755811 | primary | TASK [setup/undercloud : Perform selinux relabel on undercloud image] ********** | 14:03 |
| rfolco | or | 14:04 |
| rfolco | sh “/sbin/setfiles -q /etc/selinux/targeted/contexts/files/file_contexts /” | 14:04 |
| quiquell | rlandy: can we access the nova using the drac ? | 14:04 |
| quiquell | rlandy: I have like a virtual monitor ? | 14:05 |
| quiquell | rlandy: so we access without ssh ? | 14:05 |
| rlandy | quiquell: ack but we would have to log in | 14:05 |
| rlandy | I can get iLO | 14:05 |
| rlandy | but you won't be able to log in from outside the undercloud | 14:05 |
| quiquell | rlandy: just to ensure that is not a ssh issue in that case we can do some investigations | 14:05 |
| quiquell | rlandy: there is no password for that user :-/ | 14:06 |
| rlandy | heat-admin no | 14:06 |
| Tengu | hmm so it's a baremetal overcloud with some weird/failing SSH/other right? | 14:06 |
| quiquell | Tengu: yep | 14:07 |
| quiquell | Tengu: error is similar to this https://superuser.com/questions/1224284/linux-console-unable-to-get-valid-context-for | 14:07 |
| quiquell | Tengu: and OVB is working fine with the same DLRN hash | 14:07 |
| Tengu | duh... is there a way to get the grub invit and hack its boot? | 14:08 |
| quiquell | Tengu: like runing /bin/bash at kernel options ? | 14:08 |
| Tengu | yeah, or the "right" rhel way | 14:08 |
| quiquell | Tengu: what's that ? | 14:08 |
| quiquell | Tengu: from iLO I suppose we can do that | 14:09 |
| Tengu | don't remember, I saw that once or twice while trying to get back my centos. wait. | 14:09 |
| Tengu | quiquell: it's rhel7 right? | 14:09 |
| quiquell | Tengu: centos7 | 14:09 |
| rlandy | quiquell: I think OVB has a diff image | 14:10 |
| rlandy | http://images.rdoproject.org/master/rdo_trunk/a8f9a6277055e2c106c2af4f4c56766d417c0cd6_128c9cd8/overcloud-full.tar' | 14:10 |
| Tengu | quiquell: systemd.unit=emergency.target instead of the "init=/bin/bash" | 14:11 |
| Tengu | source: https://www.thegeekdiary.com/centos-rhel-7-how-to-boot-into-rescue-mode-or-emergency-mode/ | 14:11 |
| rlandy | a8f9a6277055e2c106c2..>2019-03-27 09:11- vs current-tripleo/2019-03-24 08:56- | 14:11 |
| quiquell | Tengu: thanks! | 14:11 |
| Tengu | that's the official way for rhel based systems | 14:11 |
| quiquell | rlandy: so OVB is using release file | 14:12 |
| * rlandy checks order of arguments | 14:13 | |
| Tengu | quiquell: wondering if a root password is needed for that kind of target. there's also "rescue.target" btw. | 14:13 |
| quiquell | Tengu: no clue | 14:13 |
| Tengu | not sure which one I used back then in order to save my local server... | 14:13 |
| quiquell | Let's see first the rlandy discovery on images | 14:14 |
| quiquell | Looks like OVB and BM are using different ones | 14:14 |
| Tengu | that would make sense imho... | 14:14 |
| Tengu | drivers and the like? | 14:14 |
| rlandy | quiquell: ack ... /home/zuul/src/git.openstack.org/openstack-infra/tripleo-ci/toci-quickstart/config/testenv/ovb-rdocloud.yml is passed after release file | 14:15 |
| Tengu | quiquell: so apparently the issue is also the same I pointed from the Access thingy. | 14:15 |
| quiquell | rlandy: so release file has preference | 14:15 |
| rlandy | quiquell: no the ovb-rdocloud env does | 14:15 |
| rlandy | the file passed last takes pref | 14:16 |
| quiquell | ahh after | 14:16 |
| quiquell | rlandy: humm that's weird | 14:16 |
| rlandy | quiquell: ok - we can fix this to match ovb | 14:16 |
| quiquell | rlandy: yes but | 14:16 |
| rlandy | quiquell: truth is release file is wrong | 14:16 |
| rlandy | promotion testing should take hash | 14:16 |
| quiquell | the OVB job was a check job | 14:17 |
| quiquell | check job has to use tripleo-current for images | 14:17 |
| quiquell | isn't it ? | 14:17 |
| rlandy | quiquell: no periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master | 14:17 |
| quiquell | rlandy: ahh was periodic | 14:17 |
| rlandy | quiquell; ack - check jobs takes current-tripleo | 14:17 |
| *** amoralej|lunch is now known as amoralej | 14:17 | |
| rlandy | the comparison job was periodic though | 14:17 |
| quiquell | rlandy: periodics should take current | 14:17 |
| quiquell | rlandy: or tripleo-ci-testing ? | 14:18 |
| rlandy | latest | 14:18 |
| rlandy | checking what hashes match what dates | 14:18 |
| quiquell | rlandy: is not tripleo-ci-testing ? since this is the tag we use before running periodics | 14:18 |
| rlandy | tripleo-ci-testing/2019-03-27 09:11- | 14:18 |
| rlandy | matches | 14:18 |
| quiquell | make sense | 14:18 |
| rlandy | a8f9a6277055e2c106c2..>2019-03-27 09:11- | 14:18 |
| quiquell | cool | 14:19 |
| *** sshnaidm_ has joined #oooq | 14:19 | |
| *** sshnaidm_ is now known as sshnaidm|bbl | 14:19 | |
| rlandy | quiquell: so - rdocloud jobs are always using latest | 14:19 |
| rlandy | quiquell: even in check | 14:19 |
| rlandy | interesting , no??? | 14:19 |
| quiquell | even in check ? | 14:19 |
| rlandy | this to me is wrong ... | 14:19 |
| rlandy | quiquell: yeah - check order of arguments | 14:19 |
| quiquell | could be considered the images like tripleo projects ? | 14:19 |
| rlandy | https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-7/promotion-testing-hash-master.yml#L24 | 14:20 |
| quiquell | tripleo projects at taken from latest at check jobs | 14:20 |
| rlandy | ^^ why not latest hash?? | 14:20 |
| quiquell | This has to be tripleo-ci-testing | 14:20 |
| rlandy | quiquell: ack - putting in patch | 14:20 |
| quiquell | and for IPA too | 14:20 |
| rlandy | quiquell: we should question ovb at tomorrow's meeting | 14:20 |
| rlandy | let the team weigh in | 14:21 |
| quiquell | Yep | 14:21 |
| rlandy | you can see the results in the logs | 14:21 |
| *** sshnaidm|rover has quit IRC | 14:21 | |
| quiquell | rlandy: let's just add temporally the correct url to the BM env file | 14:21 |
| quiquell | rlandy: to see if it works now | 14:21 |
| quiquell | rlandy: for oc and API | 14:21 |
| quiquell | s/API/IPA/g | 14:22 |
| quiquell | Tengu: we will see how it goes with same images, if not let's go back to hack into images | 14:22 |
| *** sshnaidm|bbl has quit IRC | 14:22 | |
| quiquell | Tengu: thanks btw | 14:22 |
| rlandy | quiquell: ack - I can temp merge a change in tripleo-envs for that | 14:22 |
| rlandy | quiquell: hold on - changes coming up | 14:22 |
| quiquell | rlandy: ack at the end is what OVB wrongly do :-) | 14:22 |
| rlandy | quiquell: the release file is wrong and OVB is wrong | 14:23 |
| Tengu | quiquell: np :). I quit the tmate - feel free to call me if you want some more help. | 14:23 |
| rlandy | but we are not seeing it because both files are passed | 14:23 |
| quiquell | rlandy: let's make BM correctly wrong | 14:23 |
| rlandy | Tengu: we will let you know | 14:23 |
| rlandy | ok - wrongly correcting BM | 14:24 |
| quiquell | that's the proper permutation | 14:24 |
| rlandy | this is why I love BM - it finds all the holes | 14:24 |
| quiquell | so, we were running check with latest images | 14:25 |
| quiquell | at OVB | 14:25 |
| quiquell | let's check master.yaml though | 14:25 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, (3 more messages) | 14:27 |
| rlandy | quiquell: https://code.engineering.redhat.com/gerrit/166315 Temp adding image path to env file | 14:28 |
| rlandy | quiquell: ^^ merging this | 14:29 |
| quiquell | rlandy: yep, I don't have +2 there | 14:29 |
| rlandy | quiquell: done - now exiting out of hardware | 14:29 |
| quiquell | rlandy: let's just wait for next spin | 14:30 |
| rlandy | quiquell: yep - next time we get a periodic run, we check images | 14:30 |
| quiquell | we can enforce periodic at check, but it's going to run in half hour I think | 14:30 |
| rlandy | quiquell: it's the only thing I can think of that is diff in the check vs the periodic | 14:30 |
| quiquell | rlandy: BM at check was working fine ? | 14:31 |
| quiquell | rlandy: release file for periodic and check are the same there | 14:31 |
| quiquell | rlandy: same IP and oc url | 14:32 |
| rlandy | quiquell: ack - we can access nova in check | 14:32 |
| rlandy | quiquell: but in check the repos and images match | 14:32 |
| quiquell | rlandy: humm there is something weird | 14:32 |
| quiquell | rlandy: ansibe was able to access nova | 14:32 |
| quiquell | I think | 14:32 |
| rlandy | quiquell:env B ran master check | 14:32 |
| rlandy | you can look at that env | 14:33 |
| rlandy | we have two diff envs | 14:33 |
| rlandy | two running rocky | 14:33 |
| rlandy | two running master | 14:33 |
| rlandy | one check one periodic | 14:33 |
| * rlandy check env B in tmate | 14:33 | |
| quiquell | I think I have close tmate :-( | 14:33 |
| rlandy | you can connect again | 14:34 |
| quiquell | this is check ? | 14:36 |
| rlandy | quiquell: ack - it's check | 14:38 |
| rlandy | I am waiting for my last patch to mergs | 14:38 |
| rlandy | quiquell: then will run check again to see if master passes | 14:38 |
| quiquell | rlandy: still I see the same content at both periodic and check release files | 14:39 |
| quiquell | rlandy: so how so images at check are alright ? | 14:39 |
| rlandy | quiquell: because at check the repos match the iamges | 14:39 |
| quiquell | rlandy: ahh that's right | 14:40 |
| quiquell | rlandy: repos for tripleo are at latest | 14:40 |
| quiquell | rlandy: and for the rest at current-tripleo | 14:40 |
| rlandy | quiquell; in theory, the repos should update the images | 14:40 |
| quiquell | let's just wait | 14:40 |
| rlandy | quiquell: but with overcloud not sure that happens | 14:40 |
| rlandy | quiquell: either way, we need to match the OVB workflow | 14:41 |
| rlandy | so that is what I am doing here | 14:41 |
| quiquell | rlandy: sure | 14:41 |
| quiquell | rlandy: good catch though | 14:41 |
| rlandy | quiquell: we'll see | 14:41 |
| rlandy | I may not have fixed the problem | 14:41 |
| quiquell | Let's see what's next:-) | 14:41 |
| rlandy | quiquell; rerunning check now to see | 14:43 |
| quiquell | wait | 14:44 |
| quiquell | do the periodic at check hack to use the correct release file | 14:44 |
| quiquell | so we mimic it | 14:44 |
| rlandy | yes - but I actually want to see if that env is needed for debug for https://bugs.launchpad.net/tripleo/+bug/1821744 | 14:45 |
| openstack | Launchpad bug 1821744 in tripleo "various jobs failing for overcloud deploy with missing cluster error: Could not connect to cluster (is it running?)", "" [Critical,Triaged] | 14:45 |
| *** sshnaidm|bbl has joined #oooq | 14:48 | |
| *** ykarel|away has joined #oooq | 14:53 | |
| *** derekh has quit IRC | 14:53 | |
| *** ratailor has joined #oooq | 14:54 | |
| *** ykarel|away is now known as ykarel | 14:58 | |
| marios_|ruck | zbr: any update on stein containers? was it you looking there ? rfolco cc alex is asking in tripleo | 14:59 |
| *** sshnaidm|bbl has quit IRC | 15:00 | |
| quiquell | marios_|ruck: this is about pushing master containers to tripleostein ? | 15:02 |
| quiquell | marios_|ruck: we have just created a user story for it https://tree.taiga.io/project/tripleo-ci-board/us/915?milestone=223049 | 15:02 |
| quiquell | marios_|ruck: I needed it to test the changes at emit releases script | 15:03 |
| marios_|ruck | quiquell: yeah | 15:03 |
| quiquell | zbr: I was planning on doing manual pushing https://tree.taiga.io/project/tripleo-ci-board/us/915?milestone=223049 | 15:04 |
| zbr | quiquell: if you know how to doit, please to it manually, it can only help | 15:04 |
| quiquell | zbr: ack will do it | 15:05 |
| rfolco | quiquell, do you know how to do it or need help ? | 15:10 |
| quiquell | rfolco: I have the creds and all | 15:11 |
| rfolco | quiquell, let me know if you need help, looks like this is blocking all other stein work | 15:12 |
| *** sshnaidm|bbl has joined #oooq | 15:15 | |
| marios_|ruck | arxcruz: chandankumar o/ hey this came up today in escalation call any updates for it https://trello.com/c/DfQAjJrJ/819-cixlp1802971tripleociproa-tempest-volumebootpattern-and-basicops-running-concurrently-causing-timeouts | 15:17 |
| arxcruz | marios_|ruck: so, there's nothing else to do, it can be moved to done, we are more covered as i exlain in my comment | 15:19 |
| arxcruz | marios_|ruck: I comment on the card | 15:20 |
| quiquell | humm | 15:20 |
| quiquell | rfolco: so this is current-tripleo for stein https://trunk.rdoproject.org/centos7-stein/current-tripleo/commit.yaml | 15:20 |
| marios_|ruck | arxcruz: ack cos last comment was 'ill post results here' but there wasn't follow up hence this ping ok thanks for update | 15:21 |
| arxcruz | marios_|ruck: sorry, my bad, it's very easy to lost the track on all things we do in several different tools | 15:21 |
| arxcruz | i wasn't able to start working properly on this sprint with all tempest things... :/ | 15:21 |
| marios_|ruck | arxcruz: it's ok. i will forgive you. this one time. | 15:21 |
| arxcruz | marios_|ruck: efcharisto gia tin kalosyni sou | 15:22 |
| rfolco | quiquell, what is the risk in using this hash? | 15:22 |
| quiquell | rfolco: how to I map between that hash and the containers ? | 15:23 |
| quiquell | at master ? | 15:23 |
| marios_|ruck | arxcruz: de nada | 15:23 |
| rfolco | quiquell, dunno, I'll do some research | 15:26 |
| rfolco | maybe rlandy knows? | 15:26 |
| quiquell | I think we have to rebuild the images | 15:26 |
| quiquell | The containers I mean | 15:27 |
| zbr | ykarel: i combined fixes, i hope now https://review.openstack.org/#/c/648070/ is ok, this being the 1st in a long chain. | 15:30 |
| rlandy | rfolco: notes on https://tree.taiga.io/project/tripleo-ci-board/task/888?kanban-status=1447275] | 15:31 |
| ykarel | zbr, looks like you missed comment | 15:31 |
| rlandy | rfolco: maybe I know what? | 15:31 |
| rfolco | rlandy, how to push stein containers from master ones | 15:32 |
| quiquell | rlandy: do you know how if it's possible to map a stein DLRN hash to master containers ? | 15:32 |
| rfolco | rlandy, thanks for reviewing 888 | 15:32 |
| quiquell | rlandy: Or we are force to build them ? | 15:32 |
| quiquell | rfolco: since stein and master are still the same | 15:32 |
| rlandy | quiquell; hmmm ... I don't really - what do you mean by map exactly | 15:33 |
| zbr | ykarel: not sure what is wrong with comment other than a minor typo? | 15:33 |
| quiquell | rlandy: We need to have some containers at tripleostein container repo | 15:33 |
| rlandy | quiquell: retag? | 15:33 |
| rfolco | quiquell, rlandy just hacking our check job to push to tripleostein namespace ? | 15:33 |
| quiquell | rlandy: we were thinking about just pushing from tripleomaster to tripleostein | 15:33 |
| ykarel | quiquell, no stein and master are not same, they diverted as projects have created stable/stein, so they may differ | 15:33 |
| quiquell | ykarel: so we have to build the containers | 15:34 |
| *** matbu has quit IRC | 15:34 | |
| quiquell | ykarel: yep that's what I was finding out | 15:34 |
| rfolco | brb | 15:34 |
| ykarel | zbr, https://review.openstack.org/#/c/648070/2..4/config/release/tripleo-ci/Fedora-28/master.yml | 15:34 |
| ykarel | Line 20 | 15:34 |
| quiquell | rfolco: but we have to construct with correct repos and all | 15:35 |
| *** udesale has quit IRC | 15:35 | |
| quiquell | rfolco: so a containers-builds-push for stein need to be run | 15:35 |
| quiquell | rfolco: same we are doing with rocky | 15:35 |
| ykarel | zbr, and you can update commit message to include fedora_stable ugly hack removal part | 15:35 |
| quiquell | rfolco: but forcing to current-tripleo instead of tripleo-ci-testing | 15:35 |
| ykarel | as it seems you included in that patch | 15:36 |
| quiquell | marios_|ruck: the rocky containers-build-push is working now ' | 15:36 |
| quiquell | now ? | 15:36 |
| marios_|ruck | quiquell: not green yet at https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-7-rocky-containers-build-push or not reported | 15:36 |
| quiquell | rfolco: have to drop now, will check doing it tomorrow | 15:36 |
| marios_|ruck | quiquell: the fail there is from before the fixes | 15:37 |
| quiquell | rfolco: at least containres-build-push does not depends on release file I think | 15:37 |
| marios_|ruck | but no more recent yet | 15:37 |
| quiquell | marios_|ruck: ack | 15:37 |
| *** quiquell is now known as quiquell|off | 15:38 | |
| *** matbu has joined #oooq | 15:40 | |
| *** ratailor has quit IRC | 15:41 | |
| *** saneax has quit IRC | 15:43 | |
| zbr | ykarel: i have the impression that first bullet was about fedora-stable hack removal. | 15:43 |
| *** holser_ has quit IRC | 15:43 | |
| *** saneax has joined #oooq | 15:44 | |
| ykarel | marios_|ruck, rocky container build job passed:- https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-rocky-containers-build-push/52590a7/ | 15:44 |
| ykarel | in current periodic run | 15:44 |
| marios_|ruck | ykarel: thanks 'me was trying to find via zuul status looking | 15:44 |
| marios_|ruck | ykarel: sshnaidm|bbl quiquell|off lgtm https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-rocky-containers-build-push/52590a7/etc/yum.repos.d/delorean.repo.txt.gz https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-rocky-containers-build-push/52590a7/logs/build.log.txt.gz and no excludes in https | 15:45 |
| marios_|ruck | \o/ | 15:45 |
| *** matbu has quit IRC | 15:45 | |
| ykarel | zbr, that;s different 1st bullet | 15:46 |
| ykarel | looks like mixed reviews | 15:46 |
| zbr | ykarel: ahh.. right. | 15:46 |
| sshnaidm|bbl | marios_|ruck, yay | 15:47 |
| *** holser_ has joined #oooq | 15:47 | |
| zbr | ykarel: updated commit msg, ok now? | 15:48 |
| ykarel | zbr, looking, and l20 too? | 15:48 |
| ykarel | zbr, L20 not updated yet | 15:49 |
| ykarel | https://trunk.rdoproject.org/{{ distro_ver }} --> https://trunk.rdoproject.org/{{ distro_ver }}-{{ release }} | 15:50 |
| ykarel | that was the main comment :) as per your patch you wanted the release file to be usable | 15:50 |
| *** sshnaidm|bbl has quit IRC | 15:50 | |
| ykarel | so i suggested doing that | 15:50 |
| zbr | ykarel: sure, doing it not. and really thanks for help on this. | 15:51 |
| ykarel | looks like we are out of resources:- resources.baremetal_env.resources.openstack_baremetal_servers.resources[2].resources.baremetal_server: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., | 15:54 |
| ykarel | marios_|ruck, ^^ seeing in a job | 15:54 |
| *** holser_ has quit IRC | 15:54 | |
| zbr | ykarel: if we are at this section, lets resolve few other questions. docker_openshift_tag should be the same for c7/f28? | 15:55 |
| marios_|ruck | ykarel: you have link for me ? | 15:55 |
| ykarel | seeing some job went to retry_limit at https://review.rdoproject.org/zuul/status | 15:55 |
| ykarel | marios_|ruck, one example:- https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky/d05cf41/job-output.txt.gz | 15:55 |
| marios_|ruck | ykarel: thanks | 15:55 |
| ykarel | i see only tihs | 15:55 |
| marios_|ruck | ykarel: dam. hoping for rocky promotion | 15:55 |
| marios_|ruck | ykarel: any idea what we can do about it? do we need to cancel jobs/restart them ? | 15:56 |
| ykarel | marios_|ruck, first ask at rhos-ops | 15:56 |
| ykarel | if rdo cloud is out of capacity | 15:57 |
| ykarel | also see other job logs to see how many jobs affected | 15:57 |
| ykarel | as i see some running also | 15:57 |
| marios_|ruck | ykarel: ack pinged rhos-ops | 16:00 |
| ykarel | ack | 16:00 |
| marios_|ruck | don'tknow who to ping there its just before you joined the chan not sure who will see | 16:00 |
| ykarel | marios_|ruck, ask kforde, alheran | 16:00 |
| ykarel | marios_|ruck, also looking at https://zabbix.infra.prod.eng.rdu2.redhat.com/zabbix/screens.php?elementid=224 | 16:02 |
| ykarel | seems only 33 vms possible with m1.large, i don't know how to read that | 16:03 |
| marios_|ruck | ykarel: looking | 16:04 |
| marios_|ruck | ykarel: thanks i've never seen this dash before | 16:05 |
| ykarel | np, you can bookmark it or add to ruck/rover etherpad | 16:06 |
| zbr | i guess that we should see things inside https://images.rdoproject.org/fedora28/ -- right? now is empty. | 16:06 |
| *** saneax has quit IRC | 16:06 | |
| rlandy | rfolco: quiquell|off: https://review.openstack.org/648170 promotion-testing-hash-* release file should use tripleo-ci-testing | 16:07 |
| zbr | overcloud_image_url seems to be pointing to https://images.rdoproject.org/master/ instead of https://images.rdoproject.org/centos7/master/ --- easy fix, BUT the /fedora28 folder is *empty*. | 16:07 |
| rlandy | zbr: ^^ fyi | 16:07 |
| zbr | ykarel: rlandy: re https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/Fedora-28/master.yml#L26-L27 -- my impression is that this urls are not correct, as we miss a the distro_ver part out of it. | 16:11 |
| rlandy | zbr: pls comment on review | 16:14 |
| marios_|ruck | ykarel++ | 16:16 |
| hubbot1 | marios_|ruck: ykarel's karma is now 13 | 16:16 |
| marios_|ruck | thanks | 16:16 |
| *** ykarel is now known as ykarel|away | 16:16 | |
| ykarel|away | marios_|ruck, i see notification fix released:- https://bugs.launchpad.net/bugs/1820791 | 16:19 |
| openstack | Launchpad bug 1820791 in tripleo "should use containers-default-parameters.yaml instead of /home/stack/containers-prepare-parameter.yaml for overcloud deploy" [Undecided,Fix released] - Assigned to yatin (yatinkarel) | 16:19 |
| ykarel|away | i think you can readd fs030 to queens promotion criteria if it's passing | 16:19 |
| ykarel|away | https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-7/queens.ini#L25 | 16:19 |
| marios_|ruck | ykarel|away: ack | 16:21 |
| marios_|ruck | ykarel|away: well was just skipped but will check (i mean the tempest stuff just merged the skip) | 16:22 |
| marios_|ruck | ykarel|away: sorry confused it with master issue nm | 16:23 |
| marios_|ruck | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-queens | 16:23 |
| marios_|ruck | green run there ykarel|away ^ looks like | 16:23 |
| ykarel|away | yes | 16:24 |
| rlandy | marios_|ruck: no worries on the ticket - I can delete those instances | 16:25 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, (3 more messages) | 16:27 |
| marios_|ruck | ykarel|away: https://review.rdoproject.org/r/19788 | 16:27 |
| marios_|ruck | rlandy: thank you | 16:27 |
| marios_|ruck | rlandy: never done that before | 16:27 |
| marios_|ruck | didn't even know about rhos-ops | 16:27 |
| chandankumar | see ya guys tomorrow | 16:28 |
| *** chandankumar is now known as raukadah | 16:28 | |
| *** dsneddon has quit IRC | 16:28 | |
| rlandy | marios_|ruck: see https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-cleanup | 16:28 |
| rlandy | marios_|ruck: I just pulled the server part out of that to clean up | 16:28 |
| marios_|ruck | rlandy: --nuclear | 16:29 |
| rlandy | we needed it on monday as well | 16:29 |
| marios_|ruck | ! | 16:29 |
| rlandy | NO NOT NUCLEAR | 16:29 |
| marios_|ruck | rlandy: i see. is it something we're doing wrong/? or we need capacity increase | 16:29 |
| marios_|ruck | or what | 16:29 |
| marios_|ruck | every now and again we need a nuke | 16:29 |
| marios_|ruck | but otherwise | 16:29 |
| marios_|ruck | we're good | 16:29 |
| rlandy | marios_|ruck: really it could be anything | 16:29 |
| rlandy | I haven't been watching the env close enough to tell you what it is this time | 16:29 |
| marios_|ruck | rlandy: ack thanks | 16:29 |
| rlandy | probably an outage of sorts | 16:30 |
| marios_|ruck | rlandy: thers been multiple outage/maintenance last few days if i recall on email list anyway for rdo maybe related? | 16:30 |
| marios_|ruck | like jobs geting stuck in deleted during that ^ thn staying there? | 16:30 |
| marios_|ruck | eating resources for no reason whatsoever | 16:30 |
| rlandy | anyways you should see the number of error instances dropping | 16:31 |
| rlandy | script is running | 16:31 |
| *** trown is now known as trown|lunch | 16:35 | |
| marios_|ruck | rlandy++ | 16:36 |
| hubbot1 | marios_|ruck: rlandy's karma is now 50 | 16:36 |
| marios_|ruck | thank you | 16:36 |
| *** jfrancoa has quit IRC | 16:43 | |
| *** sshnaidm|bbl has joined #oooq | 16:46 | |
| *** dsneddon has joined #oooq | 16:55 | |
| *** dsneddon has quit IRC | 17:01 | |
| *** bogdando has quit IRC | 17:05 | |
| *** kopecmartin is now known as kopecmartin|off | 17:10 | |
| * marios_|ruck out ttyl folks | 17:11 | |
| *** dtantsur is now known as dtantsur|afk | 17:21 | |
| *** dsneddon has joined #oooq | 17:31 | |
| *** trown|lunch is now known as trown | 17:50 | |
| *** jpena is now known as jpena|off | 18:03 | |
| *** ykarel|away has quit IRC | 18:16 | |
| *** holser_ has joined #oooq | 18:22 | |
| *** zul has quit IRC | 18:23 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset053, tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, (3 more messages) | 18:27 |
| zbr | quiquell|off: sshnaidm|bbl marios_|ruck please revote on https://review.openstack.org/#/c/648070/ | 18:36 |
| *** amoralej is now known as amoralej|lunch | 18:42 | |
| *** holser_ has quit IRC | 19:33 | |
| rfolco | rlandy, sshnaidm|bbl is away looks like | 19:37 |
| rfolco | wanted to understand how to push master containers to tripleostein namespace | 19:37 |
| rlandy | sshnaidm|bbl; if you are not really bbl :) - can you join bj to help rfolco figure out ^^ | 19:38 |
| sshnaidm|bbl | rlandy, which blue? | 19:38 |
| rlandy | rfolco? | 19:38 |
| rfolco | https://bluejeans.com/5878458097 | 19:39 |
| rfolco | zbr, sshnaidm|bbl rlandy ^ | 19:39 |
| rfolco | sync on stein containers | 19:39 |
| zbr | if anyone still here, it would be really helpful to wf https://review.openstack.org/#/c/648070/ today. | 20:27 |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci- (3 more messages) | 20:27 |
| *** jtomasek has quit IRC | 20:27 | |
| *** jtomasek has joined #oooq | 20:28 | |
| *** Vorrtex has quit IRC | 20:28 | |
| *** mjturek has quit IRC | 20:31 | |
| *** fultonj has quit IRC | 20:34 | |
| *** fultonj has joined #oooq | 20:35 | |
| *** jtomasek has quit IRC | 20:35 | |
| rfolco | rlandy, sshnaidm|bbl https://review.rdoproject.org/r/#/c/19791/ | 20:37 |
| rfolco | when you get a chance pls. I'll check it later to test in testproject, thanks for your help | 20:38 |
| rlandy | rfolco: commented | 20:42 |
| rlandy | I don't understand why we need to redefine the whole job | 20:42 |
| sshnaidm|bbl | rfolco, better to define a job as normal stein job, like "rocky" is defined there, "push: false" you'll pass in testproject | 20:50 |
| rfolco | sshnaidm|bbl, ack | 20:50 |
| rfolco | removing push var override | 20:51 |
| rfolco | rocky was a copy we did not need to replicate all those vars from base job | 20:51 |
| *** trown is now known as trown|outtypewww | 20:52 | |
| rfolco | sshnaidm|bbl, rlandy good now ps3 ? https://review.rdoproject.org/r/#/c/19791/ | 20:54 |
| rlandy | rfolco: looks better to me | 20:57 |
| rlandy | rfolco: actually missed branch_override | 21:03 |
| rlandy | comparing to rocky | 21:03 |
| rfolco | this is not needed | 21:04 |
| rfolco | branch_override is a oooq thing for undercloud install | 21:04 |
| rfolco | I can put it back, though | 21:04 |
| rfolco | what matters is checkout-override | 21:04 |
| rfolco | rlandy, correct me if i am wrong | 21:05 |
| rlandy | rfolco: juts following rocky pattern | 21:05 |
| rfolco | ok lets not add any other noise | 21:05 |
| rfolco | rlandy, adding back | 21:05 |
| rfolco | rlandy, sshnaidm|bbl ready | 21:08 |
| rlandy | rfolco: ok - let's merge that so you can test | 21:09 |
| rlandy | it can't do harm | 21:10 |
| rlandy | as soon as zuul votes | 21:10 |
| rfolco | rlandy, thanks | 21:10 |
| rlandy | rfolco: it's workflowed | 21:11 |
| rlandy | you can add a testproject job now to test - with push set to false | 21:11 |
| rfolco | great! rlandy do you get an example of testproject patch like that ? | 21:12 |
| rlandy | rfolco: have you ever used testproject? | 21:13 |
| sshnaidm|bbl | rfolco, https://review.rdoproject.org/r/#/c/19681/ | 21:13 |
| sshnaidm|bbl | rfolco, or that: https://review.rdoproject.org/r/#/c/19769/ | 21:14 |
| rfolco | rlandy, sshnaidm|bbl thanks !! | 21:14 |
| rlandy | rfolco: you'll need to clone https://review.rdoproject.org/r/testproject | 21:15 |
| rlandy | and then add a .zuul.d like sshnaidm|bbl's review | 21:15 |
| sshnaidm|bbl | rfolco, and need to depend on rlandys patch I think to run it | 21:15 |
| sshnaidm|bbl | rlandy, do you have your periodic patch number? | 21:15 |
| rlandy | yep getting | 21:16 |
| rfolco | why depends-on a periodic patch ? | 21:16 |
| rlandy | https://review.openstack.org/#/c/638756/ | 21:16 |
| rlandy | so zuul considers your job periodic | 21:17 |
| rfolco | got it | 21:17 |
| rfolco | :) | 21:18 |
| rlandy | watch the branch there as well | 21:18 |
| sshnaidm|bbl | rlandy, commented on https://review.openstack.org/#/c/648195/ | 21:19 |
| sshnaidm|bbl | rfolco, rlandy branch should be master | 21:19 |
| sshnaidm|bbl | all periodics run on master | 21:20 |
| rlandy | sshnaidm|bbl: agree it's not the best place to put it | 21:20 |
| rlandy | in fatc I had to make another hack for bm | 21:20 |
| rlandy | sshnaidm|bbl: I'll look for a more useful file to add that param to - but at least we know it fixes the problem | 21:21 |
| sshnaidm|bbl | rlandy, yeah, that's great | 21:21 |
| sshnaidm|bbl | rlandy, maybe ovb config in tht | 21:21 |
| rlandy | sshnaidm|bbl: it's not just ovb though | 21:22 |
| rlandy | any thing that uses common/deploy-steps.j2 | 21:22 |
| sshnaidm|bbl | rlandy, but multinode jobs pass fine | 21:22 |
| rlandy | weird, right - it should object | 21:22 |
| rlandy | sshnaidm|bbl: probably because it's not HA | 21:24 |
| rlandy | and pacemaker picked up the problem | 21:24 |
| sshnaidm|bbl | rlandy, seems like.. | 21:25 |
| rlandy | sshnaidm|bbl: it's late for you - I will put in the ovb_ha patch to get tested and we can discuss/confirm in tomorrow's meeting | 21:25 |
| sshnaidm|bbl | rlandy, sure | 21:25 |
| rlandy | sshnaidm|bbl: I feel like multinode will show some issue at some point ... but we'll see | 21:25 |
| *** aakarsh has quit IRC | 21:35 | |
| *** rlandy is now known as rlandy|bbl | 22:08 | |
| *** vinaykns has quit IRC | 22:18 | |
| hubbot1 | FAILING CHECK JOBS on master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001, tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades, tripleo-ci-centos-7-scenario007-multinode-oooq-container, tripleo-ci-centos-7-standalone-upgrade, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035, tripleo-ci-centos-7-scenario012-standalone, tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci- (3 more messages) | 22:27 |
| *** aakarsh has joined #oooq | 22:41 | |
| *** tosky has quit IRC | 23:27 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!