| *** rlandy has quit IRC | 00:01 | |
| *** slaweq has joined #oooq | 00:11 | |
| *** slaweq has quit IRC | 00:15 | |
| *** saneax has quit IRC | 00:25 | |
| *** dsneddon_ has quit IRC | 00:42 | |
| *** dsneddon_ has joined #oooq | 00:43 | |
| *** slaweq has joined #oooq | 02:11 | |
| *** slaweq has quit IRC | 02:16 | |
| *** rnoriega_ has joined #oooq | 02:19 | |
| *** rfolco has quit IRC | 03:12 | |
| *** ykarel has joined #oooq | 04:08 | |
| *** holser has joined #oooq | 04:10 | |
| *** dsneddon_ has quit IRC | 04:26 | |
| *** aakarsh has joined #oooq | 04:55 | |
| *** dsneddon_ has joined #oooq | 04:58 | |
| *** ykarel is now known as ykarel|away | 05:16 | |
| *** marios has joined #oooq | 05:20 | |
| *** slaweq has joined #oooq | 06:11 | |
| *** slaweq has quit IRC | 06:15 | |
| *** dtantsur|afk is now known as dtantsur | 06:28 | |
| *** ccamacho has quit IRC | 06:35 | |
| *** slaweq has joined #oooq | 06:35 | |
| *** jaosorior has quit IRC | 06:41 | |
| arxcruz|rover | ykarel|away: i'll rerun the scenario003 so we can have a promotion today, only fs020 is running, all the others passes so far | 06:48 |
|---|---|---|
| *** holser has quit IRC | 06:51 | |
| ykarel|away | arxcruz|rover, ack, hopefully stein should also promote today | 06:56 |
| ykarel|away | till now running good | 06:56 |
| *** tesseract has joined #oooq | 07:10 | |
| *** jfrancoa has joined #oooq | 07:14 | |
| *** jtomasek has joined #oooq | 07:21 | |
| *** jfrancoa has quit IRC | 07:23 | |
| *** tosky has joined #oooq | 07:24 | |
| *** jaosorior has joined #oooq | 07:24 | |
| *** ccamacho has joined #oooq | 07:26 | |
| tosky | uh, did a train promotion happen? | 07:27 |
| *** ccamacho has quit IRC | 07:27 | |
| *** ccamacho has joined #oooq | 07:27 | |
| arxcruz|rover | tosky: nope, i'll run the two failing jobs again, one was timeout, the other fails on tempest, i'm checking the reason | 07:38 |
| tosky | arxcruz|rover: but I've seen a successfull run of the scenario jobs that was failing | 07:39 |
| arxcruz|rover | tosky: waiting for the logs | 07:40 |
| *** amoralej|off is now known as amoralej | 07:42 | |
| *** jpena|off is now known as jpena | 07:47 | |
| arxcruz|rover | ykarel|away: https://review.rdoproject.org/r/#/c/23010/ fyi | 07:54 |
| ykarel|away | ack | 07:56 |
| *** holser has joined #oooq | 08:16 | |
| arxcruz|rover | fingers crossed for stein :D | 08:18 |
| arxcruz|rover | two more jobs :D | 08:18 |
| *** holser has quit IRC | 08:21 | |
| *** holser has joined #oooq | 08:26 | |
| *** jaosorior has quit IRC | 08:32 | |
| *** derekh has joined #oooq | 08:38 | |
| ykarel|away | arxcruz|rover, can u also track failure for train phase1:- https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-train-current-tripleo/ | 08:47 |
| ykarel|away | the tripleo job failure in ^^ | 08:48 |
| ykarel|away | Could not find or access '/home/jenkins/workspace/tripleo-quickstart-promote-train-current-tripleo-delorean-minimal/config/release/centosci/train-current-tripleo.yml' | 08:48 |
| ykarel|away | need a patch similar to https://review.opendev.org/#/c/650259/ | 08:49 |
| arxcruz|rover | ykarel|away: fs020 stein is running tempest now | 08:50 |
| ykarel|away | ack for stein, can u check ^^ also | 08:50 |
| arxcruz|rover | ykarel|away: working on this patch | 08:52 |
| ykarel|away | arxcruz|rover, Thanks | 08:52 |
| arxcruz|rover | ykarel|away: https://review.opendev.org/#/c/687249/ | 08:57 |
| arxcruz|rover | marios: panda ^ | 08:58 |
| arxcruz|rover | please +2 when you guys have chance :) promotion blocker | 08:58 |
| marios | arxcruz|rover: ack | 09:02 |
| arxcruz|rover | i mean, please, review first :P | 09:02 |
| ykarel|away | arxcruz|rover, which file u took reference to prepare that? | 09:03 |
| arxcruz|rover | ykarel|away: stein | 09:03 |
| ykarel|away | should use config/release/centosci/master-current-tripleo.yml as base and adjust for train | 09:03 |
| marios | arxcruz|rover: ack i couldn't spot something added comment on commit message | 09:03 |
| marios | ykarel|away: that points to master (config/release/centosci/master-current-tripleo.yml its a link) | 09:03 |
| marios | ykarel|away: arxcruz|rover arx i thought you used master at least thats what i compared it to | 09:04 |
| ykarel|away | marios, where? | 09:04 |
| ykarel|away | i don't see it as link | 09:04 |
| marios | ykarel|away: yeah sorry master is a link but master-current-tripleo isnt | 09:05 |
| ykarel|away | marios, yup, that should be used as a reference for preparing train-current-tripleo | 09:05 |
| arxcruz|rover | ykarel|away: marios give me a few minutes, i'll fix it and open a bug | 09:05 |
| ykarel|away | i noticed that when i saw reference to ceph-luminous | 09:05 |
| ykarel|away | arxcruz|rover, ack | 09:05 |
| panda | mmmh what happened to distro_ver ? | 09:08 |
| marios | ykarel|away: arxcruz|rover revoted (ykarel indeed the biggest diff is the ceph stuff) | 09:08 |
| ykarel|away | yes | 09:08 |
| arxcruz|rover | ykarel|away: marios done, bug opened, changed based on master | 09:12 |
| ykarel|away | Thanks looks good now | 09:13 |
| arxcruz|rover | ykarel|away: periodic-tripleo-ci-centos-7-scenario003-standalone-master passed, only fs020 now missing :) | 09:15 |
| arxcruz|rover | for master | 09:15 |
| arxcruz|rover | still waiting stein | 09:15 |
| ykarel|away | ack cool | 09:16 |
| *** rfolco has joined #oooq | 09:32 | |
| *** jaosorior has joined #oooq | 09:44 | |
| arxcruz|rover | ykarel|away: fs020 stein fail, i'll execute again, once i check what fails there | 09:55 |
| ykarel|away | arxcruz|rover, that test(test_delete_saving_image) fails randomly | 10:07 |
| arxcruz|rover | ykarel|away: yeah, i'm rerunning | 10:08 |
| ykarel|away | ack | 10:08 |
| *** ykarel|away has quit IRC | 10:09 | |
| *** slaweq has quit IRC | 10:09 | |
| *** slaweq_ has joined #oooq | 10:09 | |
| arxcruz|rover | yolanda: when weshay|ruck we can decide if we can skip this job to get the promotion also | 10:10 |
| arxcruz|rover | yolanda: sorry, wrong person :) | 10:11 |
| *** jaosorior has quit IRC | 10:18 | |
| *** amoralej is now known as amoralej|lunch | 11:12 | |
| *** slaweq_ is now known as slaweq | 11:15 | |
| *** chem has quit IRC | 11:16 | |
| *** chem has joined #oooq | 11:19 | |
| *** jpena is now known as jpena|lunch | 11:41 | |
| weshay|ruck | arxcruz|rover, howdy | 11:53 |
| arxcruz|rover | weshay|ruck: hey boss | 11:53 |
| arxcruz|rover | want to sync? | 11:53 |
| weshay|ruck | sure.. I have 6min | 11:54 |
| arxcruz|rover | lol | 11:54 |
| weshay|ruck | arxcruz|rover, https://meet.google.com/gfz-ybik-uik | 11:54 |
| arxcruz|rover | bt or meet? | 11:54 |
| weshay|ruck | arxcruz|rover, https://review.rdoproject.org/r/#/c/21672/ | 11:59 |
| weshay|ruck | arxcruz|rover, master is promoting | 12:02 |
| weshay|ruck | promoter Running: env ANSIBLE_LOG_PATH=/home/centos/promoter_logs/container-push/20191008-022432.log RELEASE=master COMMIT_HASH=12a897fb9b218be970090996da0e21a82cedda1a DISTRO_HASH=aba8ec542777d218da0a947f2fa3a801fde3696b FULL_HASH=12a897fb9b218be970090996da0e21a82cedda1a_aba8ec54 PROMOTE_NAME=current-tripleo SCRIPT_ROOT=/home/centos/ci-config/ DISTRO_NAME=rhel DISTRO_VERSION=8 ansible-playbook /home/centos/ci-config/ci-scripts/container-p | 12:03 |
| weshay|ruck | ush/container-push.yml | 12:03 |
| weshay|ruck | arxcruz|rover, hrm.. maybe that didn't happen.. I'll make it happen | 12:08 |
| *** jaosorior has joined #oooq | 12:08 | |
| panda | rfolco: https://review.rdoproject.org/r/22994 seesm to be stable, the last 18 job runs passed the point with the "connection reset" problem | 12:14 |
| weshay|ruck | 2019-10-08 12:13:45,458 12955 INFO promoter Running: env ANSIBLE_LOG_PATH=/home/centos/promoter_logs/container-push/20191008-121345.log RELEASE=master COMMIT_HASH=12a897fb9b218be970090996da0e21a82cedda1a DISTRO_HASH=aba8ec542777d218da0a947f2fa3a801fde3696b FULL_HASH=12a897fb9b218be970090996da0e21a82cedda1a_aba8ec54 PROMOTE_NAME=current-tripleo SCRIPT_ROOT=/home/centos/ci-config/ DISTRO_NAME=centos DISTRO_VERSION=7 ansible-playbook | 12:14 |
| weshay|ruck | /home/centos/ci-config/ci-scripts/container-push/container-push.yml | 12:14 |
| weshay|ruck | arxcruz|rover, this is promoting ^ | 12:14 |
| rfolco | panda, I still have to adapt the playbook to run as include_from... probably need to move vars to defaults and leave only the tasks directly there... | 12:26 |
| rfolco | 2019-10-08 09:53:49.090857 | TASK [Run tripleo-common scenario tests] | 12:26 |
| rfolco | 2019-10-08 09:53:49.140313 | rdo-centos-7 | ok | 12:26 |
| rfolco | panda, it doesn't seem to be running | 12:27 |
| *** amoralej|lunch is now known as amoralej | 12:32 | |
| panda | rfolco: want to sync ? | 12:34 |
| rfolco | panda, yes | 12:34 |
| rfolco | panda, https://meet.google.com/bqx-xwht-wky | 12:35 |
| *** jpena|lunch is now known as jpena | 12:37 | |
| *** rlandy has joined #oooq | 12:39 | |
| weshay|ruck | rfolco, panda couple things to review for promoter https://review.rdoproject.org/r/#/c/22998/ https://review.rdoproject.org/r/#/c/22892/ | 12:40 |
| weshay|ruck | rfolco, panda btw.. IMHO let's cap the retrospective at 1.5 hours max.. spend the rest of the time w/ promoter team standing up a new node | 12:41 |
| rlandy | panda: hi | 12:41 |
| weshay|ruck | panda, is about an hour to stand up a new promoter and see where we are at about right? | 12:41 |
| *** chem has quit IRC | 12:41 | |
| rlandy | panda: tests are passing other than the 'failed attempt' test that is failing in all three | 12:41 |
| rlandy | marios: are we sync'ing today? | 12:42 |
| rlandy | panda: ^^ that failed promotion never shows up in the logs | 12:42 |
| rlandy | need to check that with you | 12:42 |
| rlandy | other than that the tests pass | 12:42 |
| marios | rlandy: 15:35 < rfolco> panda, https://meet.google.com/bqx-xwht-wky | 12:43 |
| *** chem has joined #oooq | 12:43 | |
| rfolco | rlandy, come to the party | 12:43 |
| marios | rlandy: impromptu i crashed their party and now i'm inviting friends | 12:43 |
| marios | i'm _that_ guy | 12:44 |
| panda | rlandy: https://review.rdoproject.org/r/22994 seesm to be stable, the last 18 job runs passed the point with the "connection reset" problem | 12:50 |
| weshay|ruck | panda, rfolco re: retrospective day.. is that a reasonable request? | 12:52 |
| rfolco | panda, you'll have to limit retro cards to 1 or 2 max /person | 12:52 |
| rfolco | weshay|ruck, ^ | 12:52 |
| weshay|ruck | rfolco, essentially I would like to skip the card / board review | 12:53 |
| weshay|ruck | in favor of the promoter stand up | 12:53 |
| rfolco | weshay|ruck, lgtm | 12:54 |
| arxcruz|rover | weshay|ruck: fs020 master pass | 12:56 |
| weshay|ruck | arxcruz|rover, rock on | 12:56 |
| weshay|ruck | panda, ? | 12:59 |
| weshay|ruck | panda, if it fails... that's fine.. if it passes that's great.. need you to respond | 12:59 |
| *** matbu has quit IRC | 12:59 | |
| *** matbu has joined #oooq | 13:00 | |
| arxcruz|rover | weshay|ruck: master should be promoted now | 13:11 |
| panda | weshay|ruck: yes, I'm preparing some patches to make it work, I have a promoter server running, and I'm checking the things that I can | 13:12 |
| weshay|ruck | arxcruz|rover, ya.. it's uploading containers | 13:14 |
| arxcruz|rover | weshay|ruck: ok, cool | 13:14 |
| *** aakarsh has quit IRC | 13:14 | |
| arxcruz|rover | weshay|ruck: will you skip fs020 on stein, or let it go ? | 13:14 |
| weshay|ruck | panda, ok.. sounds perfect.. I think we can just include anyone who worked on it... but anyone is welcome | 13:14 |
| weshay|ruck | rfolco, please adjust the retro cal invite to 1.5 hr max, and send an additional invite to the promoter time | 13:14 |
| rfolco | weshay|ruck, additional time = remaining (1h) or extra ? | 13:15 |
| weshay|ruck | rfolco, I have a potential conflict.. let's start promoter stand up at 3pm utc | 13:16 |
| weshay|ruck | let's skip board review | 13:17 |
| rfolco | weshay|ruck, retro starts 1pm UTC, we cap it at 1.5h, so 2:30pm UTC we can start promoter standup, isn't it ? | 13:19 |
| weshay|ruck | rfolco, 3pm.. I have prod chain council 2-3 | 13:21 |
| rfolco | weshay|ruck, we'd have 2 hours for retro then | 13:22 |
| weshay|ruck | rfolco, /me notes.. keep folks on for 3 straight hours.. may be rough | 13:23 |
| weshay|ruck | rfolco, ur the tc though, up to u | 13:23 |
| rfolco | weshay|ruck, ok just confirming what you want to do.... I'll shift retro 30min or 1h then | 13:23 |
| weshay|ruck | rfolco, let's retro for 1 - 1.5 hours.. break.. and pickup the install at 3pm imho | 13:29 |
| rfolco | weshay|ruck, ok just did that, invite sent.... will send invite for promoter stand up next | 13:30 |
| rfolco | w/ same gmeet | 13:30 |
| rfolco | community call starts now at https://meet.google.com/bqx-xwht-wky | 13:31 |
| rfolco | ping marios, sshnaidm, weshay, panda, rlandy, arxcruz, rfolco, chandankumar, zbr, kopecmartin | 13:31 |
| rfolco | ci community call ^ | 13:31 |
| weshay|ruck | on a call.. myself | 13:33 |
| *** chem has quit IRC | 13:34 | |
| *** chem has joined #oooq | 13:36 | |
| weshay|ruck | arxcruz|rover, fyi.. this is how we update the status board fyi https://code.engineering.redhat.com/gerrit/182922 | 13:44 |
| weshay|ruck | arxcruz|rover, can you make sure scen010 gets into the master / train pipelines and critieria https://review.rdoproject.org/r/#/c/22867/2/zuul.d/standalone-jobs.yaml | 13:49 |
| arxcruz|rover | weshay|ruck: sure, submiting the patch | 13:50 |
| *** Vorrtex has joined #oooq | 13:50 | |
| arxcruz|rover | weshay|ruck: https://review.rdoproject.org/r/#/c/23019/ | 13:56 |
| *** ykarel|awat has joined #oooq | 13:56 | |
| *** ykarel|awat is now known as ykarel|away | 13:56 | |
| weshay|ruck | arxcruz|rover, thanks | 13:57 |
| rlandy | panda: still a no show on the failed attempt - see logs ... https://review.rdoproject.org/r/#/c/22958/ | 13:59 |
| rlandy | no trace of the 'skipping' in promotion log | 13:59 |
| panda | rlandy: I see now. TO have the skippin g essage we would need to do something different | 14:03 |
| rlandy | panda: even the py27 does not work | 14:04 |
| rlandy | panda: I am going to comment out the failed_attempt check to see that everything else works | 14:04 |
| weshay|ruck | rfolco, you guys still on a call? | 14:04 |
| rfolco | weshay|ruck, no we dropped. | 14:04 |
| rfolco | weshay|ruck, we has updates from mikhal and discussed timing w/ ppc folks | 14:05 |
| rfolco | had* | 14:05 |
| panda | rlandy: the py27 fails even before, if nothing matches that a search returns None, and you're not handling that case | 14:06 |
| *** aakarsh has joined #oooq | 14:06 | |
| rlandy | panda: it should match | 14:06 |
| rlandy | I guess in the py27, is it checking all hashes? | 14:07 |
| panda | rlandy: but to make the Skipping message show, we would need to run the promotion twice, the sequence would be: 1) inject fixtures with a missing vote, 2) run promoter, 3) inject a fake vote to dlrnapi, 4) rerun promotion. At the end the log should have a "Skipping" | 14:07 |
| weshay|ruck | rfolco, just the folks who participated in the promoter work | 14:08 |
| weshay|ruck | not the whole team | 14:08 |
| rlandy | panda: I think we should get the successful promotion check working first | 14:08 |
| rlandy | panda: wrt py27 test, | 14:08 |
| rlandy | let's talk about what's going on there ... | 14:08 |
| rfolco | weshay|ruck, ok, I thought you would like to disseminate knowledge | 14:08 |
| rfolco | weshay|ruck, optional is ok? | 14:09 |
| rlandy | panda: "if nothing matches that a search returns None" | 14:09 |
| weshay|ruck | aye | 14:09 |
| *** aakarsh|2 has joined #oooq | 14:10 | |
| rlandy | ^^ it should only check if the hashes match the promotion_candidate or the failed_attempt | 14:10 |
| *** chem has quit IRC | 14:11 | |
| *** aakarsh has quit IRC | 14:13 | |
| *** chem has joined #oooq | 14:14 | |
| *** ykarel|away is now known as ykarel | 14:15 | |
| weshay|ruck | arxcruz|rover, when you have a sec.. spot check this http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&fullscreen&panelId=9 | 14:16 |
| weshay|ruck | we still have too many 0ns | 14:16 |
| rlandy | panda: I need to take a few minutes of your time to finish this up ... per last results in https://review.rdoproject.org/r/#/c/22958/ | 14:19 |
| weshay|ruck | arxcruz|rover, master promoted, stein starting now | 14:21 |
| arxcruz|rover | weshay|ruck: it's to stand up and glorify! | 14:24 |
| arxcruz|rover | right rfolco | 14:25 |
| rlandy | panda: no where does this file : http://logs.rdoproject.org/58/22958/45/check/tripleo-ci-promotion-staging/cdea9f6/logs/stage-info.yaml have faied_attempt defined | 14:26 |
| rlandy | failed_attempt | 14:26 |
| panda | rlandy: ready to chat | 14:29 |
| rlandy | panda: https://meet.google.com/dqh-ordn-wiw | 14:31 |
| *** chem has quit IRC | 14:36 | |
| *** chem has joined #oooq | 14:38 | |
| rfolco | arxcruz|rover, :) | 14:52 |
| arxcruz|rover | weshay|ruck: 2019-10-08 14:49:59,101 24426 ERROR promoter Command '[u'env', u'ANSIBLE_LOG_PATH=/home/centos/promoter_logs/container-push/20191008-142114.log', u'RELEASE=stein', u'COMMIT_HASH=355abb693dbb43fa429939e494a33e362075f1f8', u'DISTRO_HASH=647b08e47a72f7533142c09074f227628f08f9fa', u'FULL_HASH=355abb693dbb43fa429939e494a33e362075f1f8_647b08e4', u'PROMOTE_NAME=current-tripleo', | 14:55 |
| arxcruz|rover | u'SCRIPT_ROOT=/home/centos/ci-config/', u'DISTRO_NAME=centos', u'DISTRO_VERSION=7', u'ansible-playbook', u'/home/centos/ci-config/ci-scripts/container-push/container-push.yml']' returned non-zero exit status 2 | 14:55 |
| arxcruz|rover | fail for stein | 14:55 |
| weshay|ruck | arxcruz|rover, ya.. one failed.. not uncommon, /me rekicks | 14:57 |
| *** aakarsh|2 has quit IRC | 15:01 | |
| *** aakarsh|2 has joined #oooq | 15:01 | |
| *** ykarel is now known as ykarel|away | 15:01 | |
| rfolco | panda, "Could not find or access '/tmp/stage-info.yaml' | 15:04 |
| rfolco | rlandy, ^ | 15:04 |
| rfolco | that file should exist, right? | 15:04 |
| rlandy | ack - see the logs on my patch - fixed | 15:04 |
| rlandy | rfolco: ^^ | 15:05 |
| rfolco | rlandy, did you rebase on panda's patch for unicorn ? | 15:05 |
| rlandy | yes | 15:05 |
| rfolco | so I can rebase on yours then | 15:05 |
| rfolco | ok | 15:05 |
| rlandy | rfolco: wait | 15:05 |
| rfolco | thx | 15:05 |
| rfolco | what? | 15:05 |
| rlandy | just spoke to panda - removing the one negative test | 15:05 |
| rlandy | putting in a new patch that should pass | 15:06 |
| rlandy | give me a few | 15:06 |
| rfolco | rlandy, ok I am in tc mtg, pls ping when its ready to rebase on | 15:06 |
| weshay|ruck | arxcruz|rover, we need to chat w/ the infra folks about the rhel 8 images being used.. selinux should be permissive out of the image https://bugs.launchpad.net/tripleo/+bug/1847282 | 15:07 |
| openstack | Launchpad bug 1847282 in tripleo "rhel 8 tripleo Destination directory /etc/modules-load.d does not exist" [Critical,In progress] | 15:07 |
| arxcruz|rover | weshay|ruck: ack | 15:12 |
| weshay|ruck | arxcruz|rover, https://review.opendev.org/687330 | 15:20 |
| arxcruz|rover | weshay|ruck: but it should be on the image right ? | 15:20 |
| weshay|ruck | arxcruz|rover, /me was wondering how other teams would use that.. should we check the distribution or if the command exists? | 15:20 |
| weshay|ruck | or let ignore_errors handle it.. | 15:20 |
| arxcruz|rover | weshay|ruck: ignore_errors is better | 15:21 |
| arxcruz|rover | less checking | 15:21 |
| arxcruz|rover | if it fails means, there's no selinux, the collect logs should not fail because of that right ? | 15:21 |
| weshay|ruck | arxcruz|rover, ya.. the collect logs playbook should be ignore_errors | 15:23 |
| weshay|ruck | arxcruz|rover, the playbook is not in the role though.. so another team could pick up the role.. and have a ton of failures | 15:24 |
| weshay|ruck | arxcruz|rover, hrm... https://github.com/openstack/tripleo-quickstart-extras/blob/master/playbooks/collect-logs.yml | 15:26 |
| weshay|ruck | arxcruz|rover, what the duece https://github.com/openstack/tripleo-quickstart-extras/blob/master/playbooks/collect-logs.yml#L22 | 15:26 |
| arxcruz|rover | weshay|ruck: workigng on it | 15:28 |
| weshay|ruck | http://paste.openstack.org/show/782108/ | 15:28 |
| weshay|ruck | PLAY RECAP ********************************************************************* | 15:29 |
| weshay|ruck | localhost : ok=41 changed=32 unreachable=0 failed=0 skipped=78 rescued=0 ignored=7 | 15:29 |
| weshay|ruck | overcloud-controller-0 : ok=59 changed=55 unreachable=0 failed=0 skipped=42 rescued=0 ignored=9 | 15:29 |
| weshay|ruck | overcloud-controller-1 : ok=59 changed=55 unreachable=0 failed=0 skipped=42 rescued=0 ignored=9 | 15:29 |
| weshay|ruck | overcloud-controller-2 : ok=59 changed=55 unreachable=0 failed=0 skipped=42 rescued=0 ignored=9 | 15:29 |
| weshay|ruck | overcloud-novacompute-0 : ok=58 changed=54 unreachable=0 failed=0 skipped=43 rescued=0 ignored=8 | 15:29 |
| weshay|ruck | undercloud : ok=70 changed=57 unreachable=0 failed=0 skipped=57 rescued=0 ignored=7 | 15:29 |
| weshay|ruck | anyone know where ignore_errors is set on collect logs these days? | 15:29 |
| arxcruz|rover | weshay|ruck: should be in all include tasks no ? | 15:30 |
| weshay|ruck | arxcruz|rover, ah.. https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/tasks/collect.yml#L3 | 15:30 |
| weshay|ruck | phew | 15:30 |
| arxcruz|rover | weshay|ruck: hmmm, it's not in the whole file, just https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/tasks/collect.yml#L3-L20 | 15:31 |
| arxcruz|rover | weshay|ruck: we might need to add ignore in the includes as well | 15:31 |
| weshay|ruck | arxcruz|rover, https://review.opendev.org/687335 | 15:35 |
| arxcruz|rover | or add a note :) | 15:35 |
| weshay|ruck | arxcruz|rover, https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/tasks/collect/system.yml#L3 | 15:36 |
| weshay|ruck | https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/tasks/collect/monitoring.yml#L3 | 15:36 |
| weshay|ruck | arxcruz|rover, you wrote this bit didn't you? | 15:36 |
| weshay|ruck | lolz | 15:36 |
| weshay|ruck | memory SHOT! | 15:36 |
| arxcruz|rover | weshay|ruck: that's what I was wondering now | 15:36 |
| arxcruz|rover | lol | 15:36 |
| weshay|ruck | arxcruz|rover, ok. next question... where is the node defs.. for the rhel8 ovb stack | 15:44 |
| weshay|ruck | arxcruz|rover, do you know? | 15:44 |
| weshay|ruck | or the dib | 15:44 |
| arxcruz|rover | nope | 15:44 |
| arxcruz|rover | i'm very low on rhel knowledge | 15:45 |
| arxcruz|rover | rfolco: ? ^ | 15:45 |
| weshay|ruck | arxcruz|rover, we may need to update our image build scripts | 15:46 |
| weshay|ruck | I think that is the issue actually | 15:47 |
| *** kopecmartin is now known as kopecmartin|off | 15:47 | |
| weshay|ruck | arxcruz|rover, https://github.com/openstack/tripleo-ci/blob/master/roles/oooci-build-images/templates/build-images.sh.j2 | 15:48 |
| weshay|ruck | arxcruz|rover, we need to add a dib element to disable selinux some where around https://github.com/openstack/tripleo-ci/blob/master/roles/oooci-build-images/tasks/main.yaml#L24 | 15:49 |
| arxcruz|rover | weshay|ruck: i don't think so, because the overcloud build image is called to create the image, the line 24 already have the image created | 15:51 |
| weshay|ruck | arxcruz|rover, ya.. we build the images in the promotion pipeline you gooose | 15:52 |
| weshay|ruck | :) | 15:52 |
| weshay|ruck | it uses those scripts | 15:52 |
| arxcruz|rover | weshay|ruck: yes, i understand | 15:52 |
| arxcruz|rover | but thhe build-images.sh just call the overcloud build image | 15:53 |
| weshay|ruck | https://meet.google.com/jtp-kxij-guy | 15:53 |
| weshay|ruck | arxcruz|rover, we need to add https://docs.openstack.org/diskimage-builder/2.6.1/elements/selinux-permissive/README.html | 15:54 |
| *** marios has quit IRC | 15:57 | |
| weshay|ruck | arxcruz|rover, diskimage-builder/diskimage_builder/elements/selinux-permissive | 15:58 |
| arxcruz|rover | weshay|ruck: https://github.com/openstack/tripleo-ci/blob/fedc702d84b8fb7ba7c7b2cc8b77f44f1363b537/roles/oooci-build-images/templates/pathfix_repos.sh.j2 | 15:59 |
| weshay|ruck | https://github.com/openstack/tripleo-ci/blob/master/roles/oooci-build-images/templates/build-images.sh.j2#L7 | 16:00 |
| arxcruz|rover | weshay|ruck: https://github.com/openstack/tripleo-ci/blob/master/roles/oooci-build-images/tasks/main.yaml#L24 | 16:01 |
| *** dtantsur is now known as dtantsur|afk | 16:03 | |
| *** tesseract has quit IRC | 16:08 | |
| weshay|ruck | openstack overcloud image build --type overcloud-full \ | 16:09 |
| weshay|ruck | --builder-extra-args selinux-permissive | 16:09 |
| weshay|ruck | arxcruz|rover, ^ | 16:09 |
| arxcruz|rover | weshay|ruck: rhel_image_source | 16:10 |
| arxcruz|rover | weshay|ruck: https://review.opendev.org/#/c/687345/ | 16:10 |
| weshay|ruck | arxcruz|rover, related-bug: https://bugs.launchpad.net/tripleo/+bug/1847282 | 16:20 |
| openstack | Launchpad bug 1847282 in tripleo "rhel 8 tripleo Destination directory /etc/modules-load.d does not exist" [Critical,In progress] | 16:20 |
| rfolco | rlandy, panda do you remove stage-info somewhere after your tests run ? | 16:26 |
| panda | rfolco: not anymore | 16:26 |
| panda | rfolco: now it's left for collection | 16:26 |
| panda | rfolco: correction | 16:26 |
| panda | rfolco: we copy it before it's removed | 16:27 |
| rfolco | panda, Could not find or access '/tmp/stage-info.yaml' | 16:27 |
| rlandy | panda: rfolco: fingers crossed that the current patch will work and we can merge | 16:27 |
| rlandy | that will get you the stage-info you need | 16:27 |
| panda | rlandy: logs ? | 16:27 |
| rlandy | getting | 16:28 |
| panda | rlandy: sorry, I meant rfolco | 16:29 |
| rlandy | lol | 16:29 |
| rfolco | http://logs.rdoproject.org/88/22988/14/check/tripleo-ci-promotion-staging/2b4b115/job-output.txt | 16:29 |
| *** rfolco is now known as not_rlandy | 16:29 | |
| *** not_rlandy is now known as rfolco | 16:29 | |
| *** bogdando has quit IRC | 16:29 | |
| rfolco | panda, the idea was to use the playbook exactly like in the molecule test, without any change in the tasks so i can just include tasks_from | 16:31 |
| panda | rfolco: there should be nothing there that deletes /tmp/stage-info.yaml | 16:33 |
| rlandy | panda: rfolco: https://review.rdoproject.org/r/#/c/22958/ - we're green | 16:34 |
| panda | rfolco: "message": "Could not find or access '/tmp/stage-info.yaml' on the Ansible Controller | 16:34 |
| panda | rlandy: you're trying to include it in localhost ? | 16:34 |
| panda | rlandy: \o/ | 16:34 |
| rfolco | hahaha | 16:34 |
| rfolco | lol | 16:35 |
| rfolco | no, I'm not | 16:35 |
| panda | rlandy: the images test ir RED | 16:35 |
| rfolco | rdo-centos-7 | 16:35 |
| rfolco | panda, ^ | 16:35 |
| rfolco | rlandy, looks like marios test is broken now | 16:36 |
| rlandy | panda: that wasn't red before? | 16:38 |
| panda | rlandy: I suggest you remove any reference to failed_attempt | 16:38 |
| rlandy | looking | 16:38 |
| panda | rlandy: it's probably confusing the test, and we are not going to use it anyway. | 16:39 |
| rlandy | fixing | 16:39 |
| panda | rlandy: sorry, I sneaked thos jobs in again, they were not running before, but I was sorried womthing like this would happen. | 16:41 |
| panda | the stagin environment affects every other jobs , so they should run | 16:42 |
| rlandy | panda: np - updated | 16:43 |
| panda | rlandy: rfolco the real-life scenario works at least to the point when we run the promoter itself. I'm not sure how I can test further without interfering with the other promoter | 16:44 |
| rlandy | panda: one thing at a time | 16:44 |
| panda | rfolco: really weird | 16:49 |
| rfolco | panda, do I need to include remote_src ? | 16:49 |
| rfolco | [gather stage info] step above just works | 16:49 |
| rfolco | why include_vars doesn't ? | 16:50 |
| rfolco | panda, will test w/ remote_src: yes | 16:50 |
| panda | rfolco: include_vars takes vars only from the ansible controller | 16:51 |
| *** derekh has quit IRC | 16:52 | |
| panda | rfolco: it's wokring with the other test becaus they use localhost | 16:52 |
| rfolco | panda, ok I am officially declaring war against de-duplication efforts. I'll duplicate the whole thing. | 16:54 |
| rfolco | to get things done | 16:55 |
| rfolco | molecule is molecule, zuul is zuul. period. | 16:56 |
| panda | rfolco: if you copy the file locally, you should be fine | 17:01 |
| panda | rfolco: no wait, it's already done, I copy it in /home/{{ promoter_user }} | 17:01 |
| rlandy | panda: so marios test is still failing ... the only relevant change left is https://review.rdoproject.org/r/#/c/22958/54/ci-scripts/dlrnapi_promoter/tests/staging-setup/fixtures/scenario-1.yaml | 17:03 |
| rlandy | which is required | 17:03 |
| rlandy | assertion: previous_current_tripleo.stat.islnk is defined | 17:04 |
| rlandy | correct it's not | 17:04 |
| rlandy | would need to change his test | 17:04 |
| panda | rlandy: required ? rlandy https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/promoter/molecule/promote-images/playbook.yml#L42 | 17:07 |
| panda | it's checkign for current-tripleo | 17:08 |
| rlandy | well that was what was defined before | 17:09 |
| rlandy | the next lime is dependent on that | 17:10 |
| rfolco | if remote_src does not work, will get it from promoter_user home dir, got it panda | 17:11 |
| panda | rlandy: it should check tripleo-ci-staging-promoted | 17:15 |
| rlandy | panda: I'll update - let's see | 17:17 |
| arxcruz|rover | weshay|ruck: am i missing something here? https://review.rdoproject.org/r/#/c/22800/1/zuul.d/tripleo.yaml | 17:22 |
| arxcruz|rover | the patch was merged | 17:23 |
| arxcruz|rover | weshay|ruck: do you want the ovb job ? | 17:27 |
| weshay|ruck | arxcruz|rover, ya.. don't confuse scenario 001 w/ fs001 | 17:28 |
| weshay|ruck | arxcruz|rover, we need the ovb job | 17:28 |
| weshay|ruck | that's the only place where selinux is getting us atm | 17:28 |
| *** ccamacho has quit IRC | 17:29 | |
| weshay|ruck | arxcruz|rover, before you sign off for the day.. I'm not clear what the status of https://trello.com/c/DvgaBFim/1104-cixlp1843259tripleociproa-periodic-rocky-fs020-job-fails-tempest-tests-tempestscenariotestsecuritygroupsbasicopstestsecuritygrou is | 17:29 |
| weshay|ruck | please update | 17:29 |
| arxcruz|rover | weshay|ruck: same as yesterday, the patch was merged to skip the test | 17:29 |
| arxcruz|rover | we had a promotion | 17:29 |
| arxcruz|rover | the root cause isn't fixed yet | 17:29 |
| *** Vorrtex has quit IRC | 17:32 | |
| *** jpena is now known as jpena|off | 17:32 | |
| weshay|ruck | arxcruz|rover, perfect.. thanks | 17:33 |
| arxcruz|rover | weshay|ruck: https://review.rdoproject.org/r/#/c/23022/ | 17:34 |
| weshay|ruck | arxcruz|rover, I think that is triggering because of https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleo.yaml#L546 | 17:38 |
| weshay|ruck | or https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleo.yaml#L545 | 17:38 |
| weshay|ruck | arxcruz|rover, tbh.. not 100% sure | 17:39 |
| weshay|ruck | arxcruz|rover, merged yours | 17:39 |
| arxcruz|rover | ok | 17:39 |
| *** amoralej is now known as amoralej|off | 17:47 | |
| *** chem is now known as chem|eod | 17:49 | |
| rlandy | panda: rfolco: https://review.rdoproject.org/r/#/c/22958/ - legit green now | 17:53 |
| panda | rlandy: SHOP IT! | 17:53 |
| panda | ooopp | 17:54 |
| panda | ahah | 17:54 |
| rlandy | I'm always u for shopping | 17:54 |
| rlandy | panda: I need to merge the patches underneath that | 17:55 |
| rlandy | panda: ok to merge https://review.rdoproject.org/r/#/c/22994/ | 17:56 |
| panda | rlandy: yep | 17:56 |
| rlandy | going | 17:56 |
| panda | rfolco: sorry for the wrong suggestion | 17:57 |
| panda | rfolco: /home/zuul/stage-info is still on the host, not the executor | 17:57 |
| rfolco | panda, include_vars also does not have remote_src looks like | 17:57 |
| rfolco | panda, hopefullt delegate_to works | 18:01 |
| panda | rfolco: it will not, the easiest way at this point is to fetch the file with the fetch module, then use include_vars on the fetched file | 18:03 |
| rfolco | panda, ok thx for the suggestion | 18:04 |
| rlandy | oh come on gate | 18:05 |
| panda | rfolco: if that doesn't work, the only other wasy is to cat the file from shell and register the output , then set_fact: stage_info: {{ registered_var | from_yaml }} but the vairables will all be under stage_info | 18:07 |
| rfolco | panda, ack | 18:07 |
| rlandy | panda: thanks for w+'ing patch - updating cards now with negative test cases required | 18:18 |
| *** holser has quit IRC | 18:19 | |
| rlandy | rfolco: panda: put all the promoter test cards in 'QE' column except https://tree.taiga.io/project/tripleo-ci-board/task/1284?kanban-status=1447275 | 18:24 |
| rlandy | ^^ adding negative test requirements | 18:24 |
| rlandy | we may want to complete that card and move the requirements to another card on the next sprint | 18:24 |
| rfolco | panda, "msg": "Accessing files from outside the working dir /var/opt/rh/rh-python35/lib/zuul/builds/139b330387624563be7052f1ba44da31/work is prohibited", | 18:27 |
| rfolco | panda, finding a solution | 18:28 |
| panda | rfolco: that dir is referenced in zuul.work_dir variable | 18:28 |
| rfolco | ok will give it a try | 18:28 |
| panda | rfolco: sorry, zuul.executor.work_dir | 18:29 |
| rfolco | panda, this? | 18:30 |
| rfolco | args: | 18:30 |
| rfolco | chdir: "{{ zuul.executor.work_dir }}" | 18:30 |
| panda | rfolco: yes | 18:30 |
| rfolco | thx | 18:30 |
| *** ksambor has joined #oooq | 18:56 | |
| *** ksambor has quit IRC | 18:56 | |
| *** ykarel|away has quit IRC | 19:00 | |
| mjturek | does anyone know what this file is? https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/ci-scripts/tripleo-upstream/dlrnapi_venv.sh#L3 | 19:01 |
| mjturek | the dlrnapi_venv | 19:02 |
| weshay|ruck | mjturek, dlrnapi creds | 19:03 |
| weshay|ruck | rfolco, ^ | 19:03 |
| mjturek | weshay|ruck: can we assume that they are in the cico nodes? | 19:03 |
| * mjturek is tracing through get-hash.sh in an attempt to catch any gotchas | 19:04 | |
| rfolco | mjturek, your jenkins job should be able to load env vars.... DLRNAPI_PASSWORD for ex | 19:04 |
| weshay|ruck | rlandy, when you have a moment https://review.opendev.org/#/c/687361/ | 19:05 |
| mjturek | yep, but this script seems to reference a venv file that I'm not positive exists in cico | 19:06 |
| mjturek | oh shoot nvm | 19:08 |
| mjturek | I'm dumb. | 19:08 |
| rfolco | hmm that one is venv dir | 19:08 |
| mjturek | it's just creating a venv | 19:08 |
| rfolco | yes | 19:08 |
| mjturek | sorry rfolco should be fine lol | 19:09 |
| rfolco | my answer was not right, I thought you were asking for shell env vars like dlrnapi_password | 19:09 |
| rfolco | mjturek, np | 19:09 |
| *** holser has joined #oooq | 19:34 | |
| weshay|ruck | rfolco, rlandy https://review.opendev.org/#/c/687361/ | 19:36 |
| weshay|ruck | thanks! | 19:36 |
| rlandy | weshay|ruck: anything else while I am here? | 19:37 |
| weshay|ruck | rlandy, no get out of here.. have an easy fast :) | 19:38 |
| rlandy | weshay|ruck: not yet :) | 19:38 |
| weshay|ruck | oh maybe one more then https://review.opendev.org/#/c/687330/ | 19:40 |
| weshay|ruck | rfolco, ^ | 19:40 |
| weshay|ruck | https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_cb8/687330/2/check/tripleo-ci-centos-7-containers-multinode/cb83085/logs/undercloud/var/log/extra/selinux.txt.gz | 19:40 |
| weshay|ruck | http://logs.rdoproject.org/30/687330/2/openstack-check/tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset001/4a52dff/logs/overcloud-controller-0/var/log/extra/selinux.txt.gz | 19:41 |
| weshay|ruck | oh dang.. my follow up patch was never sent | 19:41 |
| weshay|ruck | that's ok | 19:41 |
| *** jtomasek has quit IRC | 19:58 | |
| rlandy | weshay|ruck: did ovb jobs hit this wget problem? https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/periodic-tripleo-ci-centos-7-bm_envA-3ctlr_1comp-featureset001-master/ba65b5c/logs/undercloud/home/zuul/overcloud_image_build.log.txt.gz | 20:03 |
| rlandy | 2019-10-08 08:15:55 | /home/zuul/overcloud_image_build_script.sh: line 20: wget: command not found | 20:03 |
| * weshay|ruck looks | 20:04 | |
| rlandy | fixing bm | 20:04 |
| rlandy | doesn't look like it | 20:05 |
| weshay|ruck | rlandy, we're running the image build job internally? | 20:05 |
| rlandy | bm is | 20:05 |
| weshay|ruck | wget looks like a legit problem | 20:05 |
| rlandy | apprently | 20:06 |
| weshay|ruck | rlandy, I thought bm built the image like ovb? | 20:06 |
| rlandy | idk how no other job hits this | 20:06 |
| rlandy | weshay|ruck: it should | 20:06 |
| rlandy | unless bm is missing a setting | 20:06 |
| weshay|ruck | oh.. sorry.. that IS a bm job.. | 20:07 |
| * weshay|ruck keeps looking | 20:07 | |
| weshay|ruck | rlandy, we'd be smart to change that to curl | 20:07 |
| rlandy | weshay|ruck: ack I just want to know why bm is the only one afflicted | 20:08 |
| rlandy | doesn't make sense | 20:08 |
| weshay|ruck | rlandy, wget is not installed https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/periodic-tripleo-ci-centos-7-bm_envA-3ctlr_1comp-featureset001-master/ba65b5c/logs/undercloud/var/log/extra/rpm-list.txt.gz | 20:09 |
| weshay|ruck | so that's a legit fail | 20:09 |
| weshay|ruck | ya.. rlandy maybe nodepool installs wget | 20:09 |
| rlandy | https://github.com/openstack/tripleo-quickstart-extras/blame/master/roles/build-images/templates/overcloud-image-build.sh.j2 | 20:09 |
| * weshay|ruck checks | 20:09 | |
| rlandy | that code is 5 months old | 20:09 |
| rlandy | so nothing new there | 20:09 |
| weshay|ruck | rlandy, http://logs.rdoproject.org/openstack-regular/opendev.org/openstack/tripleo-ci/master/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/b2b2a5d/logs/undercloud/var/log/extra/rpm-list.txt.gz | 20:10 |
| weshay|ruck | rlandy, so it's installed on the undercloud by default | 20:10 |
| rlandy | wget-1.14-18.el7_6.1.x86_64 | 20:10 |
| weshay|ruck | http://logs.rdoproject.org/openstack-regular/opendev.org/openstack/tripleo-ci/master/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/b2b2a5d/logs/undercloud/var/log/yum.log.txt.gz | 20:10 |
| rlandy | weird | 20:11 |
| weshay|ruck | and it's not installed by us ^ | 20:11 |
| weshay|ruck | ya.. weird | 20:11 |
| weshay|ruck | let's switch it to curl | 20:11 |
| rlandy | I saw it once before | 20:11 |
| rlandy | but now it's consistent | 20:11 |
| weshay|ruck | rlandy, it could be another package had wget a dep at some point | 20:11 |
| weshay|ruck | and that was removed | 20:11 |
| weshay|ruck | is it just master | 20:11 |
| rlandy | weshay|ruck: request to spent some time downstream this sprint | 20:11 |
| weshay|ruck | ? | 20:11 |
| weshay|ruck | granted | 20:11 |
| rlandy | weshay|ruck: bring up osp 16 | 20:12 |
| rlandy | clean up | 20:12 |
| weshay|ruck | +1 | 20:12 |
| rlandy | add to cockpit | 20:12 |
| rlandy | be done here going solo | 20:12 |
| rlandy | it's stein I think | 20:12 |
| rlandy | checking | 20:12 |
| rlandy | yep - stein as well | 20:13 |
| rlandy | need to bring up train here | 20:13 |
| rlandy | anyways ... now I do have to go | 20:13 |
| weshay|ruck | rlandy, https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/baremetal-undercloud/packages/defaults/main.yml#L7 | 20:14 |
| rlandy | weshay|ruck: ok - will pick this up in next sprint ... have an easy fast ... good new year etc. | 20:14 |
| rlandy | yeah - but this is virt undercloud | 20:14 |
| weshay|ruck | thanks :) OH | 20:14 |
| panda | rfolco: drop me an email if I can continue something tomorrow morning | 20:14 |
| weshay|ruck | rlandy, | 20:15 |
| weshay|ruck | k | 20:15 |
| rlandy | yeah? | 20:15 |
| weshay|ruck | accident ping | 20:15 |
| rfolco | panda, I am still struggling with loading files that were generated locally in the host controller | 20:15 |
| weshay|ruck | rlandy, :) take care | 20:15 |
| rfolco | panda, trying to move stage-info to executor root, only from there I can fetch | 20:16 |
| *** rlandy has quit IRC | 20:16 | |
| rfolco | panda, will update you on where I stop | 20:16 |
| *** ksambor has joined #oooq | 20:23 | |
| *** ksambor has quit IRC | 20:23 | |
| *** dsneddon_ is now known as dsneddon | 20:26 | |
| *** aakarsh|3 has joined #oooq | 20:31 | |
| *** holser has quit IRC | 20:33 | |
| *** aakarsh|2 has quit IRC | 20:33 | |
| *** aakarsh|3 has quit IRC | 20:38 | |
| *** holser has joined #oooq | 20:43 | |
| *** holser has quit IRC | 20:45 | |
| *** holser has joined #oooq | 20:47 | |
| *** slaweq has quit IRC | 20:59 | |
| *** jbadiapa has quit IRC | 21:03 | |
| *** holser has quit IRC | 21:18 | |
| *** saneax has joined #oooq | 23:08 | |
| *** tosky has quit IRC | 23:10 | |
| *** aakarsh has joined #oooq | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!