Monday, 2021-08-02

« Saturday, 2021-07-31 Index Tuesday, 2021-08-03 »
*** marios is now known as marios|ruck05:38
*** pojadhav- is now known as pojadhav06:24
*** jpena|off is now known as jpena07:00
marios|ruckchandankumar: quick question https://review.opendev.org/c/openstack/tripleo-quickstart/+/803134/2#message-b9756e12bf60266583a5067a2298ffd5e876e1f1 07:01
marios|ruckchandankumar: but otherwise see previous comment are we waiting for testing or good to go? otherwise why cant we put that into the release files per ^^ 07:02
zbrsshnaidm_: if you could release a new version of podman collection it would be great.07:21
zbrwith bit of luck, that is all we need to finish switching from docker to podman on molecule scenarios07:22
chandankumarykarel: hello, is it possible to test this change https://review.opendev.org/c/openstack/tripleo-quickstart/+/803134 via testproject?07:32
ykarelchandankumar, no not possible with testproject as that's jenkins jobs07:35
ykarelcan do some tests with jenkins hack, but the change should be good to go07:35
chandankumarmarios|ruck: ^^ Done, thanks :-)07:37
marios|ruckchandankumar: ack 07:42
ykarelchandankumar, i see https://ci.centos.org/job/tripleo-quickstart-gate-master-delorean-full-featureset052/ too failing07:47
chandankumarhmm07:51
chandankumarthis one failed at Build and push container images to the local registry07:52
chandankumarif I see here https://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-gate-master-delorean-full-featureset052-1099/172.19.3.93/home/stack/builder-undercloud.log07:52
chandankumartripleo-operator-ansible got installed07:52
ykarelchandankumar, actually it's failing while looking for that role on virthost07:53
ykarelso likely need more fixes your patch07:54
ykarelas similar error is in promotion pipeline,07:54
ykarelcan hold your patch07:54
zbrmarios|ruck: chandankumar: can we please do https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/792652 ?07:57
zbrhttps://review.opendev.org/c/openstack/tripleo-repos/+/800462 is also waiting for your reviews.07:58
chandankumarykarel: Do you have logs from promotion pipeline?07:59
ykarelhttps://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-wallaby-current-tripleo-delorean-minimal-24/console.log07:59
chandankumarmarios|ruck: please remove +w from above patch08:07
chandankumarmore fix is needed for standalone08:07
*** sshnaidm_ is now known as sshnaidm08:20
sshnaidmzbr, yeah, will do today08:25
marios|ruckchandankumar: which one08:26
chandankumarmarios|ruck: sorry just updated the patch08:26
marios|ruckack 08:26
*** ykarel is now known as ykarel|lunch08:34
sshnaidmzbr, before release, need to fix: https://github.com/ansible-community/molecule-podman/pull/4609:09
zbrsshnaidm: done, merged.09:20
zbrsshnaidm: https://github.com/ansible-community/molecule-podman/issues/45 could also prove a major issue but is not a blocker. use of syncronize is very popular.09:22
sshnaidmzbr, yeah, working on this..09:24
sshnaidmzbr, after this pr is merged, I'll release: https://github.com/containers/ansible-podman-collections/pull/27909:25
sshnaidmneed to wait for tripleo jobs :D09:25
sshnaidmzbr, don't you have tripleo-ansible molecule jobs in molecule-podman repo?09:25
zbrnope, there is nothing specific to openstack in molecule & drivers.09:28
zbrthe only exception is for linter where we have an "eco"-system pipeline that tests outcomes while running on a set of 3rd party repositories, some of them are openstack/tripleon ones https://github.com/ansible-community/ansible-lint/blob/master/playbooks/eco.yml#L8-L3709:29
zbri would prefer to keep it simple, especially as we use GHA, not zuul. Only two projects have a 3rd party zuul due to limitations on GHA (molecule-libvirt and molecule-vagrant)09:30
zbrit should not matter, but if we find a bug that affects us, we can easily adapt the tests and improve them.09:31
zbrmarios|ruck: do we have known blockers or should I be confident to use retry when getting what looks like random failure?09:32
marios|ruckzbr: link? 09:32
marios|ruckzbr: not aware of gate issues today 09:32
zbrhttps://review.opendev.org/c/openstack/tripleo-ansible/+/787767 got one failure, looks random to me.09:33
marios|ruckzbr: no looks like new issue09:34
marios|ruckzbr: i will file a bug but need to dig a bit09:34
marios|ruckzbr: i see two so far similar fail https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-content-provider 09:34
marios|ruckzbr: 2021-08-02 08:25:05 | fatal: unable to access 'https://github.com/rdo-packages/tripleo-ansible-distgit.git/': Failed to connect to github.com port 443: Connection timed out 09:35
marios|ruckzbr: go ahead recheck lets see09:39
zbrsshnaidm: stuff like ^ is why I would oppose adding tripleo jobs to molecule ;)09:42
sshnaidmzbr, you don't like challenges09:57
zbrin fact I like those where there is slight chance for me to win09:58
marios|rucksoniya29|rover: o/ i am going afk for a few in a couple mins10:06
sshnaidmzbr, in GHA: fatal: unable to access 'https://github.com/ansible/ansible.git/': Failed to connect to github.com port 443: Connection timed out10:09
sshnaidmso tripleo is not related :)10:09
marios|rucksoniya29|rover: fyi https://bugs.launchpad.net/tripleo/+bug/1938684 not sure if it is transient yet10:09
sshnaidmmarios|ruck, ^^ seems like problem with github10:09
*** ykarel|lunch is now known as ykarel10:10
marios|rucksshnaidm: yeah just dont know how widespread yet if transient etc 10:11
* marios|ruck back in ~40 mins 10:13
marios|ruckthanks soniya|rover 11:03
soniya|rovermarios|ruck, ack11:11
marios|ruckanyone know what's up with SKIPPED @ https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-main ? 11:14
*** jpena is now known as jpena|lunch11:30
*** rlandy is now known as rlandy|ruck11:32
rlandy|ruckmarios|ruck: soniya|rover: hi - how are things today? any emergencies?11:34
marios|ruckrlandy|ruck: something wrong with promoter master11:41
rlandy|ruckmarios|ruck: last week it was failing on manifest push11:41
marios|ruckrlandy|ruck: http://10.0.148.74/promoter_logs/centos8_master.log failed promotion 2021-08-02 11:28:20,193 480062 ERROR    promoter TASK [containers-promote : Fail if there are missing containers] *************** 11:41
rlandy|ruckstill the issue?11:41
marios|ruckrlandy|ruck: tried to promote but fails pushing containers 11:41
* rlandy|ruck looks11:41
marios|ruckrlandy|ruck: "no such manifest: 11:41
marios|ruckrlandy|ruck: that what you meant? T^^^ 11:41
rlandy|ruckidk if it's the same - looking at logs11:42
marios|ruckrlandy|ruck: commented on gchat and ccd you... that and also master line skipped all the things some infra issue? 11:42
rlandy|ruckakahat: ^^ did you turn off manifest push>11:42
rlandy|ruckk11:43
marios|ruckrlandy|ruck: also gate blocker? transient? filed https://bugs.launchpad.net/tripleo/+bug/193868411:43
marios|ruckrlandy|ruck: but otherwise, yeah living the dream \o/ :D11:43
* rlandy|ruck looks at promoter and then the line11:43
marios|ruckrlandy|ruck: hoping that is not a gate bloker and that it is transient ^^^ 11:43
rlandy|ruckwoohoo  - another week of fun11:43
akahatrlandy|ruck, yes. i've turned it off.11:43
rlandy|ruckmarios|ruck: ^^ ok - then we have another issue11:44
rlandy|rucklooking11:44
rlandy|ruckhttp://10.0.148.74/promoter_logs/container-push/20210802-112540.log11:46
rlandy|ruckmarios|ruck: ^^ pushing containers now11:46
rlandy|rucktransient?11:47
akahatis this hash exists in registry: b18a7a08a93f7cf4643d1fd721d9658311:47
rlandy|ruckdocker.io/tripleomaster/openstack-designate-producer:6392edc0eaefb612986de9809a0195ba11:47
akahatit shows ""no such manifest: trunk.registry.rdoproject.org/tripleomaster/openstack-octavia-health-manager:b18a7a08a93f7cf4643d1fd721d96583""11:47
akahatrlandy|ruck, is is right url: http://trunk.registry.rdoproject.org:8443/oapi/v1/namespaces/tripleomaster/imagestreamtag/11:48
marios|ruckrlandy|ruck: looking11:48
marios|ruckrlandy|ruck: early on in that file there is still errror so didn't scroll to bottom i mean there http://10.0.148.74/promoter_logs/container-push/20210802-112540.log11:48
marios|ruckrlandy|ruck: lets see if it manages to promote then thanks for checking b18a7a08a93f7cf4643d1fd721d9658311:49
rlandy|ruck2021-08-02 11:28:20,067 p=480121 u=promoter n=ansible | TASK [containers-promote : Fail if there are missing containers] ***************11:50
rlandy|ruck2021-08-02 11:28:20,085 p=480121 u=promoter n=ansible | fatal: [localhost]: FAILED! => {"changed": false, "msg": "There are missing containers"}11:50
rlandy|ruckI see that11:50
rlandy|ruck "docker.io/tripleomaster/openstack-zaqar-wsgi:current-tripleo-rdo"}11:51
rlandy|ruckpushing current-tripleo-rdo11:51
rlandy|rucktrunk.registry.rdoproject.org/tripleomaster/openstack-rsyslog:b18a7a08a93f7cf4643d1fd721d96583", "no such manifest: trunk.registry.rdoproject.org/tripleomaster/openstack-unbound:b18a7a08a93f7cf4643d1fd721d96583"], 11:52
rlandy|ruckclaims it's missing all the manifests from b18a7a08a93f7cf4643d1fd721d9658311:56
rlandy|ruckchecking previous logs11:57
rlandy|rucksshnaidm: thank you - cockpit looks sane again11:57
rlandy|ruckmarios|ruck: testproject in the failed fs001 in master line11:59
rlandy|ruckhopefully this new hash will then promote11:59
rlandy|ruck2021-08-02 10:54:41 | 2021-08-02 10:54:41.378 216382 ERROR openstack [-] Message queue for ephemeral heat not created in time.: tripleoclient.exceptions.HeatPodMessageQueueException: Message queue for ephemeral heat not created in time.[00m12:00
rlandy|ruck^^ watch that12:00
marios|ruckrlandy|ruck: the latest master periodic line is all skipped https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-main 12:01
marios|ruckrlandy|ruck: i noted in the gchat, maybe some issue12:01
rlandy|ruckopenstack-periodic-integration-main12:01
rlandy|ruckit's running right now12:01
rlandy|ruckmarios|ruck: ^^ am I missing something???12:02
marios|ruckratailor__: Invalid dateSKIPPED @ https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-main 12:04
marios|ruckratailor__: sorry for noise was meant for someone else12:04
ratailor__marios|ruck, np :)12:04
marios|ruckrlandy|ruck: ^^ i see it running now https://review.rdoproject.org/zuul/status if that is what you meant yea12:04
rlandy|ruckthat was for me :)12:04
rlandy|ruckyes12:04
rlandy|ruckjust reran fs00112:04
marios|ruckrlandy|ruck: but donno why previous run is skipped 12:04
rlandy|ruckprobably infra12:05
rlandy|ruckoh12:05
rlandy|ruckgerrit was out12:05
rlandy|ruckmaybe some result12:05
rlandy|ruck2021-08-02 11:46:52,437 p=481816 u=promoter n=ansible | PLAY RECAP *********************************************************************12:08
rlandy|ruck2021-08-02 11:46:52,437 p=481816 u=promoter n=ansible | localhost                  : ok=30   changed=18   unreachable=0    failed=0    skipped=8    rescued=0    ignored=0   12:08
rlandy|ruckit supposed to have finished pushing12:08
rlandy|ruckakahat: ^^12:09
rlandy|ruckif manifest push is off12:09
rlandy|ruckand the previous completed12:09
rlandy|ruckwhat's it doing now12:09
rlandy|ruckoh nvm12:10
rlandy|rucktag12:10
rlandy|ruckI see12:10
akahat:)12:11
rlandy|ruckweshay|ruck: hey12:16
rlandy|ruckweshay|ruck: so you want to run the CIX meeting or should I?12:16
weshay|ruckI can do it 12:16
rlandy|ruckok12:16
marios|ruckbrb for cix call o/ weshay|ruck welcome back ;)12:17
chandankumarweshay|ruck: welcome back sir, Hope you had a great vacation :-)12:18
weshay|ruckhola hola everyone :)12:21
Tenguhe's back!12:22
*** jpena|lunch is now known as jpena12:28
rlandy|ruck2021-08-02 12:19:19,098 514147 INFO     promoter Qcow promote 'aggregate: 6392edc0eaefb612986de9809a0195ba, commit: b38561c845156f014892d3910972faf38d560cec, distro: ae83045246b1725cbc365b60068f180c76cda314, extended: None, component: validation, timestamp: 1627581069' to current-tripleo-rdo: Successful promotion12:30
rlandy|ruckgetting there12:30
rlandy|ruckakahat: 2021-08-02 12:32:59,550 p=548357 u=promoter n=ansible | fatal: [localhost]: FAILED! => {"changed": false, "msg": "There are missing containers"}12:35
rlandy|ruck^^ still failing master12:35
akahatrlandy|ruck, looking 12:35
akahatrlandy|ruck, i'm not able to find this hash: b18a7a08a93f7cf4643d1fd721d96583 in the rdo registry12:38
akahatcould you please recheck it.12:38
rlandy|ruckmaybe it got purged12:38
* rlandy|ruck checks12:38
marios|ruckykarel: can you check https://trello.com/c/OmLNBwGT/2041-cixlp1938283tripleociproa-master-ci-jobs-failing-randomly-as-pcs-resource-operations-actions-times-out please can we close this out ? 12:45
ykarelmarios|ruck, ack looking12:47
marios|ruckykarel: thanks just discussed in cix call can you update with some comment and we can close 12:48
ykarelsure updating12:49
rlandy|ruckakahat: what registry link are you checking?12:52
akahatrlandy|ruck, https://trunk.registry.rdoproject.org:8443/oapi/v1/namespaces/tripleomaster/imagestreamtags12:52
rlandy|ruck:6392edc0eaefb612986de9809a0195ba12:54
rlandy|ruckthat one - previous12:54
rlandy|ruckc7581691e115f2708651475b1d91a8e112:55
rlandy|ruckbut yeah - no b18a7a08a93f7cf4643d1fd721d9658312:55
rlandy|ruckidk what happened to that hash12:55
rlandy|ruckif the current run works12:56
rlandy|ruckwe can get that promoted12:56
* rlandy|ruck checks for current hash12:56
rlandy|ruckf50dfd1b5edf85ec026b5ae4a32b4219 hash is there12:57
rlandy|ruckmarios|ruck: akahat: ^^ one running now12:57
rlandy|ruckso if we get current tests to pass that are in rerun, that hash should promote12:58
akahatthis hash is uploaded at 2912:58
akahathttps://trunk.rdoproject.org/centos8-master/tripleo-ci-testing/b1/8a/b18a7a08a93f7cf4643d1fd721d96583/12:58
akahatah.. great12:58
rlandy|ruckf50dfd1b5edf85ec026b5ae4a32b4219 hoping for13:00
rlandy|ruckzbr: scrum time'13:01
chandankumarhttps://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/79092613:07
dviroelreviews on these please:13:14
dviroelfirst: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3461213:14
dviroelthen: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3476713:14
dviroelthe second patch adds a molecule job that test the roles added13:15
marios|ruckhttps://bugs.launchpad.net/tripleo/+bug/193868413:16
marios|ruckrlandy|ruck: ^^^^ 13:16
rlandy|ruckzbr: https://27d56bbd07be34be0cc1-9b6cbe136e0e918c88bba727d81cded7.ssl.cf1.rackcdn.com/787502/9/check/tox-ansible-test-sanity/db8ab1e/job-output.txt13:26
marios|ruckb18a7a08a93f7cf4643d1fd721d96583 fails container push missing containers ? http://10.0.148.74/promoter_logs/container-push/20210802-112540.log13:34
marios|ruckrlandy|ruck: ^^ 13:34
rlandy|rucksoniya29|rover: f50dfd1b5edf85ec026b5ae4a32b421913:38
marios|rucksoniya29|rover: rlandy|ruck: https://docs.openstack.org/tripleo-docs/latest/ci/chasing_promotions.html#specifying-a-particular-hash 13:39
zbrplease review and +W https://review.opendev.org/c/openstack/tripleo-repos/+/80046213:48
soniya|roverrlandy|ruck, the hash- f50dfd1b5edf85ec026b5ae4a32b4219 which you specified above, is it the same one which we want to specify by dlrn_hash_tag parameter for fs35 master with the test-project?14:13
rlandy|rucksoniya|rover: yes14:14
soniya|roverrlandy|ruck, okay, thanks14:15
*** ykarel is now known as ykarel|away14:16
rlandy|rucksoniya|rover: marios|ruck: testproject added for retry_limit job periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-ussuri14:20
rlandy|ruckwill try to get ussuri to promote today14:21
soniya|roverrlandy|ruck, ack14:21
marios|ruckrlandy|ruck: thanks14:29
sshnaidmzbr, released14:49
soniya|roverrlandy|ruck, marios|ruck, testproject added for job periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master14:50
soniya|roverrlandy|ruck, marios|ruck, https://review.rdoproject.org/r/c/test-project/+/3479914:50
rlandy|rucksoniya|rover: thanks - looking14:51
rlandy|rucksoniya|rover: so ... https://review.rdoproject.org/r/c/test-project/+/34799 - 14:53
rlandy|ruckhttps://review.rdoproject.org/r/c/test-project/+/34799/1/.zuul.yaml is correct but your review is not showing on https://review.rdoproject.org/zuul/status14:53
rlandy|ruckyour repo is wrong14:53
rlandy|rucktestproject - no hash14:54
rlandy|ruck Repo | Branch test-project | master 14:54
rlandy|ruckI mean no dash :)14:54
rlandy|ruckrepo testproject14:54
soniya|roverrlandy|ruck, okay, thanks :)14:56
marios|ruckrlandy|ruck: wana talk promotion criteria sorry just finished call 15:04
marios|ruckrlandy|ruck: fs35 isn't even in criteria? 15:06
marios|ruckrlandy|ruck: at least on the promoter15:06
marios|ruck        #- periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master15:06
marios|ruckcd ci-config/ci-scripts/dlrnapi_promoter/config_environments/rdo/CentOS-815:07
rlandy|ruckmarios|ruck: either way - let's get a run on it15:07
marios|ruckrlandy|ruck: k 15:07
marios|ruckrlandy|ruck: am gonna copy paste master there https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config_environments/rdo/CentOS-8/master.yaml#L14 now is that OK ? 15:07
marios|ruckrlandy|ruck: it will include 3615:07
marios|ruck3515:07
rlandy|rucksec15:07
marios|ruckrlandy|ruck: https://github.com/rdo-infra/ci-config/blob/4a68c3b7b062f9768410f5cafa171ff256dc0972/ci-scripts/dlrnapi_promoter/config_environments/rdo/CentOS-8/master.yaml#L18 15:08
rlandy|ruckmarios|ruck: hold on15:08
rlandy|ruckthat will include fs03515:08
rlandy|ruckpromoter is still trying to push commit: b38561c845156f014892d3910972faf38d560cec15:09
marios|ruckrlandy|ruck: ok then15:10
marios|ruckrlandy|ruck: bump tomorrow15:10
marios|rucknp15:10
rlandy|ruckon tmux15:10
rlandy|ruckmarios|ruck: so I think we should restart promoter with current config15:10
marios|ruckrlandy|ruck: i am on tmux 15:11
marios|ruckrlandy|ruck: switching pvt sec15:11
rlandy|ruckfs035 is in here15:11
soniya29|roverrlandy|ruck, marios|ruck, https://review.rdoproject.org/r/c/testproject/+/3480015:18
sshnaidmzbr, does molecule take collection from galaxy? or from rpm?15:19
rlandy|rucksoniya29|rover; +1 - thanks15:19
*** jpena is now known as jpena|off15:25
soniya29|roverrlandy|ruck, :)15:31
zbrsshnaidm: galaxy only but only when missing or outdated15:34
soniya29|roverrlandy|ruck, marios|ruck, anything else where I can help, please let me know15:34
sshnaidmzbr, ok, because rpm is not ready yet15:34
zbrthe condition for required collection is done in such way that i should not attempt to use galaxy if is already present15:35
rlandy|rucksoniya29|rover: think we are ok for now ... but if you have time, we want to check which components have not promoted and may need help/bugs15:35
rlandy|ruckso all components train -> master15:35
soniya29|roverrlandy|ruck, okay15:37
marios|ruckk thanks soniya29|rover 15:59
marios|ruckrlandy|ruck: am getting out in a bit15:59
rlandy|ruckmarios|ruck: sure - have a good night16:00
rlandy|ruckchandankumar: hmm, ... looks like we are hitting the same problem on baremetal jobs: https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2/7e94eea/job-output.txt16:12
rlandy|ruckOVB downstream looks good16:12
soniya|roverrlandy|ruck, weshay|ruck, leaving for the day16:21
weshay|rucksoniya|rover, k.. thank you!16:21
*** sshnaidm is now known as sshnaidm|afk16:25
*** marios|ruck is now known as marios|out16:26
rlandy|ruckarxcruz: 1-on-1?16:35
zbrdviroel: please double check https://review.opendev.org/c/openstack/tripleo-repos/+/800462 16:36
dviroelzbr: hi, i was checking this few minutes ago16:38
dviroelzbr: where are the sanity test outputs?16:38
zbrlook at zuul output (tox)16:39
zbryou will still see some failures, but these will be fixed later.16:39
zbr(later = in follow-up)16:40
dviroeli still don't see any errors on those tox outputs16:43
dviroelbut np, if you will continue to work on these test in a follow up16:44
dviroelthe code lgtm16:44
rlandy|rucklunch - brb16:45
rlandy|ruck  name: tripleo_overcloud_image_upload - also hitting baremetal17:30
weshay|ruckrlandy|ruck, http://dashboard-ci.tripleo.org/d/mOvYIiOMk/component-pipeline-train?orgId=1 is fixed17:42
weshay|ruckrlandy|ruck, anything else busted on the dashboard you noticed?17:42
rlandy|ruckthank you17:50
rlandy|ruckyep ussuri and victroa lines were mixed up17:50
* rlandy|ruck gets17:50
weshay|ruckrlandy|ruck, looks right to me.. http://dashboard-ci.tripleo.org/d/SjLc_1cGk/component-pipeline-victoria?orgId=117:50
rlandy|ruckidk - maybe we fixed it17:51
weshay|ruckussuri also looks ok17:51
rlandy|ruckoh ...17:52
rlandy|ruckno - looks right now17:53
rlandy|ruckwill check again17:53
rlandy|ruckshoot gate failures begin18:28
weshay|ruckrlandy|ruck, https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/80324618:51
weshay|ruckrlandy|ruck, re: gate18:51
weshay|ruckI did something wrong w/ the query though18:52
weshay|ruck2021-08-02 18:44:38.521122 | ubuntu-focal | ERROR: Git reported dirty status. Git should never report dirty status at the end of testing, regardless if status is passed, failed or aborted.18:52
rlandy|ruckwhat's the question?18:53
rlandy|ruckwhy review has zuul -118:54
rlandy|ruckyou're just giving a heads up18:54
rlandy|ruckhttps://bugs.launchpad.net/tripleo/+bug/193868418:55
rlandy|ruckmarios loged ^^ as possible gate blocker18:56
weshay|ruckrlandy|ruck, that's the gate failure18:56
rlandy|ruckk - so then we have a real issue18:56
weshay|ruckthe current one didn't fail w/ github... failed on pulling ubi818:56
rlandy|ruckdifferent then ok18:57
weshay|ruckrlandy|ruck, although we should create a query for 18:57
weshay|ruck2021-08-02 08:25:05 | fatal: unable to access 'https://github.com/rdo-packages/tripleo-ansible-distgit.git/': Failed to connect to github.com port 443: Connection timed out18:57
weshay|ruckor atleast " Failed to connect to github.com port 443"18:58
weshay|ruckrlandy|ruck, can you help me please w/ https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/803246 any idea why that file is dirty?19:00
rlandy|rucklooking19:00
rlandy|ruckha - used to see that in my local runs19:02
rlandy|ruckGit reported dirty status. Git should never report dirty status at the end of testing, regardless if status is passed19:03
rlandy|ruck2021-08-02 18:44:38.526359 | ubuntu-focal | Untracked files:19:03
rlandy|ruck2021-08-02 18:44:38.526374 | ubuntu-focal |   (use "git add <file>..." to include in what will be committed)19:03
rlandy|ruck2021-08-02 18:44:38.526395 | ubuntu-focal | output/elastic-recheck/1938718.yaml19:03
weshay|ruckam I suppose to create that file?19:04
* rlandy|ruck looks if that was added19:04
rlandy|ruckhold on19:04
rlandy|ruckweshay|ruck: can I edit19:11
weshay|ruckrlandy|ruck, always19:12
weshay|ruckssamal, you want/need an irc bouncer?19:12
ssamalweshay|ruck, I was gonna ask you but since I am not really working/communicating, thought would ask later.19:13
weshay|ruckok.. ya.. I'll create an account on mine for you and email details.. you can take setup the client side later though.  thought I would ask19:14
rlandy|ruckERROR: Git reported dirty status. Git should never report dirty status at the end of testing, regardless if status is passed, failed or aborted.19:15
weshay|ruckya.. not sure what to do w/ that19:15
rlandy|ruckI get that always in my personal runs19:16
rlandy|rucklet's see if it still bothers it19:16
ssamalweshay|ruck, that would be awesome. Thanks :)19:18
weshay|ruckbah.. quotes19:19
weshay|ruckthanks19:19
ssamalweshay|ruck, i wanted to go back and forth, I am not sure about my boundaries here. lol19:20
weshay|ruckssamal, back and forth w/ regards to irc?19:21
ssamalyes19:21
rlandy|ruck2021-08-02 19:28:44.429614 | ubuntu-focal | py38 finish: run-test  after 142.60 seconds19:30
rlandy|ruck2021-08-02 19:28:44.456943 | ubuntu-focal | ERROR: Git reported dirty status. Git should never report dirty status at the end of testing, regardless if status is passed, failed or aborted.19:30
rlandy|ruck2021-08-02 19:28:44.460517 | ubuntu-focal | On branch master19:30
rlandy|ruckweshay|ruck: ^^ also got that error19:30
rlandy|ruckoutput/elastic-recheck/1938718.yaml19:30
weshay|ruckah19:30
rlandy|ruck^^ problem with that file19:30
rlandy|ruckissue with the repo19:32
rlandy|ruckfrenzy_friday: ^^19:32
rlandy|ruckhealth-query has a repo issue19:32
rlandy|ruck.git/COMMIT_EDITMSG:Related-Bug: #193871819:33
rlandy|rucksrc/data/queries.yml:    url: https://bugs.launchpad.net/tripleo/+bug/193871819:33
rlandy|ruckrecloned19:40
rlandy|ruckwhen I recloned, I didn;t get that error19:43
rlandy|ruck Powered by Gerrit Code Review (3.2.11-13-g94b0943907-dirty) 19:43
rlandy|ruckpromoter Containers promote 'aggregate: f50dfd1b5edf85ec026b5ae4a32b4219, commit: 07e484a8616a4f51e998f0e8ae865e0781d0b20b, distro: 926d273316c9dc7cf18b78c6ff88dae77078ea01, extended: None, component: validation, timestamp: 1627921043' to current-tripleo: Attempting promotion19:54
rlandy|ruckwoohoo 19:54
rlandy|ruckmaster is on the run19:54
rlandy|rucko .. health19:56
rlandy|ruckussuri also promoting20:07
rlandy|ruckweshay|ruck: looks to be some stash error ... I can reproduce it if I leave tox dir over from fist run20:19
rlandy|ruckfirst20:19
rlandy|ruckfrenzy_friday: ^^ when you get in - pls20:19
rlandy|ruckI even added the file with git20:19
rlandy|ruckno luck20:19
*** dviroel is now known as dviroel|out20:53
weshay|ruckrlandy|ruck, anything you want me to look at?21:11
weshay|ruckrlandy|ruck, I've completed some updates to the dashboards, cockpit and component lines.. if you have other suggestions holla21:11
rlandy|ruckthat's all for the moment21:23
rlandy|ruckrecheck the gate fail21:24
rlandy|ruckweshay|ruck: ^^21:24
rlandy|ruckmaster and ussuri are promoting21:24
weshay|ruckpromotions are flowing.. we'll have to be careful it may be too many21:25
rlandy|ruckneed to decide what to do about downstream21:25
rlandy|ruckgit some debug to do there21:25
rlandy|ruckmultinode21:25
weshay|ruckI haven't looked.. re: 16.2 promotion or something else/21:26
rlandy|ruckbaremetal is failing due to chandan's change21:26
weshay|ruck?21:26
rlandy|ruckno need21:26
rlandy|ruck17 and 16.221:26
weshay|ruckrlandy|ruck, operators right?21:27
rlandy|ruckyep21:27
rlandy|ruckbeen a barrel of laughs21:28
weshay|ruckhrm.. if you want to fill me in to see if I can help.. I'm avail21:28
rlandy|ruckthere is an issue with the current patch21:29
rlandy|ruckleft a note21:29
rlandy|ruckwe can get involved if no fix by tomorrow21:29
weshay|ruckk21:29
rlandy|ruck I am sure they will sort it out by then21:29
rlandy|ruckweshay|ruck: we got a response re:  rhos-121:30
rlandy|ruckshould be fine21:30
rlandy|ruckweshay|ruck: hey22:16
rlandy|ruckweshay|ruck: promoter issue22:16
rlandy|ruckcan you view master promoter?22:17
rlandy|ruckit pushed the containers and images22:18
rlandy|rucknever promoted the hash22:19
weshay|ruckrlandy|ruck, hey22:19
weshay|ruckugh22:19
rlandy|ruckhttp://10.0.148.74/promoter_logs/centos8_master.log22:19
rlandy|ruckon the tmux22:19
rlandy|ruckallows_clients is commented out22:19
weshay|ruckk.. getting on tmux22:20
rlandy|ruckhttps://images.rdoproject.org/centos8/master/rdo_trunk/current-tripleo/22:21
rlandy|ruckthere22:21
rlandy|ruckweshay|ruck: ^^22:21
weshay|ruckya22:21
weshay|ruckso.. I wonder if there isn't a default22:21
weshay|ruckfor allowed_clients22:21
rlandy|ruckhttps://hub.docker.com/r/tripleomaster/openstack-base/tags?page=1&ordering=last_updated22:22
rlandy|ruckhas the f50... hash22:22
rlandy|ruckf50dfd1b5edf85ec026b5ae4a32b421922:22
rlandy|ruckLast pushed24 minutes ago22:22
weshay|ruckya.. I don't see in the previous logs where it attempted the promotion yet22:23
weshay|ruckhttp://10.0.148.74/promoter_logs/centos8_master_2021-07-29T22:37.log22:24
weshay|ruckmay have to wait until this finishes w/ containers22:24
rlandy|ruckit just rotates22:24
rlandy|ruckussuri promoted ok22:24
rlandy|ruckoh and now the gate is having a fit22:25
rlandy|ruckundercloud-deploy : Install the undercloud22:27
rlandy|ruckmaybe since we just updated ussuri22:28
rlandy|ruckweshay|ruck: and log rotate is messed up here22:30
rlandy|rucklooks like it's writing over the last log22:30
weshay|ruckya.. I've mentioned the log rotate issue to amol22:30
rlandy|ruckthere should be multiple for 08/0222:30
rlandy|ruckit got to pushing images22:31
rlandy|ruckand the just restarted22:31
rlandy|rucklooks like it skipped the dlrn_client22:31
rlandy|ruckcan we confirm that?22:31
rlandy|ruck#allowed_clients: dlrn_client,qcow_client22:31
weshay|ruckchecking ussuri mirrors22:31
rlandy|ruckexplicitly define allowed_clients to include dlrn_client22:32
weshay|ruckseveral mirrors not synced22:32
rlandy|ruckgate - yeah - 22:32
rlandy|ruckjust updated ussuri22:32
rlandy|ruckthrew the gate off22:32
rlandy|ruckwill recheck that22:32
weshay|ruckrlandy|ruck, master just promoted22:34
weshay|ruckhttp://images.rdoproject.org/centos8/master/rdo_trunk/?C=M;O=D22:34
weshay|ruckoops22:34
weshay|ruckhold on22:34
rlandy|ruckthat's what it did before22:35
rlandy|ruckpush conatiners, images and die22:36
rlandy|ruckno dlrn promotion22:36
weshay|ruckya... and we shouldn't need the allowed_clients22:36
weshay|ruckhttp://promoter.rdoproject.org/config/CentOS-8/wallaby.yaml22:36
rlandy|ruckthe code base is diff on the two promoters22:38
rlandy|ruckI think 22:38
rlandy|ruckcorrect for rdoproject one22:38
rlandy|ruckidk about master one22:38
weshay|ruckthe other gate failure.. I think failed on pulling ubi8 again22:39
weshay|ruckI don't see the base failure22:39
weshay|ruckthe base build log rather22:39
rlandy|ruckweshay|ruck: wrt master22:40
rlandy|ruckcan we redefine allowed_clients and retry?22:40
weshay|ruckrlandy|ruck, we should probably turn off container promotion and retry22:41
rlandy|ruckfunny - we promoted fine last week22:41
rlandy|ruckand image22:41
rlandy|ruckweshay|ruck: ^^ just dlrn22:41
rlandy|ruckextended: None, component: validation, timestamp: 1627921043': SUCCESSFUL promotion to current-tripleo22:43
rlandy|ruckchecking dlrn22:43
rlandy|ruckhttps://trunk.rdoproject.org/centos8-master/current-tripleo/delorean.repo.md522:44
rlandy|ruckweshay|ruck: ^^ yep22:44
rlandy|ruckgot it now22:44
weshay|ruckya.. bug some where22:44
rlandy|ruckwe need to put back all three22:44
rlandy|ruckweshay|ruck: ^^22:44
rlandy|ruckand restart22:44
weshay|ruckrlandy|ruck, no.. let's leave it off22:44
weshay|ruckand afaict... we should have allowed_clients remarked out22:45
weshay|rucktotally right?22:45
rlandy|ruckyes - if the code on this promoter is updated22:45
rlandy|rucktotally correct on rdoproject promoter22:45
weshay|ruckya.. they need to update the code22:45
* rlandy|ruck will check with akahat first thing tomorrow22:45
weshay|rucklet's just leave master off for now22:45
rlandy|ruckleaving ruck/rover note22:45
rlandy|ruckmay throw the gate22:46
weshay|ruckya22:46
rlandy|ruckmore promotions22:46
weshay|ruckrlandy|ruck, now we get into the grey area w/ lots of opinions w/ regards to how often to promote...22:47
weshay|ruckthe mirrors suck22:47
weshay|ruckrlandy|ruck,  I can recheck the failed gate jobs.. you should get out of here22:48
rlandy|ruckweshay|ruck: thanks ... yeah  - we need to get this mirror thing sorted22:48
weshay|ruckrlandy|ruck, we may not have the powa22:49
rlandy|ruckweshay|ruck: thanks for gate recheck ... going to step away for a bit mincha and then running time22:49
weshay|ruckrlandy|ruck, although.. part of that script chandan is writing.. should check and pound the mirrors that are not synced22:49
rlandy|ruckwill check back in a bit22:49
rlandy|ruckyep - should help22:49
rlandy|ruckthere is no good time22:49
rlandy|ruckunless we only promoted on the weekends22:50
rlandy|ruckwhich is not a possibility22:50
rlandy|ruckweshay|ruck: ^^ if we only had master, wallaby and train, we could though22:50
weshay|ruckheh22:50
weshay|ruckrlandy|ruck, come to the #tripleo mtg tomorrow22:50
rlandy|rucka girl can dream, right?22:50
weshay|rucknext step.. change release gov22:50
rlandy|ruckk - yep22:51
rlandy|ruckk - back later22:51
*** rlandy|ruck is now known as rlandy|ruck|bbl22:51
« Saturday, 2021-07-31 Index Tuesday, 2021-08-03 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!