Thursday, 2021-06-24

*** ysandeep|out is now known as ysandeep00:32
ysandeepweshay|ruck, nice addition in your latest patch about aggregates, this will enable us to check previous hashes's results00:52
* ysandeep working on unittests and will merge our patches today00:53
*** rlandy|bbl is now known as rlandy02:07
*** ykarel|away is now known as ykarel04:31
*** marios is now known as marios|ruck05:01
*** bhagyashris_ is now known as bhagyashris05:13
*** pojadhav- is now known as pojadhav06:16
anbanerj|rovermarios|ruck, O/07:18
marios|ruckgood morning anbanerj|rover 07:19
anbanerj|rovermarios|ruck, Good morning. Victoria and Ussuri promoted yesterday07:20
marios|ruckanbanerj|rover: thanks i saw good stuff07:22
*** jpena|off is now known as jpena07:36
marios|ruckanbanerj|rover: relocating back in ~1 hour thanks08:11
anbanerj|roverack08:12
*** amoralej|off is now known as amoralej08:20
*** ykarel is now known as ykarel|lunch08:34
*** sshnaidm|afk is now known as sshnaidm08:45
sshnaidmchandankumar, can you take a look please https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/796522/408:46
sshnaidmysandeep, ^08:46
* ysandeep looking08:47
chandankumarsshnaidm: sure09:08
ysandeepsshnaidm, are we collecting ~/rpmbuild/RPMS/noarch in logs? i am trying to find logs in testproject logs https://logserver.rdoproject.org/17/34117/3/check/tripleo-ci-centos-8-content-provider/7b74cd4/logs/undercloud/home/zuul/ 09:09
sshnaidmysandeep, nope09:09
sshnaidmprobably we collect all *.log, but need to check..09:09
ysandeepfor dlrn case.. generally incase of failure i usually check rpmbuild.log, for example: https://sf.hosted.upshift.rdu2.redhat.com/logs/77/247777/1/check/tripleo-ci-rhel-8-standalone-rhos-16.2/b03e757/logs/undercloud/home/zuul/DLRN/data/repos/component/tripleo/62/cc/62cc84b36f9f9bba854bba04d8debada2fb90d00_dev_dev/rpmbuild.log , so imho we should capture similiar logs for this.09:11
sshnaidmysandeep, yeah, but I'm not sure logs are in this folder09:12
sshnaidmit's for rpms09:12
sshnaidmchandankumar, do you remember where we have logs or rpmbuild? ^09:14
chandankumarsshnaidm: checking09:14
ysandeepfrom downstream.. https://sf.hosted.upshift.rdu2.redhat.com/logs/77/247777/1/check/tripleo-ci-rhel-8-standalone-rhos-16.2/b03e757/logs/undercloud/home/zuul/DLRN/data/repos/component/tripleo/62/cc/62cc84b36f9f9bba854bba04d8debada2fb90d00_dev_dev/rpmbuild.log09:15
chandankumarsshnaidm: https://e83c43581d4898374410-212ab268b5e4bbb4b3348f98a2a831ee.ssl.cf2.rackcdn.com/796812/2/check/tripleo-ci-centos-8-content-provider/9144ac4/logs/delorean_logs/component/tripleo/79/7b/797b71e0ab985cf156de3b0fae195c1ccee863e2_dev/index.html09:17
sshnaidmysandeep, actually logs is printed in stdout if it's failing: https://logserver.rdoproject.org/17/34117/3/check/tripleo-ci-centos-8-content-provider/47e2deb/job-output.txt09:17
sshnaidmchandankumar, yeah, but it's for dlrn09:18
chandankumarlooking for github one09:18
sshnaidmysandeep, we can redirect it to some log, or just to leave it printed in stdout in case of failure09:18
sshnaidmchandankumar, I think it's just printed to stdout, see above ^09:19
ysandeepsshnaidm, thanks! I am fine if they are in stdout and printed to job-output.txt 09:19
ysandeephmm.. but we don't capture logs incase of success, probably better to redirect it to some log file anyway.09:23
sshnaidmysandeep, leave a comment09:27
ysandeepsshnaidm, yes posting09:27
ysandeepsshnaidm++ really nice feature, thanks! 09:30
sshnaidmysandeep, np, if you can review this one as well: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/79746309:32
ysandeepsshnaidm: sure, checking09:33
*** marios is now known as marios|ruck09:48
*** bhagyashris_ is now known as bhagyashris09:48
ysandeepsshnaidm: commented on https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/797463 09:56
sshnaidmysandeep, replied10:03
ysandeepack make sense now, thanks!10:06
zbrmarios|ruck: re requirements and molecule, i want to add a check-job but using tripleo-ansible is a bad idea, basically due to its weird way of using molecule.10:08
zbri need to use another tripleo repo which uses molecule but less controversial, and with a very tiny/simple molecule job, preferably one using podman.10:09
marios|ruckzbr: well we are *meant* to have one there ie. we are not fulfilling our side of the contract 10:10
marios|ruckzbr: anyway just mentioned it i hope to revisit it at some point10:10
zbrthe one we just fixed from tripleo-ci would be good candidate, but i am bit afraid to use it as they may say "that is ci only"10:12
marios|ruckzbr: i.e. https://bugs.launchpad.net/tripleo/+bug/1928190/comments/4 and https://github.com/openstack/requirements/blob/ce19462764940a4ce99dae4ac2ec7a004c68e9a4/projects.txt#L245-L249 10:12
marios|ruckzbr: not 'a check job' i am talking specifically about the 'requirements-check' job10:12
zbron the other hand, we having tests inside ci repos, is not really an issue.10:12
zbrit goes both, ways. we need that in our repo and additional job inside requirements repo.10:12
zbrlet me send you a link10:13
zbrthere is zero tripleo coverage here: https://github.com/openstack/requirements/blob/master/.zuul.d/cross-jobs.yaml10:13
zbrif we want to avoid a change made to reqs to f*** us, we need one here. keep in mind that updates to reqs are done automatically10:14
zbrso my plan is to add a very small job names `cross-tripleo-molecule` that assures we do not get surprises10:14
zbrthe good part is that once done, we will be following the official openstack guidelines, no more DIY approaches10:15
zbrmy impression from few years back was that openstack reqs do not allow test requirements being added, but with pytest listed there, i doubt they can argue againts10:16
marios|ruckzbr: ack so i assume the requirements-cross-test is a requirements-check job (the jobs in that file all seem to parent from it)10:18
marios|ruckzbr: thanks for pointers 10:19
zbri am working now on doing these, just wanted to double check with you10:19
*** ykarel|lunch is now known as ykarel10:21
pojadhavmarios|ruck, 0/10:45
marios|ruckpojadhav: hello10:46
pojadhavmarios|ruck, I have added upgrades/update jobs for master in this patch https://review.opendev.org/c/openstack/tripleo-ci/+/797865. but facing issue like standalone upgrade job definition does not exists. So my concern I am not sure do we need "tripleo-ci-centos-8-standalone-upgrade-master" job or we dont need this..?10:46
marios|ruckpojadhav: no we don't commented on the review https://review.opendev.org/c/openstack/tripleo-ci/+/797865/1#message-a25c57ae86984a67bfbc3786742b942c4822b469 10:50
pojadhavmarios|ruck, thanks :)10:50
zbrmarios|ruck: where is that ever used? https://codesearch.opendev.org/?q=tripleo-jobs.py&i=nope&files=&excludeFiles=&repos=11:08
marios|ruckzbr: /me shrug... but checkout git blame ;) i am guessing that is just stale https://opendev.org/openstack/tripleo-ci/commit/a4f7707364f0f4671867d74fe4e3cc6834279cd8 11:09
marios|ruckzbr: dan prince 6 years ago :D11:09
zbrtbh, check-requirements did a good job slapping me about these11:09
anbanerj|roverHey zbr, when you get some time need a little help on the unit tests - https://review.opendev.org/c/opendev/elastic-recheck/+/729623 . It complains ypeError: startWatching() got an unexpected keyword argument 'connection_attempts' but there is connection_attempts arg in the method11:12
zbranbanerj|rover: i see a huge number of errors but not the one you mentioned11:22
*** dviroel|out is now known as dviroel11:32
anbanerj|roverzbr, https://fb0f5fc956b902bf1475-e3d5a649020fd4c80622a59c8df4599b.ssl.cf1.rackcdn.com/729623/17/check/openstack-tox-py36/ee3d4cc/job-output.txt - TypeError: startWatching() got an unexpected keyword argument 'connection_attempts'11:35
anbanerj|roverI messed up the file patch in the last patch, so it had a different error11:35
*** chem is now known as Guest42111:38
*** jpena is now known as jpena|lunch11:39
-opendevstatus- NOTICE: Our Zuul gating CI/CD services will be offline starting around 14:00 UTC (in roughly two hours from now) in order to apply some critical security updates, and is not expected to remain offline for more than 30 minutes.12:00
sshnaidmysandeep, do you know where can I see in repo what exactly is promoted in "clients" component CI?12:10
sshnaidmwhich packages I mean12:10
* anbanerj|rover lunch12:10
rlandysshnaidm: upstream or downstream?12:15
rlandysshnaidm: https://trunk.rdoproject.org/centos8-master/component/clients/component-ci-testing/ is tested in the line12:16
zbrbase on that kind of feedback i get from check-requirements, i think we would have negative ROI if we add molecule to OSP reqs, https://69bbdc3cb3b79f73266a-fb3a89cd73503eeaa3ff317ecc3301da.ssl.cf5.rackcdn.com/797871/4/check/requirements-check/b9d21ac/job-output.txt12:16
rlandyhttps://trunk.rdoproject.org/centos8-master/component/clients/promoted-components/ is promoted out12:16
ysandeepsshnaidm, this tool will help you https://gitlab.cee.redhat.com/whayutin/component_diff_check 12:18
ysandeepsshnaidm, rlandy http://paste.openstack.org/show/806916/12:22
ysandeepsshnaidm, this is basically checking the difference in version.csv https://trunk.rdoproject.org/centos8-master/component/clients/consistent/versions.csv 12:23
rlandysshnaidm: wrt excludepgs in repo_setup role: 12:28
rlandyis that what exclude is supposed to be?12:28
rlandyhttps://github.com/openstack/tripleo-quickstart/blob/master/roles/repo-setup/templates/repo_setup.sh.j2#L8612:28
rlandyjuts missing that options12:28
rlandywe have https://github.com/openstack/tripleo-quickstart/blob/master/roles/repo-setup/templates/repo_setup.sh.j2#L8812:29
ysandeeprlandy, weshay|ruck mtg time12:31
*** amoralej is now known as amoralej|lunch12:32
sshnaidmrlandy, not sure, what is the issue with exclude?12:33
rlandysshnaidm: so exclude is meant to be excludepkgs?12:33
sshnaidmrlandy, yes12:33
sshnaidmrlandy, it's exclude, not excludepkgs12:34
*** jpena|lunch is now known as jpena12:36
ysandeeparxcruz: short answer, my approach.. but we can debate next time13:00
bhagyashrisakahat, anbanerj|rover scrum..13:02
weshay|ruckREVIEWS: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3381513:07
*** amoralej|lunch is now known as amoralej13:18
*** marios|ruck is now known as marios|ruck|call13:19
marios|ruck|callweshay|ruck: anbanerj|rover: new wallaby promotion blocker fs1 https://bugs.launchpad.net/tripleo/+bug/1933504 13:19
akahatls13:20
weshay|ruckzbr, scrum13:25
marios|ruck|callakahat: sudo ls13:25
akahat:)13:25
zbrinteresting, vanished from meet, but still in calendar.13:25
zbrit only gets better... Cound' join meeting, try later using ID from calendar13:27
* zbr needs to check with postman to see if he lost a pink slip13:29
weshay|ruckzbr, thanks13:30
weshay|ruckzbr, lolz13:30
ysandeeprlandy, weshay|ruck: reviews when you have time, revert ovn workaround https://code.engineering.redhat.com/gerrit/q/topic:%22revert_ovn_pinning%22+(status:open%20OR%20status:merged) 13:40
marios|ruck|callin case folks missed it 15:00 -opendevstatus:#oooq- NOTICE: Our Zuul gating CI/CD services will be offline starting around 14:00 UTC (in roughly two hours from now) in order to apply some critical security updates, and is not expected to remain offline for more than 30 minutes.13:41
marios|ruck|callthat's in like 20 mins? 13:41
bhagyashriszbr, https://projects.engineering.redhat.com/browse/TRIPLEOCI-50613:42
rlandyysandeep: w'ed 13:45
ysandeeprlandy: thanks!13:45
weshay|ruckmarios|ruck|call, anbanerj|rover ysandeep we think this is ready for review.. and then iterate.. https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3406013:46
marios|ruck|callweshay|ruck: ack adding to my list13:47
*** marios|ruck|call is now known as marios|ruck13:49
-opendevstatus- NOTICE: Our Zuul gating CI/CD services are being taken offline now in order to apply some critical security updates, and are not expected to remain offline for more than 30 minutes.13:55
*** pojadhav is now known as pojadhav|afk14:02
rlandyweshay|ruck: pojadhav|afk: ^^ on for meeting?14:15
rlandydropping14:16
ysandeepmarios|ruck, anbanerj|rover fyi.. I have created a temporary grafana dashboard from data picked by ruck_rover.py script.. sharing incase you find it useful for ruck/rovering/ reviewing patch : http://10.0.111.235:3000/d/9msvxrRnk/promotions-tracking?orgId=1&from=now-30m&to=now  , will pm login details14:26
ysandeepyou can filter for only failed job from drop down14:29
ysandeepselecting last 30 mins gives best results14:29
anbanerj|roverysandeep, thanks! Looks great!14:31
marios|ruckysandeep: thanks14:34
weshay|ruckrlandy, ah sorry.. pojadhav|afk pushed it to tomorrow14:41
anbanerj|roverysandeep, can I edit the dashboard a little?14:46
ysandeepanbanerj|rover, sure go ahead, this is just a temporary one.14:47
anbanerj|rovercool thanks14:47
rlandyweshay|ruck: np14:52
marios|ruckoh yeah zuul is down 14:53
marios|ruckgimmeZuulIntegrationPipelines nope14:53
anbanerj|roverupstream zuul is back now14:58
rlandytht was fast15:03
chandankumarso zuul also return 'LOST' as a result15:03
rlandythat's a new thing15:07
rlandyseen that since about two weeks ago15:08
rlandymarios|ruck: anbanerj|rover: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/797146 - going to merge that review15:11
rlandysince there are if else statements - should have little impact15:11
anbanerj|roverrlandy, ack15:12
marios|ruckrlandy: looking15:14
marios|ruckrlandy: k i haven't seen that before or done any kind of digging there. thanks for the heads up if there are issues it'll be during your day ;) so i know you'll try to deal with /revert etc15:15
rlandymarios|ruck: hoping not :)15:15
marios|ruckrlandy: if there is a bug would be nice to include it for next time but don't iterate on it now. 15:15
marios|ruckrlandy: anyway zuul is down so nothing will happen now15:15
marios|ruckrlandy: you might want to merge it 15:15
rlandyzuul is back15:15
marios|ruckbut...15:15
marios|ruckoh really?15:15
rlandyyes, really15:15
rlandyupstream and rdo zuul15:15
rlandydownstream not so much15:16
marios|ruckrlandy: wow i expected at least something to go wrong cos 'upgrade' what could possibly go wrong15:16
rlandymarios|ruck: cancel departure to pub?15:16
marios|ruckrlandy: work-from-pub?15:17
rlandyyep15:17
marios|ruckrlandy: i mean, they have fish n chips, burgers, and beer so... 15:17
marios|ruckand crisps15:17
rlandyand wifi15:17
rlandyyou're good to go15:17
marios|ruckrlandy: \o/15:17
rlandychandankumar: hey- so left you a message with promotable 16.2 hash15:19
rlandyzuul downstream is still down though15:20
rlandyoh - it just came back15:20
anbanerj|roverrlandy, this is the dlrn api right, where we see jobs that passed against a hash ? https://trunk-staging.rdoproject.org/api-centos-master-uc15:21
rlandyanbanerj|rover: not sure what exactly you are looking for ... but I think you want api/civotes_agg_detail.html?ref_hash=15:23
anbanerj|roverrlandy, like this https://trunk-staging.rdoproject.org/api/civotes_agg_detail.html?ref_hash=56c50364cfebb8b151c2314b7aced551 ?15:24
anbanerj|roveroh ok, got it now15:25
rlandyi don;t think you need staging either15:25
chandankumarrlandy: can we promote that?15:34
rlandychandankumar: ack you can15:34
rlandythe BM test failed15:34
rlandymachine needed a reboot15:35
rlandybut two OVB tests passed15:35
rlandyso good to go15:35
*** ysandeep is now known as ysandeep|out15:36
anbanerj|roverrlandy, sorry one more question https://trunk.rdoproject.org/api-centos8-ussuri/api/civotes_agg_detail.html?ref_hash - what is the corresponding url for master?  ttps://trunk.rdoproject.org/api-centos8-master did not work15:37
ysandeep|outfolks o/ I will be on PTO tomorrow 15:37
rlandyanbanerj|rover: -uc15:38
rlandylike you had it15:38
anbanerj|roveroooh. Thanks15:38
chandankumarysandeep|out: enjoy :-)15:38
rlandyhttps://trunk.rdoproject.org/api-centos8-master-uc/api/civotes_agg_detail.html?ref_hash15:39
ysandeep|outchandankumar, thanks! happy weekend team, see you on Monday15:39
chandankumarrlandy: want to jump on promoter and promote together15:42
chandankumar?15:42
chandankumarwith this hash http://osp-trunk.hosted.upshift.rdu2.redhat.com/api-rhel8-osp16-2/api/civotes_agg_detail.html?ref_hash=93bf835ed8672dcec32e052836842dd715:42
rlandysec15:45
rlandystarting some 17 jobs15:45
chandankumarok15:45
rlandychandankumar: k - nvm - mhu kicked the 17 line again15:49
rlandyk - joining the promoter15:49
chandankumarrlandy: https://meet.google.com/gwa-wgaa-qqe?authuser=2&hl=en15:51
rlandychandankumar: tuned off camera - maybe easier load for you?15:52
chandankumarrlandy: done15:52
rlandychandankumar: I hear you15:53
rlandychandankumar: sec15:54
rlandygetting hash15:54
chandankumarrlandy: ok, sure15:54
rlandychandankumar: http://osp-trunk.hosted.upshift.rdu2.redhat.com/api-rhel8-osp16-2/api/civotes_agg_detail.html?ref_hash=93bf835ed8672dcec32e052836842dd715:55
rlandyexpand to 25 entries15:56
rlandychandankumar: can I edit?15:56
chandankumarrlandy: yes please15:56
chandankumarrlandy: 35 is already commented 15:57
rlandychandankumar: ok - done15:59
chandankumarrlandy: let me run then15:59
rlandychandankumar: go4it15:59
rlandychandankumar: NO candidate hash promoted to current-tripleo?16:00
rlandydid I mess up?16:00
chandankumarrlandy: yes16:00
chandankumarlet me check what we missed16:01
rlandychandankumar: current-tripleo is an older  hash 670a8b5e1fe10c6b77598816d64d2afe16:02
rlandyshoudl promote16:02
marios|ruckweshay|ruck: anbanerj|rover: going in few16:02
chandankumarhttp://10.0.11.190/promoter_logs/redhat8_osp16-2_2021-06-24T16:01.log16:02
rlandychandankumar: 93bf835ed8672dcec32e052836842dd7 not in logs16:02
rlandychandankumar: how many hashes are we checking?16:03
rlandycan we look at config again?16:03
chandankumares16:03
chandankumaryes16:03
chandankumarit is checking last 10 hashes16:04
rlandychandankumar: ssh centos@10.0.108.19616:14
rlandyjoin tmux16:15
chandankumarrlandy: please share the terminal16:16
chandankumarrlandy: now it is back16:19
anbanerj|roverzbr, weshay|ruck have you seen this error before : The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'buildset_registry'16:19
rlandychandankumar: ok - so that script is there if you need it16:19
anbanerj|roverelastic-recheck-container which parents off opendev-build-docker-image is failing, pointing to this https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/pull-from-intermediate-registry/tasks/main.yaml#L1116:20
rlandy93bf835ed8672dcec32e..> 2021-06-23 13:19 16:20
rlandychandankumar: ok to drop?16:20
chandankumarrlandy: yes16:20
rlandychandankumar: k- dropping = ping if you need me16:21
chandankumarrlandy: why docker files is from 23rd ?16:21
chandankumarhttps://sf.hosted.upshift.rdu2.redhat.com/dockerfiles/redhat8/rhos-16.2/current-tripleo/16:21
rlandychandankumar: created yesterday16:22
rlandysame date as hash16:22
chandankumarrlandy: ah, ok, got it, thanks :-)16:22
chandankumarlet me check the image server issue one from current code16:23
rlandychandankumar: quick lunch - brb16:27
*** jpena is now known as jpena|lunch16:30
*** jpena|lunch is now known as jpena16:30
*** jpena is now known as jpena|off16:31
chandankumarok, see ya tomorrow16:32
* anbanerj|rover leaving for the day16:46
*** amoralej is now known as amoooralej|off16:55
weshay|ruckanbanerj|rover, log link?17:17
anbanerj|roverweshay|ruck, https://zuul.opendev.org/t/openstack/build/b90a5e3053ef45e6afcf3e328ec0a6dc19:25
anbanerj|roverI made the job non voting - else it will never merge https://review.opendev.org/c/opendev/elastic-recheck/+/729623/1919:25
weshay|ruckaye19:26
weshay|ruckk.. will look through it.. let's sync tomorrow19:26
anbanerj|roverweshay|ruck, cool, thanks19:26
*** dviroel is now known as dviroel|afk20:25
*** dviroel|afk is now known as dviroel20:54
*** dviroel is now known as dviroel|out21:29
rlandyweshay|ruck: ok - this works now ... https://logserver.rdoproject.org/53/18953/80/check/periodic-tripleo-ci-centos-8-standalone-master/68141a5/logs/undercloud/var/log/extra/package-list-installed.txt.gz22:31
rlandyhttps://review.opendev.org/c/openstack/tripleo-quickstart/+/79754122:31

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!