Wednesday, 2021-08-11

« Tuesday, 2021-08-10 Index Thursday, 2021-08-12 »
*** ykarel|away is now known as ykarel04:46
*** marios is now known as marios|ruck05:11
*** bhagyashris_ is now known as bhagyashris05:39
*** jpena|off is now known as jpena07:32
marios|rucksoniya29|rover: quick errand biab08:18
marios|ruckthanks soniya29|rover 09:03
akahatmarios|ruck, chandankumar hey.. i've come up with small idea about promoter logging solutions: https://paste.opendev.org/show/808005/09:21
akahatif there are no promotions then it will write centos8_master_last_run.log file. This will avoid creating promotion log files09:22
akahatand if we got promotion then we will log thatin centos8_master_<timestamp>.log09:22
akahatbhagyashris, ^^09:22
marios|ruckakahat: ish... ideally we'd have a logfile per day for the last few days09:24
marios|ruckakahat: can still captured a 'promoted' version when that happens as well as the normal log09:25
marios|ruckakahat: so you mean in your proposal we only keep 2 files? now and 'everything else'?09:28
marios|ruckakahat: i think it would make the 'everything else' file pretty big 09:28
akahatmarios|ruck, no.. that last_run.log file will get purged.. and updated with current run.09:30
akahatand for promotion we have file with the timestamp.09:31
marios|ruckakahat: but then i cant check yesterday logs?09:31
marios|ruckakahat: like i want to see if there is something we can promote eg few missing jobs09:31
akahatmarios|ruck, yes. you wont09:31
akahatokay. then something else i need to think.09:32
akahatsuggestions welcome.09:32
marios|ruckakahat: why don't you like the 'daily' approach 09:32
marios|ruckakahat: and we keep n days like 3/? 5? 09:32
marios|ruckakahat: AND if there is a promotion you create a new file called 'master_promoted_timestamp.txt' 09:33
marios|ruckas well as continue the daily log09:33
akahatmarios|ruck, okay.. you are saying we can remove the logs older than 4-5 days.09:33
marios|ruckand there should be a rainbow09:33
marios|ruckakahat: yeah i think so like some configurable number maybe but few days i don't think is useful after that09:33
marios|ruckakahat: bring it to the design/planning scrum tomorrow for more opinions though this is just mine i think rlandy agrees i think weshay has other ideas 09:34
akahatmarios|ruck, okay.. got your point. 09:34
akahatyeah.. sure. 09:34
akahatmarios|ruck, thank you for suggestions. :)09:35
bhagyashrisakahat, ack09:42
bhagyashrisfolks kindly add in your review lust https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3470609:42
marios|ruckack added to list bhagyashris 09:42
zbrmarios|ruck: i think i found another problem with get_hash. apparently Ansiballz  does not include non .py files when sending them to remote host, so our get_hash is unable to find config.yaml file.09:51
zbrsolution is to make it a python file.09:51
marios|ruckzbr: fantastic09:54
marios|ruckzbr: but lets solve/focus on the initial issue of not finding the module first09:54
marios|ruckzbr: before starting to dig into that09:54
zbrthat was sorted few minutes ago09:54
zbrand while sorting it, i found that issue with config.09:55
bhagyashrismarios|ruck, thanks :)10:04
soniya29|roverchandankumar, arxcruz, kopecmartin, please edit/add today's agenda for tempest's meeting - https://hackmd.io/fIOKlEBHQfeTZjZmrUaEYQ10:37
arxcruzsoniya29|rover: we have retro today, should we also have tempest meeting? (althoug isn't overlapping)10:38
soniya29|roverarxcruz, we have moved tempest meeting before retro meeting10:38
arxcruzyup, i know 10:39
arxcruzjust wondering10:39
arxcruzthat's fine 10:39
chandankumarsoniya29|rover: I donot have any agenda to discuss , I think hackmd needs cleanup10:40
soniya29|roverchandankumar, cleanup?10:42
zbrmarios|ruck: not glad to report an  IncompleteRead with mirror.bhs1.ovh.opendev.org, i did recheck.10:46
chandankumarsoniya29|rover: I have removed unnecessary items which we already discussed in last meeting10:47
chandankumarfrom hackmd10:47
marios|ruckthanks zbr 10:54
soniya29|roverchandankumar, okay11:03
marios|ruckzbr: have the link? 11:14
akahatReview request: https://review.opendev.org/q/topic:%22utilize-tripleo-operator%22+(status:open%20OR%20status:merged)11:23
akahatReview Request: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3433411:24
*** dviroel|out is now known as dviroel11:32
*** jpena is now known as jpena|lunch11:35
*** rlandy is now known as rlandy|ruck11:47
rlandy|ruckmarios|ruck: soniya29|rover: hey11:48
soniya29|roverrlandy|ruck, hello11:48
rlandy|ruckmarios|ruck: soniya29|rover: I have a clashing meeting with the first half of the program call11:49
marios|ruckzbr: rlandy|ruck: ack np11:49
rlandy|ruckmarios|ruck: soniya29|rover: I updated the doc11:49
marios|ruckrlandy|ruck: thanks i saw11:49
rlandy|ruckmarios|ruck: soniya29|rover: so I'll miss it but sure you both have it under control11:49
rlandy|ruckping if there are questions on downstream11:50
marios|ruckrlandy|ruck: might not even comment may just ask if there are questions since its all 'green' 11:50
marios|ruckrlandy|ruck: ack np11:50
weshay|ruckrlandy|ruck, did you like my cheer?11:50
rlandy|ruckmarios|ruck: yep - fly under the radar11:50
soniya29|roverrlandy|ruck, marios|ruck, we have tempest meeting as well11:50
rlandy|ruckweshay|ruck: loving it!!11:50
soniya29|rovermarios|ruck, shall I join program call or go on with tempest meeting?11:53
weshay|ruckmarios|ruck, I just realized we should probably report on wallaby too since that is officially imported11:58
weshay|ruckmind if I add it?11:58
soniya29|rovermarios|ruck, weshay|ruck, ^^?11:58
weshay|rucksoniya29|rover, go to tempest11:58
marios|rucksoniya29|rover: up to you11:58
marios|ruckweshay|ruck: sure also green 2 daysold but the  fs35 cix is not helping us11:59
soniya29|roverweshay|ruck, marios|ruck, ack11:59
weshay|ruckmarios|ruck, this is the tempest time out issue?11:59
weshay|rucksoniya29|rover, have you gotten on a node and looked at why tempest is timing out on us?12:00
weshay|rucksoniya29|rover, this one? https://trello.com/c/U1bKNUuu/2051-cixlp1939023tripleociproa-periodic-featureset-35-wallaby-times-out-running-tempest-2-hours12:00
marios|ruckweshay|ruck: yeah that one12:02
marios|ruckweshay|ruck: see upstream bug has info on the timings12:02
marios|ruckweshay|ruck: i have poked at it but can't see why it takes 2x as long run same tests as it did before 3rd august12:02
weshay|ruckmarios|ruck, have we held an environment?12:03
marios|ruckweshay|ruck: not yet12:03
soniya29|roverweshay|ruck, tempest meeting?12:03
rlandy|ruckweshay|ruck: adding the ephemeral heat settings did not help12:03
weshay|rucksoniya29|rover, once we get an environment held.. we'll need your help to understand why tempest is inconsistently timing out12:03
soniya29|roverweshay|ruck, sure12:03
weshay|ruckrlandy|ruck, aye12:03
rlandy|ruckweshay|ruck: its did reproduce the baremetal error12:03
rlandy|ruckthough12:03
rlandy|ruckso there is some combination of settings that is off12:04
rlandy|ruckhttp://osp-trunk.hosted.upshift.rdu2.redhat.com/api-rhel8-osp16-2/api/civotes_agg_detail.html?ref_hash=e55b584d3cad08c6e6cd850c986ada4212:04
rlandy|ruckmarios|ruck: weshay|ruck: ^^ going to promote that hash for 16.2 now12:04
rlandy|ruckqe jobs looks stuck12:04
weshay|ruckmarios|ruck, if you can.. getting an environment.. especially for timeouts.. is a great way to diagnose12:04
rlandy|ruckhttps://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/pipeline/job/pipeline_integration-pcci-16.2_dlrn-rhel-8.4-virthost-3cont_2comp_3ceph-ipv6-geneve-ceph/12:05
weshay|ruckrlandy|ruck, go 4 it12:05
weshay|rucksounds like imports are not getting turned on until sept.12:05
marios|ruckweshay|ruck: sure but also sounds like a good task for soniya29|rover perhaps 12:05
marios|ruckweshay|ruck: and reaching out to hold the node etc 12:05
marios|ruckweshay|ruck: can talk on the call after this one12:05
zbrmarios|ruck: happens again, same place: that is infra issue https://zuul.opendev.org/t/openstack/build/efc4a37164974cce98b128e203871abc12:15
bhagyashrisarxcruz, zbr, sshnaidm, rlandy|ruck , marios|ruck , ysandeep, bhagyashris, svyas, soniya29|rover , pojadhav, akahat, weshay|ruck , chandankumar, frenzy_friday, dviroel,12:16
bhagyashrisTripleO CI Retrospective meeting in 14 mins 12:16
bhagyashrishttps://miro.com/app/board/o9J_l2p9CCA=/12:17
bhagyashrishttps://meet.google.com/kkp-bejs-vvo?authuser=012:17
zbrmarios|ruck: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3AIncompleteRead12:28
zbryou have the pleasure to announce it to infra folks, sic.12:28
*** jpena|lunch is now known as jpena|off12:28
zbrand they want to ditch logstash,.... i wonder how they will find stuff like this.12:29
bhagyashrisdviroel, rlandy|ruck retro time12:31
dviroeljoining12:32
weshay|ruckrlandy|ruck, https://miro.com/app/board/o9J_l2p9CCA=/12:39
bhagyashrishttps://miro.com/app/board/o9J_l2p9CCA=/12:39
weshay|ruckarxcruz, !12:45
arxcruzweshay|ruck: ?12:46
weshay|ruckzbr, dry.. do not repeat.. meh. bad joke12:52
weshay|ruckmarios|ruck, soniya29|rover if you folks can get that last thing.. re: tempest and fs035 that would be awesome13:20
soniya29|roverweshay|ruck, i had discussed it in tempest meeting13:20
soniya29|roverweshay|ruck, i and arx cruz will be following that issue13:20
weshay|ruck++13:20
weshay|ruckthank you!!13:21
soniya29|roverweshay|ruck, marios|ruck, need to go out for an hour13:21
marios|ruckweshay|ruck: define 'get that last thing' :) 13:22
marios|ruckweshay|ruck: so i've been trying to dig there over last few days and added the findings in the bug13:23
marios|ruckweshay|ruck: tempest takes 2x as long as used to 13:23
*** ykarel is now known as ykarel|away13:23
marios|ruckweshay|ruck: i've tried to get soniya29|rover|brb to check before 'cos tempest' so glad the tempest folks will check it13:23
weshay|ruckk13:23
sshnaidmchandankumar, we don't use puppet-tempest somewhere for tripleo, right?13:23
weshay|rucktimeouts suck.. and almost impossible to figure out from just logs13:23
weshay|rucksshnaidm, no.. that is packstack13:24
chandankumarsshnaidm: yes, we donot use it, it is used only in packstack and puppet-openstack-integration13:24
sshnaidmweshay|ruck, packstack? is it alive?13:24
marios|ruckweshay|ruck: but its pretty clear in this case i mean ~2 hour mark the deployment is done and tempest starts... used to cmplete in 1 hour so ~3 total, now timeout after 2 hours so 4 total 13:24
chandankumarsshnaidm: yes, rdo team have weirdo jobs on that13:25
sshnaidmchandankumar, ack13:25
bhagyashrisfolks plz add into your review list https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34688 https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34706 which will help to proceed both upstream and downstream promoter work 13:27
bhagyashristhanks 13:27
rlandy|ruckweshay|ruck: now that 16.2 and 17 promoted, I can pick up the fs035 timeout13:28
rlandy|ruckmarios|ruck: ^^13:28
weshay|ruckdviroel, ping me later for training :) when you have time13:28
weshay|ruckrlandy|ruck, we just need an env.. to hand off to soniya29|rover|brb  to find which tempest test(s) are messing w/ us13:28
rlandy|ruckweshay|ruck: on it13:29
dviroelweshay|ruck: ack13:29
chandankumarweshay|ruck: rlandy|ruck we can tempest_run file13:30
chandankumarin any timedout job13:30
chandankumarthat will give some idea13:30
rlandy|ruckchandankumar: once the node is held?13:32
chandankumarrlandy|ruck: can you pass the timedout job link?13:32
rlandy|ruckchandankumar: yep - getting13:32
rlandy|ruckhttps://trello.com/c/U1bKNUuu/2051-cixlp1939023tripleociproa-periodic-featureset-35-wallaby-times-out-running-tempest-2-hours13:33
rlandy|ruckhttps://bugs.launchpad.net/tripleo/+bug/193902313:34
rlandy|ruckchandankumar: ^^ 13:34
rlandy|ruckalso any fs035 job13:34
chandankumarrlandy|ruck: https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-wallaby/193b290/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz13:34
chandankumarit might give some idea13:35
* chandankumar reading the whole bug13:36
rlandy|ruckchandankumar: also getting a node13:39
weshay|ruckzbr, help?13:41
weshay|ruckhave a min?13:41
weshay|ruckre: that patch13:41
marios|ruckthanks rlandy|ruck 13:42
rlandy|ruckchandankumar: holding https://review.rdoproject.org/r/c/testproject/+/24995 node13:43
rlandy|ruckeventually ovb will tear down13:44
rlandy|ruckbut tempest takes ages13:44
chandankumarrlandy|ruck: ack13:44
zbrweshay|ruck: send me link about click one and I will find the code you are looking for. i have ameeting in ten mins but i will do it today.13:50
weshay|ruckzbr, k.. just need to know how to get an args object for the passed args13:50
rlandy|ruckchandankumar: looks like your keys are on 38.102.83.11413:51
rlandy|ruckso you can get on it13:51
rlandy|ruckchandankumar: also there is a job in that state in the openstack-periodic-integration-stable1 queue right now13:51
rlandy|ruckyou can get on that node and look13:51
rlandy|ruckweshay|ruck: shocker https://review.rdoproject.org/r/c/testproject/+/18953 centos 9 container builds still passing13:52
weshay|ruckrlandy|ruck, ok.. perhaps we just add that one job to the os-$next line?13:53
rlandy|ruckforget that 13:54
rlandy|rucknode didn't kick13:55
weshay|ruckheh13:55
rlandy|ruckchandankumar: do you have the access you need14:03
rlandy|ruckthere are actually a bunch of fs035 jobs14:04
rlandy|ruckin action in rdo zuul now14:04
rlandy|ruckyou could access any one14:04
pojadhavzbr, do you have any idea why "mol-tripleo_common_integration" job is failing consistently : https://review.rdoproject.org/zuul/builds?job_name=mol-tripleo_common_integration14:14
pojadhavthis blocking my 2 patches : https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/34633 and https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3457214:14
pojadhavweshay|ruck, ^^14:15
weshay|ruckpojadhav, I added that to the ruck / rover tasks14:16
weshay|ruckpojadhav, https://review.rdoproject.org/zuul/builds?job_name=mol-tripleo_common_integration14:16
weshay|ruckpojadhav, it's not voting.14:16
weshay|ruckshouldn't block14:16
pojadhavweshay|ruck, yup.. is it voting in gate?14:17
weshay|ruckno14:17
weshay|ruckpojadhav, look again14:17
pojadhavok.. then i sould put a recheck 14:17
pojadhavthanks !14:17
marios|ruckweshay|ruck: sorry i missed the cix call i updated some things earlier that can be close dout14:52
marios|ruckweshay|ruck: i guess you didnt need me 14:52
marios|rucksorry :D14:52
weshay|ruckno worries14:53
weshay|rucknot much on the board14:53
marios|ruck weshay|ruck: ack thanks14:54
rlandy|ruckmarios|ruck: was fine14:57
rlandy|ruckyou did a great job of keeping the board up14:57
*** jpena|off is now known as jpena14:57
rlandy|ruckmarios|ruck: other than fs035 - anything else you want to hand off>14:58
weshay|ruckwe may not even need a sync / hand off mtg 14:59
weshay|ruckthings are cooking pretty well.. minus 03514:59
marios|ruckrlandy|ruck: nah was hoping to get lucky with 2 hashes for that fs35 https://review.rdoproject.org/r/c/testproject/+/34907 https://review.rdoproject.org/r/c/testproject/+/34916 still didn't report hoping one of them may pass fs35 it sometimes does just within the 4 hours itmeout14:59
weshay|ruckdviroel, FYI.. normally previous ruck/rovers meet w/ new ruck/rovers to live xfer work15:00
weshay|ruckmay skip that this time15:00
weshay|ruckbut good practice15:00
marios|ruckrlandy|ruck: they didn't report yet so maybe give them another spin if they are bad , if either of them passes then wallaby will promote that hash 15:00
marios|ruckrlandy|ruck: trying to keep wllaby alive despite that timeout15:00
marios|ruckrlandy|ruck: its how we've had some wallaby promotions lately :) luck !15:01
marios|ruckrlandy|ruck: i.e. https://bugs.launchpad.net/tripleo/+bug/1939023/comments/315:01
rlandy|ruckmarios|ruck: ack15:01
rlandy|ruckthrow enough spaghetti at the wall  - something might stick15:02
rlandy|rucklovely approach15:02
marios|ruckrlandy|ruck: right .. thank you !15:02
dviroelweshay|ruck: ok :)15:12
*** sshnaidm is now known as sshnaidm|afk15:35
*** jpena is now known as jpena|off15:42
marios|rucko/ weshay|ruck rlandy|ruck off in a couple mins15:54
marios|rucksoniya|rover: o/ congrats you made it ;) 15:54
rlandy|ruckmarios|ruck: enjoy the rest15:54
marios|ruckrlandy|ruck: are you becoming foreverruck like weshay|ruck :/15:55
rlandy|ruckit's life sentence15:55
soniya|rovermarios|ruck, congrats to you as well :)15:55
marios|ruckrlandy|ruck: tshirt maybe? ;) 'tripleo-ci' 18:55 < rlandy|ruck> it's life sentence15:56
marios|ruckbosu oclock15:57
*** dviroel is now known as dviroel|away16:19
zbrmarios|ruck: update: get_hash passed on some jobs but failed on others,... due to  No module named 'requests'. 16:23
*** ykarel is now known as ykarel|away16:33
*** rlandy|ruck is now known as rlandy|ruck|afk16:51
*** rlandy|ruck|afk is now known as rlandy|ruck18:59
*** dviroel|away is now known as dviroel19:09
dviroelweshay|ruck: o/ ready when you are19:13
*** ssamal is now known as ssamal|afk19:29
weshay|ruckdviroel, ah.. still avail?19:41
dviroelyes19:41
weshay|ruckmeet.google.com/rbo-nvyt-rvb19:41
weshay|ruckdviroel, ci-config/ci-scripts/infra-setup/roles/rrcockpit/files19:58
weshay|ruckdviroel, 20:05
weshay|ruckcockpit-bridge-249-1.fc33.x86_6420:05
weshay|ruckcockpit-system-249-1.fc33.noarch20:05
weshay|ruckcockpit-ws-249-1.fc33.x86_6420:05
weshay|ruckcockpit-networkmanager-249-1.fc33.noarch20:05
weshay|ruckcockpit-storaged-249-1.fc33.noarch20:05
weshay|ruckcockpit-packagekit-249-1.fc33.noarch20:05
weshay|ruckcockpit-249-1.fc33.x86_6420:05
weshay|ruckdviroel, http://localhost:9090/system/terminal20:05
weshay|ruckdviroel, https://launchpad.net/~tripleo20:18
weshay|ruckdviroel, https://launchpad.net/tripleo20:19
weshay|ruckdviroel, https://hackmd.io/07z0xroHTFi2IbX93P5ZfQ20:29
*** dviroel is now known as dviroel|ruck20:35
*** dviroel|ruck is now known as dviroel|ruck|out21:46
*** ssamal|afk is now known as ssamal22:25
rlandy|ruckweshay|ruck: we were meant to create a new sprint board22:31
rlandy|ruckafter tomorrow?22:31
rlandy|ruckafter planning?22:31
weshay|ruckrlandy|ruck, ya.. after planning23:21
rlandy|ruckweshay|ruck: ok23:21
rlandy|ruckweshay|ruck: left a comment re fs35 failure https://bugs.launchpad.net/tripleo/+bug/1939023/23:40
rlandy|ruckchandankumar: ^^ https://bugs.launchpad.net/tripleo/+bug/1939023/comments/6 pls see what you think23:41
rlandy|ruckI think our node went down already23:41
rlandy|ruckyou can reclaim one in your morning23:41
« Tuesday, 2021-08-10 Index Thursday, 2021-08-12 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!