Wednesday, 2021-11-03

« Tuesday, 2021-11-02 Index Thursday, 2021-11-04 »
*** amoralej|off is now known as amoralej07:26
jbadiapasshnaidm, have you got a minute regarding https://review.opendev.org/c/openstack/tripleo-quickstart/+/805384 ?08:40
jbadiapaI think I need to do some changes to go back to the v2, where the collection was installed on the install_deps.sh08:41
sshnaidmjbadiapa, hi, not sure what you mean..08:41
jbadiapahttps://review.opendev.org/c/openstack/tripleo-quickstart/+/805384/2/install-deps.sh#281 08:42
jbadiapaas chandan and marios said the patch as it is today is installing the tripleo-operator twice, from rpm and sources08:43
marioso/08:48
sshnaidmI don't think we use quickstart-extras-requirements.txt in CI, tbh, do we?08:53
sshnaidmmarios, chandankumar ^08:54
chandankumarsshnaidm: yes, we use https://opendev.org/openstack/tripleo-quickstart/src/branch/master/quickstart-extras-requirements.txt08:54
jbadiapahttps://opendev.org/openstack/tripleo-quickstart/src/branch/master/ci-scripts/get-node.sh#L5608:54
sshnaidmoh, right08:55
sshnaidmI think I'm fine with overriding operator collection with RPM for now08:56
sshnaidmnot the best, but as ykarel mentioned - let's unblock quickstart installation08:56
jbadiapawe can also add a libvirt-qe-requirements.txt and add it by default08:58
sshnaidmjbadiapa, idk what it is08:58
sshnaidmchandankumar, marios please re-review https://review.opendev.org/c/openstack/tripleo-quickstart/+/805384 - seems like currently it blocks quickstart.sh installation08:59
mariossshnaidm: i am not blocking there I have +1 and it has enough votes so feel free to workflow . as commented unless there is some technical reason we can't do it with collections, then I'd prefer that09:04
marioschandankumar: ^^09:04
chandankumarmarios: I would merge this patch, will follow up on this by installing via collection09:05
chandankumarin a new review09:06
jbadiapachandankumar, https://review.opendev.org/c/openstack/tripleo-quickstart/+/805384/2 ? 09:07
chandankumarjbadiapa: yes, I like the sagi idea, we can extend this https://opendev.org/openstack/tripleo-quickstart/src/branch/master/install-deps.sh#L278 to accept tripleo-operator ansible from cli and keep tripleo-repos as default .09:10
*** jpena|off is now known as jpena10:09
*** chem is now known as Guest480210:20
*** rlandy is now known as rlandy|ruck10:35
rlandy|rucksshnaidm: hey - thanks for fixing the nginx issue - can you show me what patch you put in?10:36
sshnaidmrlandy|ruck, hey, I don't have a patch yet, fixed it on server10:37
rlandy|rucksshnaidm: k - what was the fix?10:38
rlandy|ruckwe was deleting stuff all over the place :)10:38
rlandy|ruckwes10:38
sshnaidmit's nginx network=host vs port 80 forward..10:38
sshnaidmrlandy|ruck, I'll try to prepare a patch that will work for both up and down grafanas10:39
rlandy|rucksshnaidm++ thank you 10:39
rlandy|ruckthen we can take care of this on our own next time 10:40
rlandy|ruckfrenzy_friday: hey - can you update https://trello.com/c/hs0TLPwL/2210-cixlp1949327tripleociproa-cant-run-container-rabbitmqwaitbundle-could-not-prefetch-rabbitmquser-provider-rabbitmqctl-command-is10:53
rlandy|ruckand associated bug10:53
rlandy|ruckwith status10:53
chandankumararxcruz: kopecmartin can we cancel today's tempest meeting since sonia is not around I think?10:59
kopecmartinchandankumar: sure10:59
frenzy_fridayrlandy|ruck, ack, lemme recheck 007 to get a new log.11:13
*** dviroel|out is now known as dviroel|rover11:21
frenzy_fridayI have a question - https://review.rdoproject.org/r/c/rdo-jobs/+/36507 is rebased on master, but it still shows as merge conflict, Why is that?11:22
dviroel|roverfrenzy_friday: maybe your local repo is not up to date11:25
dviroel|roverfrenzy_friday: should be on top of https://github.com/rdo-infra/rdo-jobs/commit/3fa53113c0a8182a37625d81f9819cdd42ce145111:26
dviroel|rovermorning team o/11:26
dviroel|roversshnaidm: hi o/, did the cockpit broken because of the compose update?11:28
rlandy|ruckmarios: dviroel|rover: hey - looking to merge https://review.opendev.org/c/openstack/tripleo-repos/+/81599111:29
dviroel|roversshnaidm: wrt removing the 'network_mode: host'11:29
rlandy|ruckthere is still a comment on sanity check11:29
sshnaidmdviroel|rover, yep11:29
rlandy|ruckdviroel|rover: not your patch11:30
mariosrlandy|ruck: o/ sure just inline question is why i didnt' +A it but it is good to go i think at least couldn't see any issue11:30
rlandy|rucknginx config11:30
rlandy|ruckok - merging11:30
mariosrlandy|ruck: dviroel|rover: hitting the workflow but dviroel|rover see inline question please thanks11:30
dviroel|rovermarios: rlandy|ruck going through comments atm11:30
mariosdviroel|rover: ack nothing urgent 11:31
dviroel|roverdone11:35
rlandy|ruckarxcruz: hey - looking to merge https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/816350/1/roles/collect_logs/tasks/sova.yml - can you address sshnaidm's comments?11:36
frenzy_fridaydviroel|rover, thanks, it works now11:37
arxcruzrlandy|ruck: i'll update the patch 11:37
rlandy|ruckthank you11:37
arxcruzneed a few minutes though 11:37
dviroel|roversshnaidm: next thing to try is removing 'ports:      - "80:80"', also works in local env, and nginx continues to work in network host11:39
dviroel|roverand adding back network host11:40
sshnaidmdviroel|rover, yeah, either changing 127.0.0.1 in nginx config to "grafana", but then need to use docker DNS.. Probably better to try --host 11:40
sshnaidmdviroel|rover, lemme try it in downstream.. <- rlandy|ruck, might be down for some time11:40
rlandy|rucksshnaidm: np11:40
rlandy|ruckfrenzy_friday: hey - can you rebase https://review.rdoproject.org/r/c/rdo-jobs/+/36525?11:41
rlandy|ruckthen we can merge11:41
frenzy_fridayrlandy|ruck, ack, rebasing11:42
marioschandankumar: looks like it works now with the hack in containers.conf11:44
chandankumarmarios: yeye11:44
marioschandankumar: sshnaidm: but i think we should get the 'proper' fix with sagi patches 11:45
marioschandankumar: worst case if there is some delay we can merge that for now https://review.opendev.org/c/openstack/tripleo-ci/+/816441 11:45
chandankumarmarios: yup, sure :-)11:45
marioschandankumar: please don't update it yet though i am waiting for result and fs1 to run 11:46
rlandy|ruckmarios: chandankumar: frenzy_friday: arxcruz: dviroel|rover: it's only the few of us today for the retro ...11:46
rlandy|ruckso you want to go ahead or move it to monday?11:46
rlandy|ruckwhere we have two hours booked for planning11:47
rlandy|ruckwe can retro in the first 11:47
mariosrlandy|ruck: "it was a great sprint" , "still living the dream" etc .. /END11:47
rlandy|ruckand plan in the second11:47
mariosrlandy|ruck: yeah sure fine by me11:47
rlandy|ruckmarios: keep or move11:47
rlandy|ruckplease vote11:47
mariosrlandy|ruck: i mean doesn't make sense to have just 4 people11:47
mariosrlandy|ruck: i'd say move for more people11:47
dviroel|roverrlandy|ruck: move +111:47
mariosrlandy|ruck: and we can make it quick cos it was a weird sprint anyway/ptg etc11:47
frenzy_friday+1 to moving to Monday. /me has 20% hearing ability today11:47
rlandy|ruckok - pls see monday's two hour meeting for planning11:48
rlandy|ruckyou should all have that on your calendars11:48
mariosrlandy|ruck: check11:49
frenzy_fridayrlandy|ruck, https://review.rdoproject.org/r/c/rdo-jobs/+/36525 rebased11:50
chandankumardviroel|rover: you might be interested in this one https://pagure.io/compose-utils11:51
dviroel|roverchandankumar: tks, i'll take a look11:54
rlandy|ruckdviroel|rover: I am on the program call12:01
rlandy|ruckyou are welcome to join or skip12:01
rlandy|ruckyour choice12:01
rlandy|ruckinfo is updated12:01
dviroel|roverrlandy|ruck: will skip today then, ok for you?12:04
rlandy|ruckdviroel|rover: sure12:04
dviroel|roverrlandy|ruck: noisy here12:04
dviroel|rovertoday12:04
rlandy|ruckdviroel|rover: have a good holiday?12:04
dviroel|roverrlandy|ruck: yes, had a time with mom/dad, was good12:05
rlandy|rucknice12:05
dviroel|rover:)12:05
dviroel|roverrlandy|ruck: anything that i should pay attention atm?12:05
dviroel|roveri saw that introspection 'fix' was merged12:05
rlandy|ruckdviroel|rover: no sure we have a fix yet12:06
rlandy|ruckwe are promoting12:06
rlandy|ruckbut that test is skipped12:06
dviroel|roveryep12:06
rlandy|ruckdviroel|rover: the other thing to look at:12:06
rlandy|ruckwhether we have fixed update/upgrade in check/gate12:06
rlandy|ruckwe had a few failures yesterday12:06
dviroel|roveroh, ok, let me check that12:06
rlandy|ruckbut zuul was getting restarted12:07
rlandy|ruckit could still be an issue12:07
rlandy|ruckdviroel|rover: downstream came back - waiting on 17 to promote12:07
rlandy|ruckwe have an issue with missing images12:07
rlandy|ruckchandankumar and I will be checking into that afterwards12:07
dviroel|roverack12:07
rlandy|ruckdviroel|rover; this afternoon, I'll take you through the downstream cockpit12:09
rlandy|ruckso you can help there :)12:09
dviroel|roverrlandy|ruck: great :)12:09
rlandy|ruckI'm somewhat hopeless in that department12:09
rlandy|ruckalso rafael will be back12:09
rlandy|ruckhe also had a holiday yesterday12:10
dviroel|roverrlandy|ruck: oh, yeah, same holiday12:10
rlandy|ruckdviroel|rover: marios: arxcruz: tomorrow there will be very few of us at scrum - I'd like to run through two things: what's next for tripleo-repos and c9 cockpit design(whatever you have)12:11
rlandy|rucksound ok?12:11
mariosk12:12
dviroel|roveryes12:12
chandankumararxcruz: rlandy|ruck: can we get a panel containing c8 and c9 jobs for comparison to see what is working in c8 and c912:12
chandankumarside by side12:12
rlandy|ruckchandankumar: ack - we'll talk about it12:13
rlandy|ruckarxcruz may be out then for openstack foundation12:13
rlandy|rucklet's see12:13
arxcruzyes, i'll be out12:13
rlandy|ruckotherwise we can have a design meeting later12:13
arxcruzrlandy|ruck: chandankumar it's ok to have a panel12:13
arxcruzchandankumar: i need the list of components that we are running jobs now on c9 12:13
arxcruzi was about to ask you that :) 12:13
rlandy|ruckall12:13
rlandy|ruckshould be running12:13
arxcruzok 12:14
arxcruzfor c9 right ?12:14
chandankumararxcruz: https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/component-pipeline-main-centos9.yaml12:14
chandankumarall cs9 jobs are in a file12:14
arxcruzchandankumar: cool, thanks 12:14
marioschandankumar: \o/ now new fail for fs1 :/ https://logserver.rdoproject.org/67/36267/22/check/tripleo-stream9-development-centos-9-ovb-3ctlr_1comp-featureset001-master/dd05362/logs/undercloud/home/zuul-worker/install_packages.sh.log.txt.gz 2021-11-03 08:13:51 |  Problem: package python3-tripleoclient-17.1.1-0.20211101131424.ce6a555.el9.noarch requires buildah, but none of the providers can be installed12:25
chandankumarmarios: awesome12:26
marioschandankumar: might be related to the container module12:26
marioschandankumar: yeah i am really happy about it 12:26
chandankumarlooking into that in abit12:27
Tengurlandy|ruck: hey, congrats on your 10y :)12:28
rlandy|ruckTengu: thanks - can't believe it's been so long- feels like I started yesterday :012:29
Tengu:)12:30
dviroel|roverchandankumar: i started to add support for down-url in yum-config here: https://review.opendev.org/c/openstack/tripleo-repos/+/81621112:34
arxcruzwow, 10 years!12:34
dviroel|roverchandankumar:  but if you want to change the proxy url, still need to do a sed in the end. Thinking on how to add that to yum-config. So you can download the repo file and proxy a different proxy to used in all urls12:35
dviroel|rovers/and proxy/and replace with12:36
rlandy|ruckchandankumar: hey - off meeting on the internal promoter12:37
rlandy|ruckon tmux12:37
rlandy|ruckjoining review time gmeet12:37
chandankumaramoralej: container-tools latest [e] common   Most recent (rolling) versions of podman, buildah, skopeo, runc, conmon, runc, conmon, Udica, etc as well as dependencies such as container-selinux built and tested together, and updated as frequently as every 12 weeks.12:39
chandankumarsorry12:39
chandankumarmarios: container-tools latest [e] common   Most recent (rolling) versions of podman, buildah, skopeo, runc, conmon, runc, conmon, Udica, etc as well as dependencies such as container-selinux built and tested together, and updated as frequently as every 12 weeks.12:39
chandankumarmarios: might be using old compose12:39
chandankumaron my local container, I am able to pull buildah12:40
chandankumarrlandy|ruck: joining12:40
rcastillohello12:43
dviroel|roverrcastillo: morning o/12:47
rcastillodviroel|rover o/12:48
dviroel|roverrcastillo: had a good holiday?12:48
rcastilloyup, you?12:49
marioschandankumar: k thanks getting food and will check in a bit12:49
dviroel|roverrcastillo: yeah, me too12:49
*** jbuchta is now known as jbuchta|food12:56
rlandy|ruckrcastillo: hey - sorry to get you up early12:57
rlandy|ruckretro moved12:57
rlandy|ruckto monday12:57
rlandy|ruckcan chat with you this afternoon ( my time)12:57
rcastillono worries13:06
sshnaidmdviroel|rover, I think we can try network_mode: host now, worked on server13:06
*** amoralej is now known as amoralej|lunch13:08
dviroel|roversshnaidm:  what was your change? added back network_mode and removed ports?13:09
sshnaidmdviroel|rover, yes13:09
dviroel|roversshnaidm: ok, that works in dev env too13:09
chandankumarmarios: sshnaidm https://review.rdoproject.org/r/c/rdo-jobs/+/36549 good to go13:18
chandankumardviroel|rover: https://review.rdoproject.org/r/c/rdo-jobs/+/3654913:18
chandankumardviroel|rover: I have seen the patch of wiring up, but not reviewed it yet, will look on monday13:18
dviroel|roversshnaidm: uploading a patch to fix that13:18
dviroel|roverchandankumar: ack13:19
chandankumarrlandy|ruck: please keep an eye on these two patches: https://review.opendev.org/c/openstack/tripleo-common/+/816321 and https://review.opendev.org/c/openstack/tripleo-ci/+/81631813:19
chandankumarwill log out now today,13:19
chandankumarHappy Diwali people :-)13:19
chandankumarsee ya on monday!13:19
marioshave a good one chandankumar 13:19
dviroel|rovero/ see ya13:19
rlandy|ruckchandankumar: enjoy the holiday!!13:20
dviroel|roversshnaidm: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3655013:22
*** abregman|pto is now known as abregman13:33
rlandy|ruckdviroel|rover: arxcruz: going to invite you to the dashboard session for the prod chain meeting next week 13:40
dviroel|roverok13:40
arxcruzok 13:42
rlandy|ruckwed: 9:10 - 9:5513:43
rlandy|ruckDashboards13:43
rlandy|ruckArie/David13:43
rlandy|ruckeastern time13:43
rlandy|ruckso you can translate that13:43
arxcruzrlandy|ruck: https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/816350 is ready and already with sshnaidm blessing 13:55
rlandy|ruckmarios; can you give this a quick review ^^13:56
rlandy|ruckthen I'll merge13:56
mariosrlandy|ruck: arxcruz: did a quick review can't spot something wrong minor comment for your consideration arx 14:00
arxcruzmarios: thanks, I think this way is better even though it's more verbose, but it is easier to check for errors, the ternary was failing and it was pretty hard to debug, this way we know exactly the block that will fail 14:02
arxcruzif you don't mind, I would like to keep as it is :) 14:02
mariosarxcruz: yeah +2 already so no objection just suggestion as i said14:02
mariosarxcruz: personally i would have tried that first before doing it like this but ... no objection14:03
arxcruzmarios: i just follow what sshnaidm sugested :) 14:03
*** jbuchta|food is now known as jbuchta14:07
rlandy|ruckwoohoo 17 promoted14:11
rlandy|ruckarxcruz: marios: thanks - merging that14:11
dviroel|roverawesome \o/14:12
rlandy|ruckdviroel|rover: and looky here https://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/view/phase-1-pipelines/14:13
rlandy|ruckwe should take a screen shot of that14:13
rlandy|ruckdviroel|rover: arxcruz: dashboard got moved to monday 14:13
arxcruzok14:14
rlandy|ruck9:10 - 9:55 eatsern14:14
*** amoralej|lunch is now known as amoralej14:14
dviroel|roverwon't conflict with our planning?14:15
rlandy|ruckdviroel|rover: that's an hour earlier14:15
rlandy|ruck in UTC14:15
rlandy|ruckwe move the clocks on saturday night14:15
dviroel|roverah ok14:15
* rlandy|ruck needs to start at 6am now14:19
dviroel|roverrlandy|ruck: :(14:19
rlandy|ruckrealy 5:30am14:20
rlandy|ruckbut idk about that14:20
mariosdviroel|rover: hey do you think that is OK? ('module_hotfixes') @ https://review.opendev.org/c/openstack/tripleo-quickstart/+/816313/3/config/release/tripleo-ci/CentOS-9/promotion-testing-hash-master.yml#137 ? I checked and it seems supported already https://opendev.org/openstack/tripleo-repos/src/commit/da713cb3593018485e5d784e2b5523c416ab3845/plugins/module_utils/tripleo_repos/yum_config/constants.py#L34 14:31
dviroel|rovermarios: checking14:33
mariosdviroel|rover: thanks mostly sanity check before as i started a new test for it (https://review.rdoproject.org/r/c/testproject/+/36267 ) 14:34
dviroel|rovermarios: yum-config should work fine, but we don't enabled it for C9 yet14:35
dviroel|rovermarios: only wallaby so far. And since it is stable now, will start enabling on other branches too.14:36
mariosdviroel|rover: oh you mean, the contents of https://review.opendev.org/c/openstack/tripleo-quickstart/+/816313/3/config/release/tripleo-ci/CentOS-9/promotion-testing-hash-master.yml#135 are not laid down with tripleo-repos yet? i.e. still doing it the 'old' way? 14:36
dviroel|rovermarios: yes, we need to enable a flag in release file14:36
dviroel|roveruse_yum_config_module: true14:36
mariosdviroel|rover: ack ok fine thanks for checking then should not be any issue14:36
mariosdviroel|rover: its ok, mainly i just want to check if that flag will help me get passed a new package conflict issue we are seeing14:37
dviroel|roverack14:37
dviroel|roverrlandy|ruck: should i continue as rover until monday? next rr will be on holidays, right?14:38
rlandy|ruckdviroel|rover: you don't have to14:38
rlandy|ruckI'll try cover it14:38
rlandy|ruckif I am drownng , I will ping for help14:38
rlandy|ruckdviroel|rover: tomorrow, we can chat at scrum at pinning and tripleo-repos etc.14:39
dviroel|roverrlandy|ruck: don't worry, i can help until they are back14:39
rlandy|ruckyou may want to get back to that14:39
rlandy|rucklet's see how we go14:39
dviroel|roverrlandy|ruck: yes, want to start that asap14:40
rlandy|ruckdviroel|rover: so fs035 on train ... 14:45
rlandy|ruckperiodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-trainopenstack/tripleo-cimasteropenstack-periodic-integration-stable43 hrs 18 mins 41 secs2021-11-03 07:02:14SUCCESS14:45
rlandy|rucklast run14:45
dviroel|roveryes14:45
rlandy|ruckbut that was the first good one14:46
dviroel|roverthe one before14:46
dviroel|roverfailed as timed out14:46
rlandy|ruckso I think we leave the CIX open14:46
rlandy|ruckwatch it 14:46
rlandy|ruckit was sporadic anyways14:46
dviroel|roverbut if you search for 'introspect node', it finished in 4 minutes14:46
rlandy|ruckthat is good14:47
dviroel|rover^ which is a good sign14:47
dviroel|roveryeah14:47
rlandy|ruckI think we are good to go14:47
dviroel|roverlets keep watching14:47
rlandy|ruckwill close it on monday 14:47
dviroel|roveryes14:47
dviroel|roverrlandy|ruck: wrt upgrade fails14:48
dviroel|roverhttps://6960c5fda7b7a7dfd328-b460799f36e134cb63bd04ce908dfbb3.ssl.cf2.rackcdn.com/814801/6/gate/tripleo-ci-centos-8-undercloud-upgrade/3bb41c7/14:48
rlandy|ruckthe downstream promoter works but the logging is off: http://10.0.110.143/promoter_logs/redhat8_osp17_2021-11-03T14:04.log14:48
* rlandy|ruck looks14:48
dviroel|roverrlandy|ruck: failed in a similar issue https://6960c5fda7b7a7dfd328-b460799f36e134cb63bd04ce908dfbb3.ssl.cf2.rackcdn.com/814801/6/gate/tripleo-ci-centos-8-undercloud-upgrade/3bb41c7/logs/undercloud/home/zuul/undercloud_upgrade.log14:49
rlandy|ruckone sec14:49
rlandy|ruckrestarting victoria14:49
dviroel|roverarxcruz: fyi https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3655014:49
rlandy|ruckoh you have it14:49
rlandy|ruckor someone does14:50
arxcruzdviroel|rover: i told you host is required upstream :P 14:50
rlandy|ruckthanks https://review.rdoproject.org/r/c/testproject/+/3648914:50
dviroel|roverarxcruz: yes, but now removing 'ports' works everywhere14:50
dviroel|roverarxcruz: :P14:50
rlandy|ruckNot found image: http://104.130.158.204:5001/v2/tripleomaster/openstack-ironic-pxe/manifests/88a532a1da8ec5c42541a6f4b9c06af5", "params": {}, "success": false}14:51
rlandy|ruckdviroel|rover: ^^ that was the same error before we changes the hash14:51
rlandy|ruckchecking the hash of the provider14:51
rlandy|ruckhttps://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/81480114:52
rlandy|ruckugh  zuul summary doesn;t log it14:52
rlandy|rucksshnaidm: dviroel|rover: should we pull this change into the cockpit when it merges? 14:54
rlandy|ruckupstream? and downstream?14:54
rlandy|ruckalready there14:54
rlandy|ruck?14:54
dviroel|roverrlandy|ruck: sshnaidm already tested it in downstream, might be manually changed there14:56
sshnaidmrlandy|ruck, let's wait until it's updated automatically14:57
rlandy|rucksshnaidm: the cockpit does not pull changes14:57
rlandy|ruckautomatically14:57
sshnaidmrlandy|ruck, why?14:58
rlandy|ruckcron is not set up there14:58
rlandy|ruckwe need to go stop containers and git pull, git rebase14:58
rlandy|ruckI think to prevent random changes getting in14:59
rlandy|ruckso if it's ok to go in the upstream, we can do that process on the upstream cockpit14:59
rlandy|ruckafter it merges14:59
sshnaidmrlandy|ruck, well, in upstream it has cron enabled, it just fails to update..15:00
sshnaidmit's still centos7 there, omg15:00
* rlandy|ruck gets on cockpit15:00
rlandy|rucksshnaidm: k - for now - let's just wait for this change to merge15:03
* dviroel|rover lunch, biab15:03
rlandy|ruckthen will pull it in on the upstream cockpit15:03
rlandy|ruckafter CIX call15:03
rlandy|ruck#Ansible: Run ansible-pull for continuous delivery15:05
rlandy|ruck*/5 * * * * flock -n -E 2 /var/lock/ansible-pull /usr/local/bin/ansible_pull.sh15:05
rlandy|ruckoh I see15:05
mariosrlandy|ruck: sshnaidm: chandankumar: fyi filed https://bugs.launchpad.net/tripleo/+bug/1949601 15:07
marios(c9 fs1/fs2 ovb new fail)15:07
rlandy|ruckmarios: should that be promotion-blocker?15:08
mariosrlandy|ruck: ack adding 15:08
* rlandy|ruck checks CIX board ahead of meeting15:09
rlandy|ruckhttps://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3655015:09
rlandy|rucklol ok - failed15:09
rlandy|ruck2021-11-03 15:05:59.576266 | rdo-centos-8 | /home/zuul/src/review.rdoproject.org/rdo-infra/ci-config/.tox/py36/lib64/python3.6/site-packages/coverage/data.py:118: CoverageWarning: Data file '/home/zuul/src/review.rdoproject.org/rdo-infra/ci-config/.coverage.node-0001739961.42889.552121' doesn't seem to be a coverage data file: Couldn't use data file '/home/zuul/src/review.rdoproject.org/rdo-infra/ci-config/.coverage.node-000115:10
rlandy|ruck739961.42889.552121': no such table: coverage_schema15:10
rlandy|ruckwe lost the 16.2 vs train comaparison on downstream15:19
rlandy|ruckfor the component line15:20
rlandy|ruckrcastillo: hey ... https://meet.google.com/oay-bqmr-fwg?pli=1&authuser=016:05
*** chem is now known as Guest482216:06
sshnaidmrlandy|ruck, dviroel|rover upstream grafana now is fixed and should always update automatically (if nothing else breaks)16:14
dviroel|roversshnaidm: great thanks!16:15
sshnaidmdownstream journal is full with errors like https://paste.opendev.org/show/810365/16:15
sshnaidmdviroel|rover, can you take a look? ^16:16
dviroel|roversshnaidm: sure, let me try that locally, I think that my env was reproducing that16:17
sshnaidmdviroel|rover, yeah.. maybe it's because repo doesn't update there..16:17
*** amoralej is now known as amoralej|off16:17
sshnaidmdviroel|rover, also this one https://paste.opendev.org/show/810366/16:20
dviroel|roversshnaidm: if it is c7, need to update ca-certificates16:22
sshnaidmthere is an ansible process stuck from Jun2916:23
sshnaidmain't nice16:24
rlandy|rucksshnaidm: good news16:26
rlandy|rucksshnaidm: dviroel|rover: downstream we aren't picking up the upstream components16:27
rlandy|ruckdviroel|rover: might be the errors you are seeing16:27
rlandy|ruckhttp://tripleo-cockpit.usersys.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline?orgId=116:28
rlandy|ruck^^ broken16:28
dviroel|roverrlandy|ruck: yes, failing in rr tool command16:28
rlandy|ruckdviroel|rover: cert one is ok16:28
rlandy|ruckit works with that error16:28
rlandy|ruckI checked it with wes16:28
rlandy|ruckbut the rr thing is broken16:28
rlandy|ruckwe need a fix there16:29
sshnaidmrlandy|ruck, did you update downstream cockpit via git?16:29
rlandy|rucksshnaidm: yes yesterday - but not since then16:30
* rlandy|ruck just getting out of meetings :)16:30
sshnaidmrlandy|ruck, I mean - via patches?16:30
rlandy|ruckno patches16:31
rlandy|ruckonly master16:31
* rlandy|ruck not sure what you are asking16:31
rlandy|ruckNov 03 00:00:01 incockpit dockerd-current[3547]:     url = config[stream]['criteria'][distro][release]['comp_url']16:32
rlandy|ruckNov 03 00:00:01 incockpit dockerd-current[3547]: KeyError: 'upstream'16:32
sshnaidmrlandy|ruck, quick call?16:32
rlandy|ruckyep16:32
sshnaidmif you're out of meetings, there is a time! :D16:32
rlandy|ruckhttps://meet.google.com/ygn-kfkq-yfh?pli=1&authuser=016:32
rlandy|ruckdviroel|rover: sshnaidm: ^^16:32
*** marios is now known as marios|out16:43
dviroel|roverrlandy|ruck: https://code.engineering.redhat.com/gerrit/gitweb?p=openstack/rrcockpit.git;a=blob;f=files/telegraf_py3/conf_ruck_rover.yaml;h=2d0341afac1015106392a7e900dcd5dab17b671f;hb=HEAD17:06
dviroel|roverrlandy|ruck: sshnaidm https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/rrcockpit/files/grafana/Dockerfile17:22
*** jpena is now known as jpena|off17:48
rlandy|rucklunch - brb17:58
dviroel|roverchasing ussuri and train18:52
rlandy|ruckthanks19:06
rlandy|ruckdviroel|rover: any conclusions about update/upgrades19:18
rlandy|ruckotherwise will look at that more19:18
rlandy|rucktripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-train19:19
rlandy|ruckjust failed19:19
dviroel|rover^ https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_cb1/816346/1/gate/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-train/cb12da9/logs/undercloud/home/zuul/overcloud_update_data_migration.log19:20
dviroel|roverrlandy|ruck: 19:20
rlandy|ruck2021-11-03 17:45:59 | 2021-11-03 17:45:59.627 190233 ERROR openstack [-] Failed to generate inventory: tripleoclient.exceptions.InvalidConfiguration: Failed to generate inventory[00m19:21
rlandy|ruckdifferent error19:22
dviroel|roveryep, do we have more of this?19:22
rlandy|ruckcheck other logs19:24
rlandy|ruck2021-11-03 17:45:20.221891 | primary | TASK [tripleo-upgrade : run online data migration] *****************************19:24
rlandy|ruck2021-11-03 17:45:20.221910 | primary | Wednesday 03 November 2021  17:45:20 +0000 (0:00:00.034)       0:10:23.103 ****19:24
rlandy|ruck2021-11-03 17:45:59.791279 | primary | fatal: [undercloud]: FAILED! => {19:24
rlandy|ruckidk - we usually don't see the error there19:24
rlandy|ruckhttps://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-train19:25
rlandy|ruckbeen clean for some time19:25
rlandy|ruckpretty green test19:25
dviroel|roveryes19:25
dviroel|roverwe need to wait next run19:26
rlandy|ruckdviroel|rover: k - digging more19:26
dviroel|roverk19:26
dviroel|roverso, the only one that fails in upgrades is this one so far: https://6960c5fda7b7a7dfd328-b460799f36e134cb63bd04ce908dfbb3.ssl.cf2.rackcdn.com/814801/6/gate/tripleo-ci-centos-8-undercloud-upgrade/3bb41c7/logs/undercloud/home/zuul/undercloud_upgrade.log19:33
* dviroel|rover need to solve an errand, will be back later...19:45
*** dviroel|rover is now known as dviroel|rover|afk19:45
rlandy|rucksure19:50
*** chem is now known as Guest484820:59
*** dviroel|rover|afk is now known as dviroel|rover21:29
dviroel|roverrlandy|ruck: one more https://7fd22f456ac9c994eaa4-c86e09fdca941b34a3fcb115361b332e.ssl.cf1.rackcdn.com/816244/2/gate/tripleo-ci-centos-8-undercloud-upgrade-wallaby/54d9ad8/logs/undercloud/home/zuul/undercloud_install.log21:30
rlandy|ruck"Not found image: http://158.69.74.140:5001/v2/tripleovictoria21:31
rlandy|ruckso there we have the first hash21:31
rlandy|ruckwe only fixed part of the problem21:31
rlandy|ruckf9563b13d92dcc5b99bff7a41699f6bd21:32
rlandy|ruck5fd2c0a5ce28b1c2f48eaa4636518afe21:33
rlandy|ruckcurrent-tripleo21:33
rlandy|ruckhttps://trunk.rdoproject.org/centos8-victoria/current-tripleo/delorean.repo.md521:34
rlandy|ruckoops 21:34
rlandy|ruckf9563b13d92dcc5b99bff7a41699f6bd21:34
rlandy|ruckf9563b13d92dcc5b99bff7a41699f6bd/2021-11-03 18:43 -  21:35
dviroel|roverUNDERCLOUD_TARGET_HASH=5fd2c0a5ce28b1c2f48eaa4636518afe21:35
dviroel|roveryep21:35
rlandy|ruckhttps://trunk.rdoproject.org/centos8-victoria/current-tripleo/5f/21:36
rlandy|ruckhttps://trunk.rdoproject.org/centos8-wallaby/current-tripleo/delorean.repo.md521:37
rlandy|ruckwallaby21:37
rlandy|ruck5fd2c0a5ce28b1c2f48eaa4636518afe21:37
rlandy|ruckchecking content-provider21:37
rlandy|rucktripleovictoria/openstack-ovn-controller:bb05cd497dfcae51f78d885d8277471e - victoria21:38
dviroel|roverhttps://trunk.rdoproject.org/centos8-victoria/current-tripleo/delorean.repo.md5 -> f9563b13d92dcc5b99bff7a41699f6bd21:39
dviroel|rover"Not found image: http://158.69.74.140:5001/v2/tripleovictoria/openstack-haproxy/manifests/f9563b13d92dcc5b99bff7a41699f6bd"21:39
rlandy|ruckbb05cd497dfcae51f78d885d8277471e/2021-11-01 20:33 -  21:40
rlandy|ruck^^ content provider21:40
dviroel|roverhumm21:40
rlandy|ruckso we promoted 5fd2c0a5ce28b1c2f48eaa4636518afe21:40
rlandy|ruckand installed bb05cd497dfcae51f78d885d8277471e21:40
rlandy|ruckso marios is right21:40
dviroel|roveryeah, i see now21:41
rlandy|ruckis the install not the upgrade21:41
rlandy|ruckwe addressed the second half - first is still broken21:41
rlandy|ruckok - that's our next task21:41
rlandy|ruckthat and the downstream promoter image symlinks21:42
rlandy|ruckthink have a solution there21:42
dviroel|roveroh great21:43
rlandy|ruckstage_root 21:44
rlandy|ruckchecking with amol if I can just edit that 21:44
rlandy|ruckdviroel|rover: the rest is fine21:44
rlandy|ruckyou can take off21:44
rlandy|ruckwe'll pick it up tomorrow21:44
rlandy|ruckwill recheck the current failures21:44
rlandy|rucktripleo-ci-centos-8-content-provider-wallaby21:45
rlandy|ruckalso out21:45
rlandy|ruckdlrn packge21:45
dviroel|roveryeah, sae this error in other failed jobs21:47
* rlandy|ruck needs to go to volunteer job22:00
rlandy|rucksee you tomorrow22:00
*** rlandy|ruck is now known as rlandy|ruck|afk22:00
dviroel|roverrlandy|ruck|afk: https://snapshot.raintank.io/dashboard/snapshot/qAlimduFU16rrzLqhOjSoo0qbWBwP9a6?orgId=2 working here, just updated conf_ruck_rover.yaml22:17
dviroel|roverrlandy|ruck|afk: we sync again tomorrow22:17
*** dviroel|rover is now known as dviroel|rover|out22:18
« Tuesday, 2021-11-02 Index Thursday, 2021-11-04 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!