Wednesday, 2021-04-28

*** jmasud has joined #oooq00:14
*** jmasud has quit IRC00:25
*** jmasud has joined #oooq02:04
*** jmasud has quit IRC03:19
*** jmasud has joined #oooq03:26
*** ykarel has joined #oooq03:41
*** ratailor has joined #oooq04:37
*** ykarel has quit IRC05:09
*** marios has joined #oooq05:19
*** saneax has joined #oooq05:30
*** udesale has joined #oooq05:41
*** jpodivin has joined #oooq05:50
*** slaweq has joined #oooq06:00
*** ysandeep|away is now known as ysandeep06:17
*** ykarel has joined #oooq06:56
*** amoralej|off is now known as amoralej06:58
*** ykarel is now known as ykarel|pto07:13
*** skramaja has joined #oooq07:14
*** ykarel|pto has quit IRC07:21
*** jpena|off is now known as jpena07:21
mariosanbanerj|ruck: o/ so we decided to keep both of those? https://review.opendev.org/c/openstack/tripleo-ci/+/788258 https://review.opendev.org/c/openstack/tripleo-ci/+/78825708:00
*** derekh has joined #oooq08:01
*** ysandeep is now known as ysandeep|lunch08:02
mariosanbanerj|ruck: i commented, personally would prefer to squash into one unless there is a better reason not to08:04
mariosanbanerj|ruck: i think you can just abandon 788257 and include the fixes in 78825808:16
*** jmasud has quit IRC08:40
*** ratailor_ has joined #oooq09:22
*** jmasud has joined #oooq09:22
*** jmasud has quit IRC09:22
*** ratailor has quit IRC09:25
*** ysandeep|lunch is now known as ysandeep09:26
*** tosky has joined #oooq09:45
*** ratailor__ has joined #oooq09:55
*** ratailor_ has quit IRC09:57
soniya29kopecmartin, akahat, arxcruz, weshay|ruck, we have retrospective colliding with today's retrospective meeting10:06
akahatsoniya29, we can move it tomorrow 6 PM if possible10:08
arxcruzsoniya29: no problem for me10:09
soniya29akahat, arxcruz, but tommorow we have planning meeting as well10:20
soniya29akahat, arxcruz, but both meetings won't collide so no problem10:24
arxcruzok10:24
soniya29kopecmartin, weshay|ruck, raukadah, Let's move the tempest meeting tommorrow since we have retro today10:25
kopecmartinack10:26
mariosneeds reviews please https://review.opendev.org/c/openstack/tripleo-ci/+/78439210:31
arxcruzbhagyashris: where can I see ppc containers ?10:49
bhagyashrisarxcruz, ppc container should be here https://trunk.registry.rdoproject.org:8443/oapi/v1/namespaces/tripleomaster/imagestreamtags but currently ppc container build job is failing https://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/10:58
bhagyashristhat's why those are not available10:59
bhagyashrisarxcruz, fyi yesterday i pinged mjturek regarding the failure and totald me he will  restored job ASAP11:03
arxcruzbhagyashris: so we can blame mjturek !! I knew it!!!11:04
bhagyashristold *11:04
arxcruzbhagyashris: can you schedule a meeting so we can discuss the manifest thing ?11:12
arxcruzbhagyashris: from what I understood, we need to have 3 different repos11:12
arxcruzfor example11:12
arxcruzquay.io/tripleomaster/openstack-nova11:12
bhagyashrisarxcruz, sure11:12
arxcruzthat will be the principal one11:12
arxcruzthen quay.io/tripleomaster/openstack-nova-amd6411:12
arxcruzand quay.io/tripleomaster/openstack-nova-ppc11:13
arxcruzthen we write a manifest to quay.io/tripleomaster/openstack-nova:hash11:13
arxcruzpointint to openstack-nova-amd64:hash and openstack-nova-ppc:hash11:13
bhagyashrisyes11:14
arxcruzthe sha for the hashs must be different to use the annotate command to set ppc and amd6411:14
bhagyashrisi didn't get last point ^11:15
bhagyashrisarxcruz, is that ok to discuss after repro11:16
bhagyashrisretro11:16
arxcruzbhagyashris: sure11:16
bhagyashrisokay11:16
arxcruzbhagyashris: add weshay|ruck and ronelle as well :)11:16
bhagyashrisarxcruz, yes11:16
bhagyashrisarxcruz, asfik, the amd64 and ppc containers will be available on source registry means trunk.registry.rdoproject.org and from there will pull those containers on promoter server and will create manifest and push that to docker.io11:21
bhagyashriscurrently we are only pushing amd6411:21
arxcruzbhagyashris: so, from docker manifest documentation11:21
arxcruzyou create a manifest with:11:22
arxcruzdocker manifest create openstack-nova:latest11:22
arxcruzfor example11:22
arxcruzthis is pointing to openstack-nova11:22
arxcruzthen you can do a docker manifest add openstack-nova-ppc:latest11:22
arxcruzdocker manifest add openstack-nova-amd64:latest11:22
arxcruzand then annotate with docker manifest annotate --arch amd64 openstack-nova-amd64:latest11:23
arxcruzand docker manifest annotate --arch ppc openstack-nova-ppc:latest11:23
arxcruzthen you can push with docker manifest push openstack-nova:latest quay.io/tripleo/openstack-nova:latest11:24
bhagyashrisyes11:24
bhagyashrisarxcruz, here we are doing same stuff https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/container-push/roles/containers-promote/tasks/manifest-push.yml that you said11:28
mariosbhagyashris: arxcruz: fyi here are the notes from when i created that manifest-push.yml ^^^ https://tree.taiga.io/project/tripleo-ci-board/us/122111:30
mariosbhagyashris: arxcruz: not following your discussion but the commands are there as examples11:30
* marios fetch coffee refill11:30
*** jpena is now known as jpena|lunch11:31
arxcruzmarios: thanks this helps11:33
arxcruzso, it seems podman you're able to tag an image with ppc or amd11:33
arxcruzops11:33
arxcruzwith docker you can tag the same image with ppc and amd and add on the manifest11:33
arxcruzbut podman requires the sha11:34
arxcruzso the sha for both images would be the same and this won't work11:34
bhagyashrisohh ok so podman stuff is little different thats why i was wondering about sha stuff ...11:35
bhagyashrisarxcruz, ^ :D11:35
*** rlandy has joined #oooq11:38
*** rlandy is now known as rlandy|rover11:38
*** jfrancoa has quit IRC11:40
rlandy|roverbhagyashris: hi - weshay|ruck and I have the prod chain meeting at the time you booked the promoter sync11:43
*** udesale_ has joined #oooq11:43
*** udesale has quit IRC11:44
arxcruzbhagyashris: yeah, it's different, so i don't know how will we handle that11:46
bhagyashrisrlandy|rover, ack11:51
*** dviroel|afk is now known as dviroel11:51
*** udesale has joined #oooq11:52
*** udesale_ has quit IRC11:53
*** jfrancoa has joined #oooq11:54
*** udesale_ has joined #oooq11:55
*** udesale_ has quit IRC11:56
*** udesale_ has joined #oooq11:57
*** udesale has quit IRC11:57
*** rlandy|rover is now known as rlandy|rover|mtg11:58
*** ysandeep is now known as ysandeep|afk12:12
anbanerj|ruckmarios, ok, updating the reverts in one patch12:17
*** ratailor__ has quit IRC12:19
*** jpena|lunch is now known as jpena12:19
*** amoralej is now known as amoralej|lunch12:22
mariosbhagyashris: o/ hey do we need to create a UniSPrint43 tag?12:31
weshay|ruckarxcruz, zbr, sshnaidm, rlandy, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, weshay, chandankumar, frenzy_friday   Retrospective :)12:36
*** ysandeep|afk is now known as ysandeep12:56
*** skramaja has quit IRC13:02
*** skramaja has joined #oooq13:06
*** amoralej|lunch is now known as amoralej13:15
ysandeepweshay|ruck, rlandy|rover|mtg fyi.. i have added details about "How to open ticket for baremetal issues" on our tripleo-ci-infra doc13:31
rlandy|rover|mtgysandeep++13:34
weshay|ruckrlandy|rover|mtg, if there is time/space we should raise psi13:34
weshay|rucknot detail.. but13:34
rlandy|rover|mtgyay - ussuri promoting13:55
rlandy|rover|mtgweshay|ruck: ^^ can you kick openstack-periodic-integration-stable7 line - so we clear the last outstanding line before new ruck/rovers take over13:59
weshay|ruckk13:59
rlandy|rover|mtgbhagyashris: I don't think I am permissioned to start new jira boards etc.14:07
rlandy|rover|mtgdo you have access to check/grant permissions?14:08
rlandy|rover|mtgweshay|ruck: I'll join CIX14:25
bhagyashrisrlandy|rover|mtg, let me check14:26
rlandy|rover|mtgthanks14:32
*** rlandy|rover|mtg is now known as rlandy|rover14:41
weshay|ruckrlandy|rover, https://sf.hosted.upshift.rdu2.redhat.com/logs/74/238474/2/check/periodic-tripleo-ci-rhel-8-bm_envA-3ctlr_1comp-featureset035-rhos-17/3c73663/job-output.txt14:47
weshay|ruckcheckout 2021-04-27 19:07:11.588827 | primary | TASK [build-test-packages : Copy from ready directories if configured] *********14:47
weshay|ruckrlandy|rover, have you seen that w/ other depends-on internally?14:48
rlandy|roverweshay|ruck: not really14:49
bhagyashrisrlandy|rover, i have assigned the administration role to you. please check if you have rights to start/complete the sprint board15:06
rlandy|roverbhagyashris: ack - I have the complete sprint button available to me now15:07
rlandy|roverthank you15:07
bhagyashrisrlandy|rover, ack15:07
weshay|ruckmarios, comments on the get-hash patch15:07
weshay|ruckbhagyashris, how are you handling the jira sprint? closing it out like last time?15:07
bhagyashrisweshay|ruck, yeah,15:09
mariosweshay|ruck: yeah i saw from alex thanks i was thinking about what to reply i mean indeed we have had the same thought. so in 2 minds about whether we should merge for now and move later if we need to?15:10
weshay|ruckmarios, I'm just curious where he would see this being packaged.. which would play into where it lives15:10
weshay|ruckperhaps in rhos-bootstrap??? don't know.. it may be easier to do this in ci-config or it's own repo in rdo15:11
weshay|ruckget it built in pip and move on15:11
weshay|ruckseems like rdo would be good, as it pairs more w/ the dlrn server code15:12
weshay|ruckthan anything else I think of15:12
bhagyashrisweshay|ruck, fyi i have create new sprint board for sprint 44 https://projects.engineering.redhat.com/secure/RapidBoard.jspa?rapidView=428515:12
bhagyashriscreated*15:12
weshay|ruckzbr, don't forget to sync w/ soniya29 to figure out what's possible and not possible w/ regards to py39 tox in tempest repos15:13
mariosweshay|ruck: sorry arguing about puppet with alex on tripleo will check and reply on the get hash in a bit one argument at a time ;015:14
zbrok15:14
weshay|ruckmarios, np15:15
*** dviroel is now known as dviroel|lunch15:22
rlandy|roverweshay|ruck: ysandeep: fyi - downstream dlrn broken15:30
rlandy|rover[rhos-dev] [outage][OSP Trunk] OSP Trunk builds currently not working15:30
ysandeepyeah.. yup saw that email from javier15:31
rlandy|rovermarios: on get-hash review now15:47
*** jpodivin has quit IRC15:47
rlandy|roversorry about delay15:47
mariosanbanerj|ruck: o/ hey can you update those soon so we can be ready - likely posting the release patch tomorrow so we need those to be voting again https://review.opendev.org/c/openstack/tripleo-ci/+/788258 & https://review.opendev.org/c/openstack/tripleo-ci/+/788257 (imo abandon 788257 and just include everythin on 78825815:52
mariosack rlandy|rover thanks15:52
*** jmasud has joined #oooq15:53
weshay|ruckzbr, bah.. abandon is busted in gri :)15:56
*** ysandeep is now known as ysandeep|away15:56
weshay|ruckgri -f abandon15:56
anbanerj|ruckmarios, looks like I am doing something wrong - when I pull the patch with Revert it doesnt show the changes on my local. lemme sort that out15:57
mariosanbanerj|ruck: ack15:58
*** rlandy|rover is now known as rlandy|rover|mtg16:01
*** udesale_ has quit IRC16:03
zbrweshay|ruck: what do you mean by busted?16:09
zbrit does not work?16:09
zbrif it remember well it requires http access for these operations, but i do remember it as working but I did not use it myself.16:10
zbri wanted to avoid accidents16:10
*** marios is now known as marios|out16:13
*** ysandeep|away is now known as ysandeep16:14
ysandeepweshay|ruck, around?16:17
*** jpena is now known as jpena|off16:18
anbanerj|ruckrlandy|rover|mtg, marios|out https://review.opendev.org/c/openstack/tripleo-ci/+/788542 - abandoned both of the reverts and updated everything in this patch (wallaby non voting)16:18
weshay|ruckysandeep, aye16:18
ysandeepweshay|ruck, http://pastebin.test.redhat.com/95971616:19
*** dviroel|lunch is now known as dviroel16:20
ysandeepi think https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_bootstrap/tasks/main.yml#L38 - this will try to install using local repo not any url16:20
weshay|ruckysandeep, https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-1ctlr_2comp-featureset020-internal-rhos-17/afa781c/logs/overcloud-controller-0/var/log/dnf.log.txt.gz16:21
ysandeepweshay|ruck, i don't think the deploy failure and ^^ is related, because we are not passing any url to the ansible task https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_bootstrap/tasks/main.yml#L3816:22
*** marios|out has quit IRC16:30
ysandeepweshay|ruck, another observation,  cloud-init started running at 10:0816:30
ysandeephttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-1ctlr_2comp-featureset020-internal-rhos-17/afa781c/logs/overcloud-controller-0/var/log/cloud-init.log.txt.gz16:30
ysandeepbut the dnf.logs are from 05:0816:30
weshay|ruckysandeep, meet.google.com/ysq-kzxq-adt16:31
ysandeepafaik.. cloud-int should run first.. i wonder those dnf.logs are from image build16:31
*** dsneddon has joined #oooq16:31
ysandeepweshay|ruck, joining16:31
ysandeepweshay|ruck, https://sf.hosted.upshift.rdu2.redhat.com/images/redhat8/rhos-17/16:34
*** amoralej is now known as amoralej|off16:42
*** derekh has quit IRC17:04
akahatopenstack upshift is down?17:04
*** ysandeep is now known as ysandeep|away17:14
akahatweshay|ruck, there?17:21
weshay|ruckakahat, aye17:21
weshay|ruckwhat are you trying to hit?17:21
akahatweshay|ruck, i want to run this template in testproject: https://review.rdoproject.org/r/gitweb?p=rdo-jobs.git;a=blob;f=zuul.d/ci-config-molecule-python-pipeline.yaml17:21
akahatand i tried it here: https://review.rdoproject.org/r/c/testproject/+/31611/7/.zuul.yaml17:21
akahatbut it just run noop job.. what went wrong here ^^17:22
weshay|ruckakahat, rdo runs in vexhost17:23
weshay|ruckhttps://review.rdoproject.org/zuul/status17:23
weshay|rucklooks up17:23
akahatweshay|ruck, on upshift i was trying to start my machine for testing... But it seems down17:24
weshay|ruckakahat, it's just running the no-op17:24
weshay|ruckakahat, ya.. psi/upshift may be borked... migration blew things up17:25
akahatweshay|ruck, i want to run all the jobs in templates.17:26
weshay|ruckakahat, trying to login17:27
weshay|ruckseems down17:27
weshay|ruckcheck announce-list and outtage list.. psi has been all over it17:28
weshay|ruckakahat, /me gets Gateway Timeout17:28
akahatweshay|ruck, same.17:28
weshay|ruckakahat, you using rdo-ci-fx2-01-s8-drac.mgmt.rdoci.lab.eng.rdu2.redhat.com ?17:29
akahatweshay|ruck, yes.17:29
weshay|ruckstart a vm on that17:29
akahatweshay|ruck, ok17:29
rlandy|rover|mtgweshay|ruck: 1-on-1?17:31
rlandy|rover|mtgugh ... we are delete failed again downstream17:33
weshay|ruckrlandy|rover|mtg, in ovb17:34
weshay|ruck?17:34
weshay|rucklet's chat tomorrow17:34
rlandy|rover|mtgweshay|ruck: k- sending one question in ovt17:35
rlandy|rover|mtgpvt17:36
*** rlandy|rover|mtg is now known as rlandy|rover17:36
rlandy|roverweshay|ruck: yep ... ovb - manually deleting17:37
*** anbanerj|ruck has quit IRC18:19
*** anbanerj|ruck has joined #oooq18:20
rlandy|roverweshay|ruck: k- finally getting somewhere with OVB stack deletes19:06
rlandy|roverport are down but not deleted19:07
rlandy|rovermay be our workaround19:07
rlandy|rovermay be more OVN issue19:07
rlandy|rovermanual port delete works19:07
rlandy|roverbut you have to find all the ports associated19:07
weshay|ruckysandeep|away, ^ you, myself and rlandy|rover should meet up tomorrow re: internal fun and we'd like to see the logs to the internal cleanup19:20
rlandy|roveragreed19:20
rlandy|roverdoing some manual fun and games here19:20
weshay|ruckmay have to be friday19:20
weshay|ruckdepending the planning mtg19:20
rlandy|roverprobably not the best way about it19:20
rlandy|roverI am basiaclly pulling out virtual wires19:20
rlandy|rovernot elegant19:20
rlandy|roverthree stack left to go19:21
rlandy|roverbut these have active ports19:21
rlandy|roverwill need more investigation19:21
rlandy|roverthe other stacks had down ports19:21
rlandy|rover2021-04-28 19:32:43,507 7195 WARNING  promoter Candidate label 'tripleo-ci-testing': NO candidate hash promoted to current-tripleo19:35
rlandy|roverugh19:35
rlandy|roverweshay|ruck: need to hack the promoter for queens19:35
rlandy|roverc7 one19:35
rlandy|roverno images19:35
weshay|ruckrlandy|rover, ah crud.. k19:35
rlandy|roverweshay|ruck: we didn't get a local override for that yet?19:36
weshay|ruckjust turn off that module then turn it back on19:36
rlandy|roveryeah19:36
weshay|ruckrlandy|rover, don't think so19:36
rlandy|roverok19:36
rlandy|roverweshay|ruck: just trying to clear the decks for the new ruck/rovers19:36
rlandy|rovergive them some runway19:36
rlandy|roverweshay|ruck: k - hacked up the global vars and the releases19:46
rlandy|roverwill revert those when queens is done19:46
weshay|ruckrlandy|rover, ?19:46
weshay|ruckglobal vars and releases?19:46
rlandy|roverdlrn-promoter.sh19:46
weshay|ruckoh k19:46
rlandy|roverand global_defaults19:46
*** saneax has quit IRC20:06
*** jmasud has quit IRC20:07
*** jmasud has joined #oooq20:15
rlandy|rover2021-04-28 20:17:19,784 16285 INFO     promoter Summary: Promoted 1 hashes this round20:24
rlandy|roverreverting20:24
*** jfrancoa has quit IRC20:24
rlandy|roverweshay|ruck: am I crazy?20:27
rlandy|roverdid queens promote or not?20:27
weshay|ruckheh.. /me looks.. OH you can't tell from the logs.. lolz20:27
rlandy|roverdon't joke20:28
rlandy|roverI think it promoted one hash20:28
rlandy|roverbut dlrn didn;t move20:28
rlandy|roverallowed_clients: registries_client,dlrn_client20:29
rlandy|rover^^ left in dlrn_client20:29
weshay|ruckrlandy|rover, https://hub.docker.com/r/tripleoqueens/centos-binary-base/tags?page=1&ordering=last_updated20:29
weshay|ruckTAG20:29
weshay|ruckfc3eaa73026fec9feec3ef4e6295d08215d8b4ce_aaad73da20:29
weshay|ruckdocker pull tripleoqueens/centos-binary-base:fc3eaa73026fec9feec3ef4e6295d08215d8b4ce_aaad73da20:29
weshay|ruckLast pushed23 minutes agobyrdotripleomirror20:29
weshay|ruckhttps://trunk.rdoproject.org/centos7-queens/current-tripleo/delorean.repo20:29
rlandy|rovercommit.yaml2021-04-22 00:53 1.1K20:30
rlandy|rover^^ 04/2220:30
rlandy|rover7 days ago20:30
rlandy|rovercurrent-tripleo20:30
rlandy|roverqueens20:30
rlandy|roverCentOS-720:30
rlandy|rover2 days ago20:30
rlandy|roverweshay|ruck: ^^ says there was an update two days ago20:31
weshay|ruckwas the hash old?20:31
weshay|ruckcould have been20:31
rlandy|roverhttps://trunk.rdoproject.org/centos7-queens/tripleo-ci-testing/20:31
rlandy|rover^^ 04/2620:31
rlandy|roverperiodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-queensopenstack/tripleo-cimasteropenstack-periodic-integration-stable7000000020:32
rlandy|rover1 hr 56 mins 39 secs2021-04-28 16:28:20FAILURE20:32
rlandy|roverweshay|ruck: oh shoot, it failed20:32
* rlandy|rover rekicks20:32
rlandy|roverk - one stack left - too many ports :(20:43
rlandy|rovertraining21:14
*** jmasud has quit IRC21:20
*** slaweq has quit IRC21:25
*** jmasud has joined #oooq21:27
*** dviroel is now known as dviroel|away22:14
*** rlandy|rover is now known as rlandy|rover|bbl22:24
*** tosky has quit IRC22:48
*** gchamoul has quit IRC23:11
*** gchamoul has joined #oooq23:11

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!