Monday, 2020-02-17

*** dsneddon has joined #oooq00:05
*** dsneddon has quit IRC00:10
*** jmasud has quit IRC00:13
*** jmasud has joined #oooq00:14
*** dsneddon has joined #oooq01:13
*** dsneddon has quit IRC01:18
*** dsneddon has joined #oooq01:21
*** dsneddon has quit IRC01:26
*** rlandy has joined #oooq01:32
*** rlandy has quit IRC01:33
*** dsneddon has joined #oooq01:55
*** dsneddon has quit IRC02:00
*** saneax has quit IRC02:58
*** dsneddon has joined #oooq03:56
*** dsneddon has quit IRC04:00
*** ykarel|away is now known as ykarel04:05
*** udesale has joined #oooq04:10
*** jmasud has quit IRC04:11
*** jmasud has joined #oooq04:15
*** dsneddon has joined #oooq04:46
*** skramaja has joined #oooq04:50
*** dsneddon has quit IRC04:51
*** dsneddon has joined #oooq05:00
*** dpawlik has joined #oooq05:21
*** dsneddon has quit IRC05:24
*** dsneddon has joined #oooq05:52
*** saneax has joined #oooq05:57
*** ratailor has joined #oooq05:58
*** marios|ruck has joined #oooq06:23
*** dpawlik has quit IRC06:31
*** dpawlik has joined #oooq06:31
*** dpawlik has quit IRC06:42
*** dpawlik has joined #oooq06:48
*** jfrancoa has joined #oooq06:54
*** ykarel is now known as ykarel|afk07:01
*** jbadiapa has joined #oooq07:05
chandankumarmarios|ruck, Good morning07:13
chandankumarmarios|ruck, https://review.rdoproject.org/r/#/c/24976/ https://review.opendev.org/707359 https://review.opendev.org/707574 patches in sequence for image server migration, please have a lookwhen free, thanks!07:13
*** ykarel|afk is now known as ykarel07:14
marios|ruckchandankumar: o/ will do07:20
*** apetrich has joined #oooq07:46
sshnaidmmarios|ruck, please in your time: https://review.opendev.org/#/c/702100/  https://review.opendev.org/#/c/706797/08:10
marios|rucksshnaidm: ack08:11
sshnaidmchandankumar, dpawlik do you know why we have python2 available in rhel8 images?08:12
*** apetrich has quit IRC08:19
chandankumarsshnaidm, let me check08:20
*** ykarel is now known as ykarel|lunch08:22
chandankumarsshnaidm, I will be checking with jpena|off in another 30 mins via call08:26
sshnaidmchandankumar, great, it started to break things, so I hope we can get rid off it there08:27
*** tesseract has joined #oooq08:30
*** amoralej|off is now known as amoralej08:37
*** ccamacho has joined #oooq08:37
*** dtantsur|afk is now known as dtantsur08:41
*** apetrich has joined #oooq08:45
matbuHi folks, I'm wondering why the overcloud-full-centos-7 is failing on this review https://review.opendev.org/#/c/706170/ ?08:51
*** tosky has joined #oooq08:51
marios|ruckmatbu: o/ checking might be a new one08:51
matbumarios|ruck: thx, i don't get if the failure is due to the review or not08:52
marios|ruckException raised: No module named http08:52
marios|ruckmatbu: hm its a known one but maybe we missed one of the fixes08:52
*** jpena|off is now known as jpena08:52
marios|ruckmatbu: https://bugs.launchpad.net/tripleo/+bug/1862674 i'll look more into that in a sec thanks for ping08:54
openstackLaunchpad bug 1862674 in tripleo "build-containers jobs failing on "no module named http"" [Critical,Fix released] - Assigned to yatin (yatinkarel)08:54
marios|ruckmatbu: https://review.opendev.org/#/c/707927/ fix08:54
matbumarios|ruck: ack cool, thx :) I'm happy to see its not related to my changes :)08:55
marios|ruckmatbu: yeah sorry about that it broke all the things upstream/downstream and i thought it was done but looks not08:55
*** bogdando has joined #oooq08:56
matbumarios|ruck: no pb09:09
zbrmarios|ruck: have you read the artcl comments from the the last week? like requirement from upstream to have logs browsable?09:18
zbrhttps://docs.openstack.org/infra/system-config/third_party.html09:18
zbrquoting: "logs must be browsable; logs requiring download, installation or login to access are not acceptable"09:19
zbrmainly producing tar.gz archives would break this, even individual gz files are borderline.09:19
sshnaidmmarios|ruck, this is good to go https://review.opendev.org/#/c/707574/09:22
chandankumarmarios|ruck, we merged https://review.rdoproject.org/r/#/c/24976/ swthiching to vexxhost image server, let me and dpawlik know if you see any breakage on rhel8 image build09:26
marios|ruckack sshnaidm chandankumar09:27
dpawlikchandankumar, ack09:27
sshnaidmmarios|ruck, so we have enabled centos8 building containers?09:27
sshnaidms/so/do/09:28
sshnaidmmarios|ruck, I'd like to try them on vexxhost09:28
dpawliksshnaidm, marios|ruck, chandankumar: please let me know or jpena when you switch everything to vexxhost, so we will be able to move forward with migration09:30
sshnaidmdpawlik, well, we're far from switching everything to vexxhost :)09:31
marios|rucksshnaidm: yes the check jobs but not periodic afaik09:31
marios|rucksshnaidm:https://review.opendev.org/#/c/706572/09:31
sshnaidmmarios|ruck, ack, can I make periodic from it? Are there any known blockers?09:32
dpawliksshnaidm, so if you are with one leg in vexxhost ;)09:32
sshnaidmdpawlik, yeah, we'll be in this weird position another month at least :)09:32
*** derekh has joined #oooq09:33
dpawliksshnaidm, but it will get better over time :)09:34
marios|rucksshnaidm: we have periodic09:36
marios|rucksshnaidm: but not wired into the layout yet09:36
marios|rucksshnaidm: https://github.com/rdo-infra/review.rdoproject.org-config/blob/d3af3a1dc6edf68ac2b14ec073bf7479c67ed866/zuul.d/tripleoci.yaml#L37009:37
*** chem has joined #oooq09:40
*** ratailor has quit IRC09:43
*** ratailor has joined #oooq09:44
chandankumarsshnaidm, I was speaking with jpena, this sprint delivery team is working on vexxhost migration https://tree.taiga.io/project/morucci-software-factory/taskboard/sprint-22-79 and they are managing migration using this https://softwarefactory-project.io/r/gitweb?p=software-factory/sf-infra.git;a=tree;h=6f4d5a25cadc990ac4e34f18ca2080ef6d33d86e;hb=refs/heads/master09:48
chandankumarsshnaidm, they are using a new config language dhal https://softwarefactory-project.io/r/gitweb?p=software-factory/sf-infra.git;a=blob;f=package.dhall;h=afdc0ba67012984a3289d046ba7e7666ef672fa0;hb=refs/heads/master, Do we also want to manage our infra there using same repo09:49
chandankumaras a code?09:49
chandankumarsshnaidm, https://dhall-lang.org/09:50
chandankumardpawlik, regarding centos 8 image upload https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/ci-scripts/tripleo-upstream/upload-cloud-images.sh#L2709:58
chandankumardpawlik, does the user is uploader there also?09:59
sshnaidmchandankumar, idk, it doesn't look very readable https://softwarefactory-project.io/r/gitweb?p=software-factory/sf-infra.git;a=blob;f=playbooks/vars/infra-sf.dhall;h=5831788e1366d02f31d3104626ca6e3404f0ca26;hb=refs/heads/master10:00
chandankumarjpena, ^^10:00
*** udesale_ has joined #oooq10:00
dpawliksshnaidm, you can switch to yaml10:00
dpawlikchandankumar, you mean10:00
dpawlikif zuul is able to push content via uploader?10:00
sshnaidmdpawlik, so what's advantage to use dhall?10:01
dpawlikcc @tristanC ^^10:01
*** udesale_ has quit IRC10:01
*** udesale has quit IRC10:01
*** udesale has joined #oooq10:01
dpawliksshnaidm, he is a proper person to ask :)10:01
dpawlikchandankumar, so if you mean if zuul is able to push , he will be able after https://review.rdoproject.org/r/#/c/24770/10:02
dpawlikand to do that it will be nice to switch domain...10:03
chandankumardpawlik, we can switch domains but before that let's merge above review, let's have one periodic run finishes for c7 and rhel8 so that we have all stuff working10:05
*** ykarel|lunch is now known as ykarel10:05
chandankumarsshnaidm, https://review.rdoproject.org/r/#/c/24770/10:06
dpawlikchandankumar, ack10:07
sshnaidmchandankumar, is this used? https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/secrets.yaml#L152:L15210:08
chandankumarsshnaidm, this was for the old rcm server hosted on rdocloud, since we made the switch, so I think it will be no longer needed10:11
chandankumarbetter we can remove that tomorrow or day after tomorrow once everything is working fine10:11
chandankumarsshnaidm, https://review.rdoproject.org/r/#/c/24915/7/ci-scripts/tripleo-upstream/upload-cloud-images.sh10:12
dpawliksshnaidm, https://review.rdoproject.org/r/#/c/24770/ you merge that, so we will switch domains when https://review.opendev.org/#/c/707359/ and https://review.opendev.org/#/c/707574/ gates finish tests10:24
chandankumarsshnaidm, marios|ruck https://review.rdoproject.org/r/#/c/24944/10:36
sshnaidmchandankumar, I don't think you'll see something with default verbosity10:37
chandankumarsshnaidm, https://bugs.launchpad.net/tripleo/+bug/1863224/comments/310:38
openstackLaunchpad bug 1863224 in tripleo "podman-package-rhel-8 job is giving post_failure after RHEL-8.1 movement" [Critical,Confirmed]10:38
*** udesale has quit IRC10:38
sshnaidmchandankumar, ack10:39
chandankumarmarios|ruck, is this known https://logserver.rdoproject.org/59/707359/5/openstack-check/tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001/79020ec/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz10:39
chandankumarmarios|ruck, Error: The following files were not found: /home/zuul/overcloud-baremetal-deployed.yaml10:39
sshnaidmchandankumar, and if there's no rhui like in centos?10:40
sshnaidmchandankumar, task may fail10:40
marios|ruckchandankumar: not seen that one thanks, noting in etherpad for now. theres a few gate issues at the moment not sure which are transient yet10:40
sshnaidmchandankumar, need to remove pipefail10:40
chandankumarsshnaidm, https://review.rdoproject.org/r/#/c/24944/2/playbooks/podman/pre.yaml10:40
chandankumarsshnaidm, it will run only for rhel810:41
chandankumarmarios|ruck, looking into If I get early, will propose the fix10:41
marios|ruckchandankumar: k10:42
marios|rucksshnaidm: do you think we need anything further @ https://bugs.launchpad.net/tripleo/+bug/1861694 ? afaics https://review.opendev.org/#/c/705446/ merged and the fails recently for that job are not related (been checking https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset00110:50
openstackLaunchpad bug 1861694 in tripleo "Nonstop restarting ovn_metadata_haproxy container" [Critical,Triaged]10:50
marios|rucksshnaidm: i am going to mark the bug as fix released if someone disagrees hopefully they can move it back10:51
*** ratailor has quit IRC11:16
*** ratailor has joined #oooq11:23
dpawliksshnaidm, chandankumar, 25 min afk11:31
ykarelmarios|ruck, just saw ur comment wrt ovn restart CIX11:32
ykarelin upstream we worked around the issue, didn't fixed11:32
ykareli didn't followed up what's the work against actual fix,11:32
marios|ruckykarel: k do you have something to point to for 13:32 < ykarel> in upstream we worked around the issue, didn't fixed11:36
ykarelmarios|ruck, https://review.opendev.org/#/c/705446/11:36
ykarelwe stopped collecting the command which was failing11:36
ykarelso we put workaround, didn't fixed that11:36
marios|ruckykarel: ack i added pointer to that in the trello. i see thanks for info it is already on the bug and merged. so you're saying we should leave it open a while longer yet11:37
ykarelmarios|ruck, my point is: u need to track such that u can revert the workaround after original fix lands11:38
zbrmarios|ruck: chandankumar : artcl https://review.opendev.org/#/c/706237/11:40
marios|ruckthanks ykarel will add another note in the lp in a bit11:40
sshnaidmmarios|ruck, it's easy to check if it's solved11:46
sshnaidmmarios|ruck, lemme do a patch11:46
ykarelbtw what solved it?11:47
ykarelif it's solved good to confirm11:47
*** ratailor has quit IRC11:47
marios|rucksshnaidm: thanks11:54
* marios|ruck going afk food before calls biab11:54
*** ykarel is now known as ykarel|afk11:54
*** marios|ruck is now known as marios|ruck|food11:55
sshnaidmwrt https://bugs.launchpad.net/tripleo/+bug/1861694 - it should be assigned to networking DFG12:00
openstackLaunchpad bug 1861694 in tripleo "Nonstop restarting ovn_metadata_haproxy container" [Critical,Triaged]12:00
sshnaidmmarios|ruck|food, ^12:00
*** rfolco has joined #oooq12:04
*** jmasud has quit IRC12:12
*** marios|ruck|food is now known as marios|ruck12:14
marios|ruckthanks sshnaidm12:14
*** jmasud has joined #oooq12:14
marios|ruckchandankumar: https://bugs.launchpad.net/tripleo/+bug/1863224/comments/512:25
openstackLaunchpad bug 1863224 in tripleo "podman-package-rhel-8 job is giving post_failure after RHEL-8.1 movement" [Critical,Fix released]12:25
chandankumarmarios|ruck, aye, fixed released12:26
marios|ruckthanks chandankumar12:28
*** udesale has joined #oooq12:31
marios|ruckzbr: please when you get a minute https://bugs.launchpad.net/tripleo/+bug/1862941/comments/9 thanks12:33
openstackLaunchpad bug 1862941 in tripleo "upstream all and rdo tox jobs broken due to cannot import name ensure_text" [Critical,In progress] - Assigned to yatin (yatinkarel)12:33
*** jpena is now known as jpena|lunch12:35
*** fuzzball81 is now known as jjoyce12:36
*** jaosorior has joined #oooq12:37
marios|ruckweshay|ruck: chandankumar: wdyt i moved it fix released https://bugs.launchpad.net/tripleo/+bug/1860810/comments/812:44
openstackLaunchpad bug 1860810 in tripleo "os_ansible_tempest fails w/ create_network() got an unexpected keyword argument 'dns_domain'" [High,Fix released]12:44
chandankumarmarios|ruck, keep it open, until we get a proper fix12:44
chandankumarit depends on ansible 2.9 and latest openstack sdk version for stein12:45
marios|ruckchandankumar: k moving back12:46
marios|ruckchandankumar: https://bugs.launchpad.net/tripleo/+bug/1860810/comments/912:46
openstackLaunchpad bug 1860810 in tripleo "os_ansible_tempest fails w/ create_network() got an unexpected keyword argument 'dns_domain'" [High,In progress]12:46
chandankumarmarios|ruck, aye, thanks!12:46
marios|ruckthank you chandankumar12:47
marios|ruckhttps://review.opendev.org/#/c/707927/1 merge please sshnaidm rfolco panda arxcruz chandankumar if you hav time for reviews thanks12:49
*** rlandy has joined #oooq12:55
*** marios|ruck is now known as marios|ruck|call13:01
rfolcorlandy, weshay|ruck zbr panda arxcruz sshnaidm chandankumar ping scrum?13:01
rfolcohttps://meet.google.com/oiv-geho-mai?authuser=113:01
rfolcozbr, arxcruz last call13:02
rfolcowe are departing13:02
arxcruzrfolco: sorry, bio issues :P13:04
rlandyrfolco: https://review.rdoproject.org/r/#/c/24966/13:11
rlandyhttps://review.rdoproject.org/r/#/c/24960/13:12
rlandytest13:12
*** amoralej is now known as amoralej|lunch13:12
weshay|ruckhttps://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw?both13:16
*** jpena|lunch is now known as jpena13:34
chandankumarsshnaidm, dpawlik Do we want to wait for today and then remove vexxhost_server hack from config?13:40
chandankumartomorrow13:40
sshnaidmchandankumar, why to wait? let's run w/o hack and we'll see13:41
chandankumardpawlik, we are keeping images-vexxhost.r.o and images.r.o url both na?13:41
chandankumarsshnaidm, let me propose the patch then13:41
*** saneax has quit IRC13:44
sshnaidmchandankumar, dpawlik so where to do we promote built images now?13:46
*** ykarel|afk is now known as ykarel13:46
chandankumarsshnaidm, promoted images will go to images-vexxhost server13:48
dpawlikchandankumar, sshnaidm it really depends of you. If you want to wait, we can do it tomorrow13:51
*** amoralej|lunch is now known as amoralej13:53
chandankumarsshnaidm, I would suggest, let's wait for tomorrow, https://review.rdoproject.org/r/24984 made the changes for images.rdoproject.org13:53
*** Goneri has joined #oooq13:53
chandankumardpawlik, we can currently disable the rdo-cloud image server13:53
*** marios|ruck|call is now known as marios|ruck13:54
weshay|ruckfyi.. folks you do not need to join the infrared planning meeting13:54
weshay|ruckrfolco, ^13:54
weshay|ruckplease help me spread the word13:55
chandankumarweshay|ruck, i am heading home13:56
weshay|ruckchandankumar, ack.. thanks13:56
rfolcozbr, marios|ruck chandankumar sshnaidm panda arxcruz rlandy: IGNORE (infrared) PCCI planning meeting13:58
arxcruzrfolco: no meeting then?13:58
rfolcounless you have anything to discuss there w/ pcci team13:59
weshay|ruckmarios|ruck, we broke ci.centos jobs w/ the python six / virt env work arounds https://ci.centos.org/view/rdo/view/promotion-pipeline/13:59
weshay|ruckmarios|ruck, will be putting up a patch later to deal w/ that13:59
arxcruzrfolco: i have a UniSprint22 planning starting now13:59
zbrjust ping me, if i need to join.13:59
rfolcothats the thing you can ignore unless you have shared goals to discuss13:59
rfolcomaybe collect-logs folks need to join weshay|ruck ?14:00
*** marios|ruck is now known as marios|ruck|call14:00
*** marios|ruck|call is now known as marios|ruck14:00
marios|ruckweshay|ruck: :/ ok let me know what to check/lets talk once you're done on calls14:02
weshay|ruckyou really don't need to be on this mtg14:03
weshay|ruckunless you of course want to be14:03
marios|ruckweshay|ruck: so this is planning only for amnon team?14:03
weshay|ruckchandankumar, before you go..14:04
marios|ruckweshay|ruck: i thought i'd rejoin but i see most folks dropped14:04
weshay|ruckwhat's the status on images?14:04
marios|ruckweshay|ruck: merging sec 09:13 < chandankumar> marios|ruck, https://review.rdoproject.org/r/#/c/24976/ https://review.opendev.org/707359 https://review.opendev.org/707574 patches in sequence for image server migration, please have a lookwhen free,14:04
rlandypanda: do you want to join the daily centos8 sync call?14:07
rlandyI can add you - if you want to skip, that's fine14:07
weshay|rucksshnaidm, everything ok w/ the image server?14:09
sshnaidmweshay|ruck, we'll see I suppose14:10
weshay|rucksshnaidm, k.. i'll monitor it14:10
weshay|rucksshnaidm, made some updates to the job-explorer page14:10
sshnaidmok14:11
pandarlandy: add me, it will help me review14:14
rlandyrfolco: panda: chandankumar: weshay|ruck: centos 8 sync call  Tues, Wed, Fri 1pm UTC - invite sent14:17
rfolcorlandy, nice, thanks for setting this up14:18
rlandyarxcruz: zbr: ^^ if you have any interest in joining14:18
arxcruzrlandy: sure, please, invite me too :)14:24
rlandyarxcruz: sent - join the party14:26
rfolcorlandy, wait on https://review.rdoproject.org/r/#/c/24511/14:31
rfolconeed to fix it14:32
rfolcoits not just train thing14:32
rfolcow-1 for now14:32
rlandyrfolco: I didn't merge it, just rebased14:32
rfolcok, just communicating :)14:32
rlandyrfolco: we should sync on the order of reviews there14:33
rfolcoyes gimme a few to fix that patch and we can sync14:33
chandankumarmarios|ruck, all relevant patches are merged now14:34
marios|ruckthanks chandankumar saw it was replying to weshay|ruck14:42
weshay|rucksshnaidm, provider = rdo-cloud or rdo-cloud-tripleo?14:45
weshay|ruckto best monitor 3rd party ovb?14:45
marios|ruckchandankumar: arxcruz: are you aware of something related to         * https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-train/a0106ee/logs/undercloud/var/log/tempest/stestr_results.html.gz14:46
marios|ruckchandankumar: arxcruz: i see last 3 runs did that ^         * test_cross_tenant_traffic[compute,id-e79f879e-debb-440c-a7e4-efeda05b6848,network] fail14:46
arxcruzmarios|ruck: checking14:46
arxcruzmarios|ruck: this is a network issue14:47
marios|ruckthanks arxcruz i we have 3 in a row... i'll file a lp in a sec14:49
arxcruzmarios|ruck: the vm is being created, but you can't ssh into it14:49
chandankumarsshnaidm, weshay|ruck https://softwarefactory-project.io/r/#/c/17428/ it is not going to affect rhel8 image related jobs14:50
chandankumarit is good to go14:50
rlandychandankumar: when you have a moment, bm is timing out on tempest since 02/09 ... is this something you know about https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/periodic-tripleo-ci-centos-7-bm_envC-1ctlr_1comp-featureset001-train/ab44194/logs/undercloud/home/zuul/tempest/tempest.log?14:51
rlandy'ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f50268ea190>, 'Connection to 10.9.121.218 timed out. (connect timeout=60)')': /v2.1/os-floating-ips: ConnectTimeoutError: (<urllib3.connection.VerifiedHTTPSConnection object at 0x7f50268ea190>, 'Connection to 10.9.121.218 timed out. (connect timeout=60)')14:51
marios|ruckarxcruz: chandankumar: looks like we have a bug for that already https://bugs.launchpad.net/tripleo/+bug/185601614:51
openstackLaunchpad bug 1856016 in tripleo "Tempest basic ops test_cross_tenant_traffic tests failed on fs020 train with Timed out waiting for 10.0.0.102 to become reachable from 10.0.0.117" [Critical,Fix released] - Assigned to chandan kumar (chkumar246)14:51
chandankumarmarios|ruck, that was for train14:52
marios|ruckchandankumar: yes it is hitting train14:52
marios|ruckchandankumar: last 3 runs14:52
marios|ruckhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-train14:52
marios|ruckchandankumar: adding comment we need to reskip i think14:52
marios|ruckblocks train promotino14:52
chandankumarmarios|ruck, ok14:52
weshay|ruckzbr, ping.. re: collect logs molecule14:56
zbryeah14:57
*** TrevorV has joined #oooq14:58
dpawlikchandankumar, so can we switch today the domains or not yet?15:02
rlandyrfolco: don't shoot me - had to -1 https://review.rdoproject.org/r/#/c/24511/ .. please see comment15:02
rfolcorlandy, this should be the label when a component is promoted15:03
chandankumardpawlik, will switch the domain tomorrow since we are keeping vexxhost-images so it will not cause any issue15:03
rfolcorlandy, this should be in sync w/ dlrn/promoter changes, panda please review ^15:03
rlandyrfolco,: so you're saying my it should be consistent?15:04
chandankumardpawlik, https://softwarefactory-project.io/r/#/c/17428/ is good to go15:05
dpawlikchandankumar, yeah, but can not merge that until record A is not set to proper host. So let's move that for tomorrow15:06
dpawlikweshay|ruck, should I send next email about that ^^ ?15:06
chandankumardpawlik, ok15:06
dpawlikcc jpena ^^15:06
rfolcorlandy, promoted_components is the new label when the standalone job in component pipeline passes and meets the criteria to promote an individual component.15:07
rfolco(instead of 'current-tripleo' from the legacy promotion pipeline)15:07
jpenadpawlik: ack, so we switch DNS entries tomorrow?15:08
rlandyrfolco: yes - but your review says consistent15:08
rlandyhttps://review.rdoproject.org/r/#/c/24511/7/zuul.d/tripleoci.yaml15:09
rfolcorlandy, oh my goodness I am sorry15:09
rfolcorlandy, updating 3 patches at the same time, now i see15:09
rfolcopardon me15:09
rlandyrfolco: np - take your time - then we can review slowly15:10
dpawlikjpena, yes, tomorrow...15:11
dpawlikweshay|ruck, jpena, is it required to notify one more time on the mailing list ?15:12
weshay|ruckdpawlik, I don't know what the current status is.. so I would say .. yes please15:13
weshay|ruckmarios|ruck, k.. out of meetings15:18
marios|ruckweshay|ruck: o/15:18
weshay|ruckmarios|ruck, option to sync.. but not required15:18
marios|ruckweshay|ruck: sure up to you, as we said in other chat... things on ruck|rover pad right now. latest ... train promotion re-blocked on tempest posted revert https://review.opendev.org/70813715:19
weshay|ruckaye k15:19
weshay|rucklet's skip.. :)15:19
weshay|ruckthanks for the rucking15:19
marios|ruckweshay|ruck: ack am around while longer will re-check before i go if you want to sync then15:20
rlandyhttps://softwarefactory-project.io/r/#/c/17327/15:20
rlandy^^ what are we waiting for on this?15:20
marios|ruckweshay|ruck: gonna look into the red (older) promotions a bit15:20
*** ykarel is now known as ykarel|away15:20
weshay|ruckmarios|ruck, if you spend time.. try to run quickstart on a bm box15:21
marios|ruckweshay|ruck: any more context?15:21
rlandypanda: ^^ do you know? can we ping for merge?15:21
weshay|ruckmarios|ruck, https://ci.centos.org/job/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/673/console15:21
weshay|ruckthis is acutally blocking people15:21
marios|ruckweshay|ruck: ah k15:21
weshay|ruck./quickstart.sh --working-dir /home/jenkins/workspace/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/ --no-clone --bootstrap --requirements requirements.txt --requirements quickstart-extras-requirements.txt --requirements ci-scripts/ci-base-requirements.txt --playbook noop.yml 127.0.0.215:22
weshay|ruck03:51:18 sudo: no tty present and no askpass program specified15:22
weshay|ruck03:51:21 Build step 'Execute shell' marked build as failure15:22
* weshay|ruck writes bug15:22
ykarel|awayweshay|ruck, checy if soniya29 already written the bug15:22
ykarel|awayi asked in morning wrt it^15:22
* ykarel|away also checks15:23
ykarel|awayjust to avoid duplicates15:23
chandankumarykarel|away, weshay|ruck I have not seen the bug yet15:23
chandankumarykarel|away, she was not able to reproduce the same15:23
marios|ruckweshay|ruck: 10:51:21 + echo 'ERROR: Config file minimal is not found.'15:23
ykarel|awaychandankumar, mmm, on friday it was reproducable15:23
ykarel|awaypossibly she missed something15:23
weshay|ruckmarios|ruck, https://bugs.launchpad.net/tripleo/+bug/186363215:24
openstackLaunchpad bug 1863632 in tripleo "lastest virtenv workarounds broke native tripleo-quickstart install" [Critical,Triaged]15:24
marios|ruckweshay|ruck: 10:51:21 head: cannot open ‘/home/jenkins/workspace/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal/cico_key.txt’ for reading: No such file or directory15:24
ykarel|awayhmm possibly she missed, no bug15:24
weshay|ruckmarios|ruck, that means the python setup.py install failed15:24
marios|ruckweshay|ruck: looking (sorry seeing multiple error: there)15:24
weshay|ruckykarel|away, ya.. I think she did as well15:24
* weshay|ruck look15:24
weshay|ruckykarel|away, she never wrote it up15:26
weshay|ruckno bugs from her15:26
ykarel|awayweshay|ruck, yes right she possibly missed, i don't find any recent bug related to it https://bugs.launchpad.net/tripleo/+bugs?orderby=-id&start=015:26
weshay|ruckshe did talk to me about it though15:26
ykarel|awayokk15:27
ykarel|awayso we are good no duplicates15:27
ykarel|awaybut looks like u posted two bugs :)15:27
weshay|rucksoniya29, don't be afraid to write bugs :))  we'll walk you through the procedure :)15:27
weshay|ruckykarel|away, I clicked twice by accident15:27
weshay|ruckand it posted15:28
ykarel|awayack close one15:28
weshay|rucktoo much coffee?15:28
rlandyrfolco: hmmm ... https://review.rdoproject.org/r/#/c/24512/ also has an issue :(15:28
rfolcorlandy, please fix, https://review.rdoproject.org/r/#/c/24987/ is on top of it :(15:29
rlandyrfolco: we can't get rid of playbooks/tripleo-ci-base-promote-consistent-to-tripleo-ci-testing/run.yaml15:29
rlandy^^ need that for non-component based releases15:30
rfolcorlandy, notice that I consolidate promote-hash parent at https://review.rdoproject.org/r/#/c/24511/15:31
rlandyrfolco: meaning?15:32
rfolcorlandy, meaning we won't need a playbook for each promote, we can define promote_source and promote_target in the child job15:32
rfolcofor that patch we need to keep that I agree15:33
chandankumarrlandy, I am logging out, send me the notes so that i can pick in the morning15:33
*** skramaja has quit IRC15:33
chandankumarsee ya15:33
*** chandankumar is now known as raukadah15:33
rlandychandankumar: ack - have a good night15:33
dpawlikweshay|ruck, is it ok: http://pastebin.test.redhat.com/836611 ?15:34
rlandyrfolco: k - fixing https://review.rdoproject.org/r/#/c/24512/15:34
weshay|ruckdpawlik, looks perfect to me.. ack to send to rdo-discuss and openstack-dev[tripleo] thank you :)15:35
rfolcorlandy, how about merging this first ? https://review.rdoproject.org/r/#/c/24511/15:40
rfolcothis oculd be the top one15:40
rfolcothen we can merge https://review.rdoproject.org/r/#/c/24512/ and https://review.rdoproject.org/r/#/c/24987/15:41
rlandyrfolco: hold on - fixing rename15:41
rfolcorlandy, in 24511 I create a parent job for promoting any hash15:41
rfolcohttps://review.rdoproject.org/r/#/c/24511/7/zuul.d/tripleo-rdo-base.yaml15:41
rfolcothat means we no longer need a separate playbook for those jobs that promote hashes15:43
rfolcowe set promote_source and promote_target in the job definition and parent off the new promote-hash job https://review.rdoproject.org/r/#/c/24511/7/zuul.d/tripleo-rdo-base.yaml15:43
rfolcorlandy, let me know if this makes sense15:43
rlandyrfolco: will ping you when I am done here15:44
rlandyneed to undo all this renaming15:44
rfolcorlandy, if you read what I just said you don't need to15:45
rfolcorlandy, let's change the promote-hash jobs to use the new parent, just that15:46
rlandyrfolco: let's meet - this makes no sense15:46
rlandyrfolco: https://meet.google.com/umz-wdpj-ayh?pli=1&authuser=015:48
rlandyjpena: hi ... what's holding up merging https://softwarefactory-project.io/r/#/c/17327/?15:51
jpenarlandy: mostly me not receiving email notifications from that review :-/15:52
jpenalet me merge the whole patch chain15:52
rlandyjpena: great - thanks15:52
rfolcorlandy, https://review.rdoproject.org/r/#/c/24987/15:53
rfolcothis replaces the abandoned one15:53
weshay|ruckpanda, you still around?15:54
weshay|ruckpanda, what's next re: reviews15:54
rfolcorlandy, https://review.rdoproject.org/r/#/c/24511/15:55
weshay|ruckreviews please https://review.rdoproject.org/r/#/c/24966/15:56
pandaweshay|ruck: https://review.rdoproject.org/r/2467615:57
weshay|ruckthanks15:57
pandacontainers promotion passed in integration pipeline \o/15:57
weshay|ruckpanda, sshnaidm you guys may want to sync up a bit on what you think the server requirements may want to be on the promoter server in vexxhost15:58
weshay|ruckpanda, which test?15:58
*** ccamacho has quit IRC15:58
*** ccamacho has joined #oooq15:58
weshay|ruckrfolco, please review https://review.opendev.org/#/c/706952/4/roles/build-containers/templates/build.sh.j215:59
weshay|ruckrfolco, trying to get more output in a sane way.. when we have python module issues15:59
pandaweshay|ruck: local functional tests. All functional tests pass, I have aggregate promotion for dlrn, qcows and containers working now. Ready to proceed with the promoter integration now. Then it's just docstrings, polish, and distill all that code into single msaller reviews.16:01
weshay|ruckpanda, very nice sir.. was that local or in a review?16:02
weshay|ruckmarios|ruck, please review https://review.rdoproject.org/r/#/c/24676/ zbr you as well :)16:02
zbrin progress16:03
pandaweshay|ruck:  https://review.rdoproject.org/r/24818, but it's huge.16:03
weshay|ruckah k16:03
weshay|ruckzbr, thanks16:03
*** ccamacho has quit IRC16:05
marios|ruckack weshay|ruck16:06
weshay|ruckthanks16:07
*** d0ugal has quit IRC16:10
*** d0ugal has joined #oooq16:12
*** jmasud has quit IRC16:12
*** jmasud has joined #oooq16:14
rlandyweshay|ruck: marios|ruck: are you guys ok with our merging https://review.rdoproject.org/r/#/c/24511/10/zuul.d/tripleo.yaml16:14
pandazbr: responded to your comment.16:14
rlandywill add currently failing job to the master pipeline16:15
rlandyrfolco: ^^16:15
marios|ruckrlandy: not been following there since becoming ruck16:15
marios|ruckrlandy: but no objection from me... it isn't in criteria etc so...16:15
* weshay|ruck looks16:17
weshay|ruckrlandy, I am..16:17
rlandyk - thanks ... just wanted to give ruck/rover heads up16:18
weshay|ruckrlandy, later today.. let's fire up a hackmd for centos-8 zuul config16:18
rlandyweshay|ruck: ack - just merging this first set of patches with rfolco16:18
rlandyweshay|ruck: also see ... https://softwarefactory-project.io/r/#/c/17327/ merged16:19
weshay|ruckya.. saw it16:19
rlandyok - so we're one step closer16:19
weshay|ruckrfolco, this is a 8 character review.. please check it out https://review.opendev.org/#/c/70695216:21
rlandyrfolco: https://review.rdoproject.org/r/#/c/24511/ w+'ed16:26
rlandylet's testproject that16:26
marios|ruckweshay|ruck: did you want to sync? i posted testproject train fs20 with depends-on that revert & also full-tempest last run failed https://review.rdoproject.org/r/2498816:39
marios|ruckweshay|ruck: that and others noted at https://etherpad.openstack.org/p/ruckroversprint2216:39
marios|ruckweshay|ruck: and rocky currently promoting (9 days old)         * http://38.145.34.55/centos7_rocky.log * 2020-02-17 15:46:38,552 16350 INFO     promoter Promoting the container images for dlrn hash e534332c1acf749d962bfa2015b16d9464f8dfbb on rocky to current-tripleo16:42
rfolcoweshay|ruck, rlandy https://review.opendev.org/#/c/706952 adds --debug to list-image only, not sure if this is what you really want16:46
weshay|ruckmarios|ruck, sure16:46
weshay|ruckrfolco, well.. we don't need it for anything other than getting the trace on uninstallable reqs from tripleo-common and python-tripleoclient16:47
weshay|ruckwe're blind w/o debug some where16:47
weshay|ruckmarios|ruck, https://meet.google.com/khx-kqex-vbf?authuser=116:47
marios|ruckweshay|ruck: k sec16:49
rlandyrfolco: ok if I testproject periodic-tripleo-centos-8-master-promote-promoted-components-to-tripleo-ci-testing?16:53
rfolcorlandy, please16:53
rfolcorlandy, knowing that its supposed to fail and not break anything16:53
rlandyrfolco: ack - let's just see that it fails legitimately and not on something we broke16:54
rfolcorlandy, yes that16:54
weshay|ruckmarios|ruck, https://opendev.org/openstack/tripleo-quickstart/src/branch/master/ci-scripts/full-deploy.sh16:55
*** marios|ruck is now known as marios|ruck|call16:55
rlandyrfolco:  https://review.rdoproject.org/r/24989 Test new promote wf16:56
rfolcorlandy, ok, the component label patch is in the oven, should be ready soon16:56
rlandy:)16:57
marios|ruck|callweshay|ruck: https://review.opendev.org/#/c/706797/1016:57
* rlandy waits for ping16:57
rlandywhy are there two sections for reviews needing attention?17:01
rlandyrfolco: ^^?17:01
* rlandy makes rfolco crazy today17:01
rlandycan we just use one?17:02
*** bogdando has quit IRC17:02
*** dtantsur is now known as dtantsur|afk17:03
*** marios|ruck|call is now known as marios|out17:04
rlandyhow do I rename a hackmd?17:09
rfolcorlandy, I warned weshay|ruck that I had created one first17:09
rfolcohe created one below17:09
rlandyrfolco: ok - pick whatever one you like17:09
rlandyrfolco: how do I rename a hackmd?17:10
*** udesale has quit IRC17:17
*** ccamacho has joined #oooq17:20
*** dpawlik has quit IRC17:22
rlandyrfolco: https://logserver.rdoproject.org/89/24989/1/check/periodic-tripleo-centos-8-master-promote-promoted-components-to-tripleo-ci-testing/6e6426b/job-output.txt17:30
rlandy ERROR! the role 'get_hash' was not found in17:30
rlandyrfolco: you need to add roles17:31
rfolcorlandy, I thought you have added already in your patch17:31
*** marios|out has quit IRC17:32
rlandyrfolco: base job17:32
rlandyhttps://review.rdoproject.org/r/#/c/24511/10/zuul.d/tripleo-rdo-base.yaml17:32
rlandyrfolco: I added them for container builds17:32
rlandytwo different jobs17:32
rlandyyou need to add the role where the playbook is called17:32
rfolcohmmm17:33
rfolcohttps://review.rdoproject.org/r/#/c/24962/2/zuul.d/tripleoci.yaml17:33
rfolcotrue17:33
rlandyrfolco: do you want me to patch?17:33
rfolcorlandy, please, if you can17:33
rlandysure - coming up17:34
sshnaidmzbr, what is the problem you see in patch https://review.opendev.org/#/c/706797/ ? what is not browsable there?17:43
rlandyrfolco: merged  - testproject'ing again17:49
*** derekh has quit IRC18:00
*** tesseract has quit IRC18:02
rfolcorlandy, https://review.rdoproject.org/r/#/c/24512/ is updated, please review it18:02
rlandylooking18:02
rfolcorlandy, still Depends-On: https://review.rdoproject.org/r/2493818:03
rfolcoso we need to merge this one first ^18:03
weshay|ruckrfolco,  just in the nick of time :)18:06
rlandyrfolco: I know ... weshay|ruck -1 that based on parenting strategy ... in progress  ... https://hackmd.io/4K_NU1PxSDiTYGecnAZNCA18:06
weshay|ruckthanks18:06
weshay|ruckrfolco, rlandy ya.. we need to expose the plan.. zuul can be complex :)18:07
rlandyweshay|ruck: yeah - busy writing that up18:07
rlandyjust looking at 24512 now18:07
rfolcowell, the plan is: 24512 depends on 2493818:07
rfolconow the deprecation patch gets much much simpler https://review.rdoproject.org/r/#/c/24987/18:08
*** jpena is now known as jpena|off18:09
rfolcobut still struggling w/ molecule18:09
weshay|ruckkopecmartin, please review https://review.opendev.org/#/c/706797/18:12
weshay|ruckthink this does what you were going after18:12
rlandyrfolco: weshay|ruck: so this will hose downstream but I think we just go for it and fix downstream afterwards18:12
rlandyit's so broken anyways18:12
*** jmasud has quit IRC18:14
*** jmasud_ has joined #oooq18:14
rfolcoits so hard to not break downstream if I don't see any jobs in upstream patches18:15
weshay|rucksshnaidm, https://review.opendev.org/#/c/705871/18:15
rlandyrfolco: ok - I +1'ed that ... will finish hackmd on strategy so we can merge the depend on18:16
rlandyrfolco: do you have any other patches that need to merge or is it just my component and integration pipeline ones now?18:17
rfolcorlandy, my last one is to deprecate get-hash, not really necessary18:18
rfolcorlandy, so now its all your patches18:18
rlandyrfolco: k - hold on, then - will ping you18:18
rfolcorlandy, ok meanwhile fixing molecule18:19
rlandyack18:19
rfolcoon deprecation patch18:19
*** amoralej is now known as amoralej|off18:52
rlandyrfolco: pls see https://review.rdoproject.org/r/#/c/24938/18:55
rlandyI ma nit sure of I should do all the standalone parenting in that job or create a separate one18:55
rlandyam not18:55
rlandythinking separate one18:56
*** sshnaidm is now known as sshnaidm|afk18:57
rfolcorlandy, separate is better19:01
rfolcoIMHO19:01
rlandyyep - doing that19:01
*** jmasud_ has quit IRC19:04
*** jmasud has joined #oooq19:06
*** Tengu has quit IRC19:07
*** Tengu has joined #oooq19:09
*** Tengu has quit IRC19:14
*** Tengu has joined #oooq19:15
*** Tengu has quit IRC19:21
*** Tengu has joined #oooq19:22
weshay|ruckrlandy, rfolco you folks want to parent centos-7 off of centos-8?19:26
rfolcono centos-8 jobs parent off a new base centos-8 parent job19:26
*** jmasud has quit IRC19:26
rlandyweshay|ruck: yep ... see https://review.rdoproject.org/r/#/c/24992/19:29
rlandyweshay|ruck: master only19:29
rlandybecause eventually we can ditch centos 719:30
rlandyif we keep parenting off centos7 it will never leave us19:30
weshay|ruckrlandy, k.. the alternative I can think of.. not sure if it's better19:31
weshay|ruckbut is to have a full line of centos-7 and a full line for centos-819:31
rlandyweshay|ruck: so I thought about that .... but duplication19:31
weshay|ruckk.. yes19:31
rlandyand if we change a setting in one, we forget the other etc. etc.19:32
rlandyweshay|ruck: I thought of changing the base jobs19:32
weshay|ruckright.. I would assume.. some things should not be inherited19:32
rlandythis thing can make you nuts19:32
weshay|ruckso you have duplication lower in the stream19:32
weshay|ruckya..19:32
rlandyso many options19:32
rlandylike buying cereal in america19:32
weshay|ruckI just want to make sure we're considering the options we have19:32
weshay|ruckso playing devil's advocate19:33
rlandyweshay|ruck: open to better suggestions here19:33
rlandychat?19:34
rlandyweshay|ruck: ^^?19:34
rlandylet's get this right as I am rewriting ALL the jobs now19:34
weshay|ruckit's worth discussing for sure19:34
weshay|ruckrfolco, ^19:35
weshay|ruckhttps://meet.google.com/wbv-xaxd-jxe?authuser=119:35
rlandyI figured I'd present at tomorrow's sync19:35
rlandyjoining19:35
rlandyrfolco: ^^ please19:35
weshay|ruckrlandy, ya.. we'll need to19:35
weshay|rucklet's not do it twice19:35
rlandyhttps://hackmd.io/4K_NU1PxSDiTYGecnAZNCA19:36
rlandyweshay|ruck: ^^19:36
rfolcosorry coming19:38
*** Tengu has quit IRC19:44
*** Tengu has joined #oooq19:46
*** openstackstatus has quit IRC19:49
*** Tengu has quit IRC19:51
*** Tengu has joined #oooq19:55
*** jaosorior has quit IRC20:02
*** jmasud has joined #oooq20:23
rfolcoweshay|ruck, rlandy started hackmd to ease visualization https://hackmd.io/WhSj5wLKQeSpzCHePUhZKQ?view20:38
rfolcouber time20:38
rfolcobbl20:38
rlandyrfolco: cool - do you want to link it to the other hackmd?20:39
rlandyrfolco: 2020-02-17 18:32:44.467603 | primary |   "msg": "The task includes an option with an undefined variable. The error was: 'workspace' is undefined\n\nThe error appears to be in '/var/lib/zuul/builds/51df5e6b49f944f39fa3e37ca218aa24/trusted/project_3/review.rdoproject.org/rdo-infra/ci-config/ci-scripts/infra-setup/roles/get_hash/tasks/get_hash.yaml': line 3, column 3, but may\nbe elsewhere in the file depending on the exact20:45
rlandysyntax problem.\n\nThe offending line appears to be:\n\n\n- name: Ensure legacy workspace directory\n  ^ here\n"20:45
rlandyhttps://logserver.rdoproject.org/89/24989/1/check/periodic-tripleo-centos-8-master-promote-promoted-components-to-tripleo-ci-testing/51df5e6/job-output.txt20:45
*** jfrancoa has quit IRC20:58
*** Goneri has quit IRC20:59
rlandyhttps://review.rdoproject.org/r/24995 Test multinode centos 821:01
rlandyweshay|ruck: rfolco: ^^ let's see the damage21:02
weshay|ruckthanks21:02
weshay|ruckrlandy++21:02
weshay|ruckvisible > unknown21:02
rlandybetter to know what we don't know21:03
rlandyugh - trying again21:17
*** TrevorV has quit IRC21:33
rlandyhttps://logserver.rdoproject.org/95/24995/1/check/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-master/139c5be/job-output.txt21:58
rlandyrfolco: weshay|ruck: no bad  fails at container login21:59
weshay|ruckrlandy, just saw.. that's an unexpected place to fail21:59
rlandyshouldn't fail there22:00
rlandyinvalid username/password\" in registry_login_podman.results.0.stderr",22:00
rlandylooks like we are missing a secret22:01
rlandywhich is weird22:02
weshay|ruckwhere is this file tripleo_podman_login.yml22:02
rlandyall I changed is the nodeset22:02
weshay|rucknot what I asked22:03
weshay|ruckdidn't ask what you changed22:03
rlandyhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/dfc842f50d513173442be16e7ed9a857aa8c7258/playbooks/tripleo-rdo-base/container-login.yaml22:04
weshay|ruckya22:04
weshay|ruckhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-rdo-base/container-login.yaml#L3322:04
*** Goneri has joined #oooq22:04
weshay|ruckhttp://codesearch.openstack.org/?q=tripleo_podman_login&i=nope&files=&repos=22:05
weshay|ruckmust be in the registry setup role22:05
rlandyhttps://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_podman/tasks/tripleo_podman_login.yml22:07
rlandyweshay|ruck: ^^22:07
weshay|ruckperfect.. thanks22:07
weshay|ruckrlandy, perhaps we're missing a playbook that sets up.. REGISTRY_USERNAME22:08
weshay|ruckmeh22:08
rlandyfor multinode22:09
weshay|ruckhttps://github.com/rdo-infra/review.rdoproject.org-config/search?q=REGISTRY_USERNAME&unscoped_q=REGISTRY_USERNAME22:09
rlandyyeah - need to expand that22:10
rlandyrfolco: around?22:28
rlandyotherwise I'll fix22:28
rlandyhttps://review.rdoproject.org/r/24996 Add misisng workspace var for base promote hash playbook22:39
rlandyrfolco: ^^22:39
*** openstackstatus has joined #oooq23:01
*** ChanServ sets mode: +v openstackstatus23:01
rlandyhttps://logserver.rdoproject.org/89/24989/1/check/periodic-tripleo-centos-8-master-promote-promoted-components-to-tripleo-ci-testing/3a2dbc7/job-output.txt23:37
rlandyrfolco: ^^ looks better :)23:37
*** tosky has quit IRC23:49

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!