Thursday, 2020-09-10

« Wednesday, 2020-09-09 Index Friday, 2020-09-11 »
*** rlandy has quit IRC00:21
*** udesale has joined #oooq03:07
*** skramaja has joined #oooq03:24
*** ratailor has joined #oooq03:46
*** ykarel|away has joined #oooq04:22
*** ykarel|away is now known as ykarel04:23
*** ykarel has quit IRC04:30
*** bhagyashris|rove is now known as bhagyashri|rover04:32
*** ykarel has joined #oooq04:37
*** ykarel_ has joined #oooq04:51
*** ykarel has quit IRC04:54
*** marios has joined #oooq04:54
*** ykarel__ has joined #oooq05:12
*** ykarel_ has quit IRC05:14
*** jtomasek has joined #oooq05:17
*** ysandeep|away is now known as ysandeep05:18
*** ykarel__ is now known as ykarel05:22
*** jtomasek has quit IRC05:22
*** ykarel_ has joined #oooq06:01
*** ykarel has quit IRC06:03
*** ykarel_ is now known as ykarel06:15
*** jbadiapa has joined #oooq06:47
*** jaosorior has joined #oooq06:58
*** jtomasek has joined #oooq07:00
*** amoralej|off is now known as amoralej07:11
*** bogdando has joined #oooq07:25
*** tosky has joined #oooq07:32
*** jpena|off is now known as jpena07:40
*** saneax has joined #oooq07:53
*** ysandeep is now known as ysandeep|lunch08:34
*** ykarel_ has joined #oooq08:35
*** ykarel has quit IRC08:36
*** ykarel_ is now known as ykarel|lunch08:40
*** jtomasek has quit IRC08:45
*** apetrich has joined #oooq08:46
*** apetrich has quit IRC08:50
*** apetrich has joined #oooq08:52
zbrchandankumar: arxcruz|ruck: https://review.opendev.org/#/c/750904/ -- fixed regression from last night, as infra switched to focal for py38 testing.08:52
*** derekh has joined #oooq08:52
zbrif curious, the broken jobs can be seen on https://review.opendev.org/#/c/746890/08:53
bhagyashri|roverchandankumar, around?08:53
*** ykarel|lunch has quit IRC09:12
*** dtantsur|afk is now known as dtantsur09:14
*** ykarel has joined #oooq09:19
*** ysandeep|lunch is now known as ysandeep09:35
*** skramaja has quit IRC09:39
*** skramaja has joined #oooq09:39
zbrmarios: are you around? the ^ is blocking me from doing anything on e-r.09:40
marioszbr: o/09:43
zbrthanks!09:49
*** jaosorior has quit IRC10:24
bhagyashri|roverarxcruz|ruck, hi good afternoon i want to know about stein is there any open issue for stein?10:50
*** udesale_ has joined #oooq10:59
*** udesale has quit IRC11:01
*** sanjayu_ has joined #oooq11:01
*** saneax has quit IRC11:02
*** sanjayu__ has joined #oooq11:04
*** sanjayu_ has quit IRC11:07
*** jpena is now known as jpena|lunch11:31
zbre-r container building ready for review: https://review.opendev.org/#/c/750958/11:48
*** amoralej is now known as amoralej|lunch12:03
*** arxcruz|ruck is now known as arxcruz|pto12:05
*** rlandy has joined #oooq12:06
mariosbhagyashri|rover: fyi filed that but am working on it already https://bugs.launchpad.net/tripleo/+bug/189513812:09
openstackLaunchpad bug 1895138 in tripleo "centos-8 standalone-upgrade-ussuri fails build-test-packages issue creating /root/DLRN" [Critical,Triaged] - Assigned to Marios Andreou (marios-b)12:09
bhagyashri|rovermarios, ack thank you :)12:10
*** rfolco has joined #oooq12:11
*** rfolco is now known as rfolco|ruck12:18
rlandyysandeep: weshay|ruck: rhos-17 is back to passing ... can promote today - 16.2 as well12:18
rlandyalso need to promote client components to fix baremetal12:19
rlandyper https://bugzilla.redhat.com/show_bug.cgi?id=187699912:19
openstackbugzilla.redhat.com bug 1876999 in tripleo-ansible "OSP-17: OVB and baremetal jobs are failing the overcloud deployment on 'Create plan' - swift client command" [Unspecified,On_dev] - Assigned to ramishra12:19
rlandychecking versions now12:19
rlandyysandeep: pojadhav: hi - we should probably also sit together to work out multinode - it's never really passed downstream12:20
rlandytomorrow?12:21
ysandeeprlandy, 17?12:21
ysandeepmultinode in 17?12:21
rlandyI think both - last time I spoke with pojadhav I think there were issues on 16.2 as well12:21
rlandyperiodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-rhos-17openstack/tripleo-cimasteropenstack-periodic-rhos-17master56 mins 35 secs2020-09-10 01:00:21FAILURE12:22
rlandy^^ failing a long time12:22
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?pipeline=openstack-periodic-rhos-17&job_name=periodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-rhos-1712:22
rlandysince 8/1112:22
rlandyysandeep: checking if we can fix bm and ovb on 17 first though12:22
ysandeeprlandy we debug 16.2 multinode and found this issue http://pastebin.test.redhat.com/900658 . I posted a patch - https://review.opendev.org/#/c/750900/ which will probably fix 16.2 multinode12:24
pojadhavrlandy, ack12:24
rlandyk - we should look at 17 as well12:25
ysandeeprlandy, i think its failing with same error as ovb and bm - https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-tripleo-rhos-17/fa11c27/logs/undercloud/home/zuul/overcloud_deploy.log12:26
rlandyysandeep: ok- so I'm working on that12:26
weshay|ruckrlandy, k12:27
ysandeeprlandy, rabi have provided some useful comments for bz.. that will clear the issue.12:28
rlandyysandeep: yeah - but I don't want to pin if we can promote12:28
rlandybetter to keep it clean12:29
*** ratailor has quit IRC12:29
ysandeepi got info from shrestha (she spoke with rabi).. they updated the deps as per upstream - python-ansible-runner-1.4.6-3.el8ost is available for 17.0, will available in compose when automated compose run12:30
ysandeeprlandy, weshay|ruck funny issue with envc.. a pendrive is attached to one of the envc baremetal.. in introspection pendrive is detecting as root disk hence no valid host.. Do you know if we can disable pendrive i.e from idrac setting/ bios12:32
rlandyysandeep: not sure - you can boot into the bios and dig around there12:34
ysandeeprlandy, ack..12:35
ysandeepI laughed a lot when i found a pendrive is causing this :)12:36
rlandypython-ansible-runner-1.4.6-3.el8ost we got at 6 in the morning12:37
rlandyso we need to rebuild 17 containers12:37
rlandyand get tripleo-ansible patch12:37
ysandeepyup.. currently in gate12:38
*** jpena|lunch is now known as jpena12:39
rlandyysandeep: let me see if I can rebuild 17 with the changed dep and run OVB12:39
ysandeepsure12:39
rlandyweshay|ruck: oh ... dependency approach 2 is running now12:41
rlandyworks fine if we place the file in tq12:41
weshay|ruckah nice12:47
*** ykarel_ has joined #oooq12:48
*** pojadhav is now known as pojadhav|brb12:50
*** ykarel has quit IRC12:51
*** derekh has quit IRC12:58
*** pojadhav|brb is now known as pojadhav13:02
weshay|ruck zbr, rlandy, marios, ysandeep,  svyas, pojadhav, akahat, weshay,13:04
weshay|ruckmtg13:04
weshay|ruckzbr, need you :)13:05
akahatmarios, your ssh key issue might be because "\n" in ssh key...13:07
*** sanjayu__ has quit IRC13:16
*** amoralej|lunch is now known as amoralej13:19
*** ykarel_ is now known as ykarel13:25
*** Goneri has joined #oooq13:26
mariosakahat: thanks13:27
*** TrevorV has joined #oooq13:30
rfolco|ruckbhagyashri|rover, do we have a bug for the container build in master ?14:03
rfolco|ruckbhagyashri|rover,  Get http://trunk.registry.rdoproject.org/v2/: dial tcp 38.102.83.107:80: connect: connection refused14:03
rfolco|ruckbhagyashri|rover, just happened once, nevermind14:04
bhagyashri|roverbhagyashri|rover, nope but it pass here https://review.rdoproject.org/r/#/c/29227/14:04
bhagyashri|roveryes14:04
bhagyashri|roverwaiting for the next run14:07
*** ykarel is now known as ykarel|away14:18
akahatmarios, it worked?14:19
mariosakahat: don't know didn't try yet14:19
akahatokya14:19
*** jtomasek has joined #oooq14:24
ysandeepweshay|ruck, https://review.opendev.org/#/c/750900/14:29
weshay|ruckhttps://review.opendev.org/#/c/733659/14:31
bhagyashri|roverweshay|ruck, i am worried about stein promotion. Others are doing good :)14:39
bhagyashri|roverweshay|ruck, so should we take care about stein  atm14:39
*** ykarel|away has quit IRC14:45
weshay|ruckbhagyashri|rover, k.. /me looks14:46
weshay|ruckbhagyashri|rover, let's start w/ periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein14:48
weshay|ruckbhagyashri|rover, if you want.. we can jump on a call14:49
weshay|ruckand debug together14:49
weshay|ruck2020-09-10 02:14:05 | 2020-09-10 02:14:05.665 102786 ERROR openstack [-] Error: The following files were not found: /home/zuul/containers-default-parameters.yaml: CommandError: Error: The following files were not found: /home/zuul/containers-default-parameters.yaml14:50
weshay|ruckhttps://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein/9b5a301/logs/undercloud/home/zuul/overcloud_update_prepare.log.txt.gz14:50
bhagyashri|roverok14:50
weshay|ruckperhaps we copy... https://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein/9b5a301/logs/undercloud/home/zuul/containers-prepare-parameter.yaml.txt.gz14:50
weshay|ruckto containers-default-parameters.yaml14:51
bhagyashri|roverhttps://meet.google.com/wzg-jvrm-udw14:51
bhagyashri|roverweshay|ruck, ^14:51
weshay|ruckbhagyashri|rover, test project... a change that does that.. w/ periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein and periodic-tripleo-ci-centos-7-standalone-upgrade-stein14:52
rfolco|ruckbhagyashri|rover, weshay|ruck: what about ussuri timing out ? let it run next round or testproject it ?14:59
weshay|ruck- job:14:59
weshay|ruck    name: tripleo-ci-centos-7-scenario010-multinode-oooq-container14:59
weshay|ruck    parent: tripleo-ci-base-multinode14:59
weshay|ruck    voting: false14:59
weshay|ruck    branches: ^(stable/(queens|rocky|stein)).*$14:59
weshay|ruck    vars:14:59
weshay|ruck      nodes: 1ctlr14:59
weshay|ruck      featureset: '038'14:59
weshay|ruck      extra_tags:14:59
weshay|ruck        - octavia14:59
weshay|ruckbhagyashri|rover, ^14:59
weshay|ruckrfolco|ruck, I'm not sure what ur referring to yet15:00
rfolco|ruckbhagyashri|rover raised this15:01
rfolco|ruckhttps://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable1&job_name=periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset030-ussuri15:01
weshay|ruckbhagyashri|rover, rfolco|ruck fyi https://opendev.org/openstack/openstack-tempest-skiplist15:01
rfolco|ruckweshay|ruck, its timing out on deployment, not tempest15:04
rfolco|ruckweshay|ruck, what did you mean?15:04
weshay|ruckwell.. a job name would help15:04
rfolco|ruckhttps://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable1&job_name=periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset030-ussuri15:05
rfolco|ruckbrb15:07
bhagyashri|roverweshay|ruck, thanks15:11
chandankumarweshay|ruck: when get some time, need some pointer here https://review.opendev.org/749645 https://review.opendev.org/#/c/749645/40/roles/container-build/tasks/non_tripleo_containers.yml@2915:11
chandankumarweshay|ruck: while pushing it is giving Error: Error copying image to the remote destination: Error writing manifest: Error uploading manifest v0.17.0 to 0.0.0.0:5001/tripleomaster/node-exporter: received unexpected HTTP status: 500 Internal Server Error15:13
bhagyashri|roverweshay|ruck, rfolco|ruck jfyi, i had pushed patch https://review.opendev.org/#/c/750987/ but that was already address by rabimishra https://review.opendev.org/#/c/750902/315:13
chandankumarweshay|ruck: logs here https://a8f39c72b3f232748c21-c5cf373c2a0096f484e7e3416a394fa7.ssl.cf2.rackcdn.com/749645/40/check/tripleo-ci-centos-8-content-provider/0c12257/job-output.txt15:13
chandankumarotherwise I need to reproduce tomorrow15:13
weshay|ruckrfolco|ruck, can you please try 030 w/ a very long timeout in a testproject15:14
weshay|rucklet's see if we just need to adjust zuul settings or if we have a real issue15:14
weshay|ruckchandankumar, sec15:18
zbrrlandy weshay|ruck: when you have few minutes, take a look https://review.opendev.org/#/c/750958/ -- and thy running make locally on it.15:18
rlandysec - just reseating hardware15:20
rlandyanother fun job15:20
weshay|ruckrlandy, join #rhos-infrared15:20
weshay|ruckrlandy, please help out Attila15:20
weshay|ruckchandankumar, hrm.. interesting15:22
weshay|ruckchandankumar, why are you using docker://15:22
weshay|ruckchandankumar, do you have a running node w/ a regitry?15:24
weshay|ruckregistry?15:24
weshay|ruckchandankumar, if not.. let's get one running locally or in a tenant15:24
weshay|ruckchandankumar, we should be able to figure this out .. outside of a job15:24
weshay|ruckwdyt15:25
weshay|ruckchandankumar, perhaps use buildah push vs. podman15:26
weshay|ruckhttps://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/image/builder/buildah.py#L18215:26
weshay|ruckand docker:// looks right I guess15:26
chandankumarchecking15:35
*** skramaja has quit IRC15:35
weshay|ruckchandankumar, also we have a podman module15:38
weshay|ruckchandankumar, also need --debug turned on15:38
*** skramaja has joined #oooq15:38
weshay|ruckchandankumar, also does the running registry have a log?15:40
weshay|ruckchandankumar, not seeing much inhttps://a8f39c72b3f232748c21-c5cf373c2a0096f484e7e3416a394fa7.ssl.cf2.rackcdn.com/749645/40/check/tripleo-ci-centos-8-content-provider/0c12257/logs/undercloud/var/log/extra/journal.txt15:40
rlandyysandeep: I have node a idrac reset and a virtual reset on that machine - still same network error - pls put in tocket15:42
rlandyticket15:42
chandankumarweshay|ruck: running registry does not have logs, updated the patch to pull the images once tcib build is done15:42
chandankumarwith verbose15:43
ysandeeprlandy, thank you for trying.. i will log a ticket15:43
rlandyysandeep: pls include all the machines you want looked at in the ticket15:48
rlandyso we get everything updated as you need it15:48
ysandeeprlandy, today i am opening ticket for envc baremetal node to remove pendrive, i will check all the node one by one and will open a ticket for them to update firmware/ console issue15:51
rlandyysandeep: thanks - ping if you need help15:51
*** skramaja has quit IRC15:59
weshay|ruckchandankumar, https://docs.docker.com/engine/reference/commandline/registry_joblogs/16:02
weshay|ruck:(16:02
weshay|ruckchandankumar, imho.. we shouldn't use the registry container16:02
weshay|ruckwe should use the same dang registry every job has16:02
weshay|rucknot sure what you think about it16:02
*** marios is now known as marios|out16:04
*** amoralej is now known as amoralej|off16:09
*** udesale_ has quit IRC16:11
rlandyysandeep: rebuilding and pushing 17 containers ... https://code.engineering.redhat.com/gerrit/#/c/195884/16:15
rlandyif that works, we can promote16:15
rlandyand get new deps16:16
rlandyupdated python-ansible-runner16:16
*** marios|out has quit IRC16:21
*** dtantsur is now known as dtantsur|afk16:22
ysandeeprlandy, thanks! fyi.. i have cc'ied you in pnt ticket which i opened for envc pendrive removal16:24
rlandyzbr: got your review https://review.opendev.org/#/c/750958 ... ran make build ...16:25
rlandydocker build -t elastic-recheck .16:25
rlandySending build context to Docker daemon 119.9 MB16:25
rlandyStep 1/10 : FROM opendevorg/python-builder:3.7 as elastic-recheck-builder16:25
rlandyError parsing reference: "opendevorg/python-builder:3.7 as elastic-recheck-builder" is not a valid repository/tag: invalid reference format16:25
rlandymake: *** [Makefile:27: build] Error 116:25
ysandeepSee you tomorrow guys o/16:25
*** ysandeep is now known as ysandeep|away16:25
rlandyysandeep|away: thanks16:25
*** bogdando has quit IRC16:34
weshay|ruckrfolco|ruck, rlandy ysandeep|away fyi.. https://review.rdoproject.org/r/#/c/29218/ adding scenario01216:44
weshay|ruckrfolco|ruck, keep an aye.. and add as criteria if they pass16:44
rlandyok16:44
rfolco|ruckweshay|ruck, k16:44
rlandyweshay|ruck: where do you want to go with the dep pipeline work - go forward with approach 2 and add stream job to the pipeline?16:45
*** jpena is now known as jpena|off16:49
weshay|ruckrlandy, ya.. let's roll w/ approach #217:16
*** jbadiapa has quit IRC17:18
rlandyweshay|ruck: ack - ok17:27
weshay|ruckrfolco|ruck, re: cix board https://bugs.launchpad.net/tripleo/+bug/1892702 is marked dupe of https://bugs.launchpad.net/tripleo/+bug/189079818:15
openstackLaunchpad bug 1890798 in tripleo "duplicate for #1892702 periodic centos8 Ussuri multinode minor update job fails: tderr": "Error: resource 'ip-192.168.24.16' is not running on any node" [Critical,Fix released]18:15
openstackLaunchpad bug 1890798 in tripleo "periodic centos8 Ussuri multinode minor update job fails: tderr": "Error: resource 'ip-192.168.24.16' is not running on any node" [Critical,Fix released]18:15
rfolco|ruckweshay|ruck, I did not get there yet... which cix is that one18:16
rfolco|ruckweshay|ruck, question... how this would help with the conflicts bug ? https://review.opendev.org/#/c/713204/18:20
weshay|rucksearch by the bug id18:20
weshay|ruckthat's what the patch says18:20
weshay|ruckbut taking their word for it18:20
rfolco|ruckok, so its wip18:20
rfolco|ruckweshay|ruck, did you move 1892702 to done ?18:21
rfolco|ruckweshay|ruck, the dup one you mentioned18:21
weshay|ruckI haven't moved anything there recently18:22
weshay|ruckshould say in the history18:22
rfolco|ruckweshay|ruck, you mentioned 2 cix cards that are done/dead, I was trying to understand why18:31
rlandywoohoo - multinode on 17 is fixed18:32
rlandymaybe also bm and ovb?18:32
weshay|ruckah nice18:32
weshay|ruckrfolco|ruck, just following the lp bugs18:32
weshay|ruckone was marked as dupe.. both the dupe and the orig were marked promotion_blocker18:33
weshay|ruckso there may be a dupe cix card18:33
weshay|ruckrefer to the bug numbers18:33
rfolco|ruckweshay|ruck, got it, both cix cards are done.18:33
weshay|ruckrock on18:35
*** rfolco has joined #oooq18:54
*** rfolco|ruck has quit IRC18:56
rlandyjust rerunning ovb on 16.2 - then we can promote19:10
*** rfolco is now known as rfolco|ruck19:12
weshay|ruckrlandy, k.. ping me if you want19:13
rlandyweshay|ruck: I think maybe I19:13
rlandyll do the promotion tomorrow morning with sandeep19:13
rlandyso he can see it19:13
weshay|ruckok. +119:13
weshay|ruckrlandy, please ask him about selinux19:14
weshay|ruckwe need to get that fixed19:14
rlandyweshay|ruck: we've had enough promoter fun for one week19:14
weshay|ruckaye19:14
rlandyack - will do19:14
weshay|ruckrlandy, I suppose w/ Attila's jobs passing .. we should be ok19:14
weshay|ruck99% sure they are default selinux enabled19:14
rlandyweshay|ruck: I think just disabled selinux for debug19:15
rlandyit should be put back19:15
rlandyone setting per env19:15
rlandyweshay|ruck: attila is rocking it though19:16
rlandycomponent jobs and everything19:16
* rlandy retires and buys a yacht 19:16
weshay|ruckyup19:16
weshay|ruckwell done.. enjoy the high life19:17
rlandyweshay|ruck: 's/enjoy the high life/enjoy working on parent-child jobs/'19:17
weshay|ruckrlandy, once we make progress there.. it may also pay to show it to Attila/Pavel.. see if they are interested in jumping in19:19
weshay|rucknot sure.. but it's a thought19:19
weshay|ruckany new deps kill them as much as it kills us19:19
rlandyweshay|ruck: sure, misery loves company - sure19:22
rlandymore usage, the better19:23
weshay|ruckrlandy, I think the first demo should go to the prod-chain.. perhaps interest will be raised there19:24
rfolco|ruckweshay|ruck, I don't know what to do with this one https://bugs.launchpad.net/tripleo/+bug/188763319:24
openstackLaunchpad bug 1887633 in tripleo "periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-train job fails with pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'overcloud.ctlplane.ooo.test' ([Errno 113] EHOSTUNREACH)")" [Critical,Triaged]19:24
rlandyok - on it19:24
rlandyrfolco|ruck: that shoudl be on ade's plate19:25
rlandynow that we did our bit to get master passing19:25
rfolco|ruckrlandy, ade lee ?19:25
rlandyhttps://review.opendev.org/#/c/741035/19:25
rlandyshoot - patch failed gates19:25
rlandyrfolco|ruck: yeah  - I will work with him on it when ^^ passes gates19:27
rfolco|ruckrlandy, this relates to another bug19:27
rfolco|ruckrlandy, is it the same issue but on master ?19:28
rlandyrfolco|ruck: the basic thought is that we are missing a backport in train19:28
rlandydiff from master19:28
rlandymaster was related to the DNS servers on diff cloud providers19:28
rlandybut they would not touch train unti master was working19:28
rfolco|ruckok so first fix master issue w/ https://review.opendev.org/#/c/741035/19:29
rfolco|ruckthen try fixing train (diff issue)19:29
rfolco|ruckrlandy, did I get it right ?19:30
rlandyyes19:30
rfolco|ruckok cool thanks19:30
rlandyweshay|ruck: so I can just delete the CR-repo jobs and flat replace them with stream, right?19:47
rlandyno use for CR anymore19:48
rlandy?19:48
weshay|ruckrlandy, fine by me19:48
rlandyk- here we go19:48
*** Trevor_V has joined #oooq19:53
*** TrevorV has quit IRC19:57
*** rfolco|ruck has quit IRC21:00
*** Trevor_V has quit IRC21:09
*** jtomasek has quit IRC21:11
*** apetrich has quit IRC21:48
*** Goneri has quit IRC21:49
*** rlandy has quit IRC22:04
*** tosky has quit IRC22:42
*** sanjayu__ has joined #oooq23:05
« Wednesday, 2020-09-09 Index Friday, 2020-09-11 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!