Monday, 2020-02-10

*** joelinux has quit IRC00:00
*** cgoncalves has quit IRC00:00
*** dsneddon has quit IRC00:01
*** Pharaoh_Atem has quit IRC00:02
rdogerritrdo-trunk proposed rdoinfo master: Promote CBS tags update for rocky-release  https://review.rdoproject.org/r/2485600:04
rdogerritrdo-trunk proposed rdoinfo master: Bump rdoinfo ussuri-uc tags to latest upper-constraints  https://review.rdoproject.org/r/2479600:05
*** dsneddon_ has quit IRC00:49
*** cloudnull has quit IRC01:04
*** cloudnull has joined #rdo01:05
*** slaweq has joined #rdo01:06
*** openstackstatus has joined #rdo01:17
*** ChanServ sets mode: +v openstackstatus01:17
*** dsneddon_ has joined #rdo01:21
*** dsneddon_ has quit IRC01:26
*** dsneddon_ has joined #rdo01:53
*** dsneddon_ has quit IRC01:58
*** jmlowe has joined #rdo02:21
*** dsneddon_ has joined #rdo02:27
*** jmlowe has quit IRC02:28
*** jdandrea has joined #rdo02:31
*** dsneddon_ has quit IRC02:35
*** chasmo77 has quit IRC02:43
*** dsneddon_ has joined #rdo03:03
*** dsneddon_ has quit IRC03:09
*** dsneddon_ has joined #rdo03:35
*** dsneddon_ has quit IRC03:40
*** tkajinam has quit IRC03:40
*** tkajinam has joined #rdo03:41
*** ysandeep|afk is now known as ysandeep04:03
*** udesale has joined #rdo04:09
*** dsneddon_ has joined #rdo04:13
*** dsneddon_ has quit IRC04:18
*** ykarel|away is now known as ykarel04:27
*** ramishra has joined #rdo04:29
*** ykarel is now known as ykarel|afk04:34
rdogerritMerged rdoinfo master: Promote CBS tags update for rocky-release  https://review.rdoproject.org/r/2485604:39
rdogerritMerged rdoinfo master: Promote CBS tags update for rocky-testing  https://review.rdoproject.org/r/2486004:39
rdogerritMerged rdoinfo master: Promote CBS tags update for train-testing  https://review.rdoproject.org/r/2486204:39
*** skramaja has joined #rdo04:46
*** jdandrea has quit IRC04:48
*** dsneddon_ has joined #rdo04:54
*** ykarel|afk is now known as ykarel04:56
*** dsneddon_ has quit IRC04:59
rdogerritYatin Karel created openstack/neutron-tempest-plugin-distgit rpm-master: [DNM] Test with drop py2 patch  https://review.rdoproject.org/r/2486305:07
*** dsneddon_ has joined #rdo05:32
*** dsneddon_ has quit IRC05:37
*** dsneddon_ has joined #rdo05:42
*** raukadah is now known as chkumar|rover05:44
*** dsneddon_ has quit IRC05:46
*** dsneddon_ has joined #rdo06:17
*** marios has joined #rdo06:21
*** dsneddon_ has quit IRC06:22
*** aufi has joined #rdo06:30
*** lmiccini has joined #rdo06:30
*** cgoncalves has joined #rdo06:32
*** dpawlik has joined #rdo06:44
*** bandini has joined #rdo06:47
rdogerritMerged rdoinfo master: Promote CBS tags update for stein-testing  https://review.rdoproject.org/r/2486106:49
*** jfrancoa has joined #rdo06:49
*** dpawlik has quit IRC06:50
*** dpawlik has joined #rdo06:55
*** jlibosva has joined #rdo06:57
*** dsneddon_ has joined #rdo06:57
*** numans_ has joined #rdo07:02
*** dsneddon_ has quit IRC07:02
*** jtomasek has joined #rdo07:07
*** jtomasek_ has joined #rdo07:09
*** jtomasek has quit IRC07:12
*** dpawlik has quit IRC07:16
*** dpawlik has joined #rdo07:19
*** dsneddon_ has joined #rdo07:31
*** aufi has quit IRC07:35
*** tosky has joined #rdo07:35
*** dsneddon_ has quit IRC07:36
*** dciabrin has joined #rdo07:37
*** ykarel is now known as ykarel|lunch07:38
*** ysandeep is now known as ysandeep|lunch07:38
*** Tengu has quit IRC07:44
*** Tengu has joined #rdo07:46
*** tmazur has joined #rdo07:47
rdogerritChandan Kumar created rdo-jobs master: [DNM] master promotion  https://review.rdoproject.org/r/2486407:47
*** slaweq has quit IRC07:53
*** maciejjozefczyk has joined #rdo08:07
*** slaweq has joined #rdo08:09
*** iurygregory has joined #rdo08:10
*** dsneddon_ has joined #rdo08:10
*** dsneddon_ has quit IRC08:16
*** marios has quit IRC08:16
*** jcapitao has joined #rdo08:19
*** gfidente has joined #rdo08:20
*** tkajinam has quit IRC08:23
*** tesseract has joined #rdo08:30
*** ysandeep|lunch is now known as ysandeep08:32
*** ykarel|lunch is now known as ykarel08:33
rdogerritChandan Kumar created rdo-jobs master: [DNM] master promotion  https://review.rdoproject.org/r/2486508:35
rdogerritChandan Kumar proposed rdo-jobs master: [DNM] train promotion  https://review.rdoproject.org/r/2486508:37
*** ralonsoh has joined #rdo08:38
*** amoralej|off is now known as amoralej08:38
*** rpittau|afk is now known as rpittau08:41
*** alexmcleod has joined #rdo08:42
*** dsneddon_ has joined #rdo08:46
*** rascasoft has joined #rdo08:48
*** migarcia|away is now known as migarcia08:49
*** dsneddon_ has quit IRC08:51
*** jpich has joined #rdo08:54
*** jpena|off is now known as jpena08:54
*** dtantsur|afk is now known as dtantsur09:01
*** jlibosva has quit IRC09:06
*** apevec has joined #rdo09:09
rdogerritMatthias Runge created puppet/puppet-tripleo-distgit train-rdo: Drop dependencies to removed services  https://review.rdoproject.org/r/2486609:20
*** apetrich has joined #rdo09:21
*** dsneddon_ has joined #rdo09:23
*** dsneddon_ has quit IRC09:27
rdogerritAlfredo Moralejo created rdo-infra/ci-config master: Promote puppet-passed-ci only in the public server  https://review.rdoproject.org/r/2486709:28
rdogerritMartin Schuppert proposed rdo-jobs master: [DNM] testing https://review.opendev.org/706741  https://review.rdoproject.org/r/2462609:29
dciabrinamoralej, hey o/ just saw your ping last week about mariadb 10.3 pkg not installing. Do you know how I can rebuild the mariadb container like rdo does?09:32
rdogerritMartin Schuppert created rdo-jobs master: [DNM] testing https://review.opendev.org/706741 cell  https://review.rdoproject.org/r/2486809:33
amoralejdciabrin, the issue i saw was not in creating containers09:33
amoralejjust trying to install the pkg09:33
dciabrinamoralej, right, i would like to spare you more work and try a full container image install to avoid back and forth :D09:34
dciabrini obviously didn't try simple install in the first place, so...09:34
amoralejdciabrin, you can use kolla with tripleo-common overrides09:34
amoralejand manually add the ci temporary repo09:35
amoralejwell, or your own repo with your local rebuild09:36
dciabrinamoralej, ack I'll try that and come back to you shortly. sorry for the hiccup09:36
rdogerritChandan Kumar proposed config master: Add periodic-tripleo-ci-centos-7-ovb-1ctlr_1cellctrl_1comp-featureset063 to master and train  https://review.rdoproject.org/r/2480009:36
amoralejdciabrin, https://logserver.rdoproject.org/62/24862/1/check/rdoinfo-tripleo-train-centos-7-scenario001-standalone/95c137e/logs/build_containers.sh09:37
amoralejthat's the script in ci09:37
amoralejif it helps09:37
amoralejit's missing the repo config part09:37
dciabrinamoralej, great, totally what i was looking for!09:37
*** derekh has joined #rdo09:38
*** sshnaidm has quit IRC09:41
*** avivgt has joined #rdo09:41
*** sshnaidm has joined #rdo09:42
rdogerritMerged rdo-jobs master: Adds fs063 multicell periodic job for master and train  https://review.rdoproject.org/r/2480109:43
*** marios has joined #rdo09:47
*** paramite has joined #rdo09:50
rdogerritAlfredo Moralejo proposed rdo-infra/ci-config master: Promote puppet-passed-ci only in the public server  https://review.rdoproject.org/r/2486709:53
*** jpena is now known as jpena|brb09:58
*** aufi has joined #rdo09:59
*** dsneddon_ has joined #rdo10:03
*** dsneddon_ has quit IRC10:08
rdogerritAlfredo Moralejo proposed rdo-infra/ci-config master: Promote puppet-passed-ci only in the public server  https://review.rdoproject.org/r/2486710:09
rdogerritChandan Kumar created rdo-jobs master: [DNM] stein scenario 2 and 4 tempest failure  https://review.rdoproject.org/r/2486910:10
*** dsneddon_ has joined #rdo10:15
*** Tengu has quit IRC10:17
*** Tengu has joined #rdo10:18
rdogerritMerged config master: Add periodic-tripleo-ci-centos-7-ovb-1ctlr_1cellctrl_1comp-featureset063 to master and train  https://review.rdoproject.org/r/2480010:24
rdogerritChandan Kumar proposed rdo-jobs master: [DNM] master promotion  https://review.rdoproject.org/r/2486410:28
rdogerritChandan Kumar proposed rdo-jobs master: [DNM] train promotion  https://review.rdoproject.org/r/2486510:28
rdogerritAlfredo Moralejo proposed rdo-infra/ci-config master: Promote puppet-passed-ci only in the public server  https://review.rdoproject.org/r/2486710:31
rdogerritYatin Karel proposed rdo-infra/ci-config master: Promote puppet-passed-ci only in the public server  https://review.rdoproject.org/r/2486710:31
ykarel:(, same time10:31
*** Tengu has quit IRC10:31
*** Tengu has joined #rdo10:40
*** jpena|brb is now known as jpena10:43
*** udesale has quit IRC10:48
ykarelchkumar|rover, can u review https://review.rdoproject.org/r/#/c/24867/11:00
*** jlibosva has joined #rdo11:04
chkumar|roverjpena, hello11:06
chkumar|roverjpena, please have a look at this https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-singlenode-featureset027-train/0c7fdcd/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz11:06
chkumar|roverjpena, container image prepare for train is failing in most of the job during undercloud job,11:07
jpenachkumar|rover: I'm currently in a meeting, I'll have a look at it as soon as I'm finished11:07
chkumar|roverjpena, ok11:07
rdogerritMerged rdo-infra/ci-config master: Promote puppet-passed-ci only in the public server  https://review.rdoproject.org/r/2486711:08
rdogerritChandan Kumar proposed rdo-jobs master: [DNM] testing https://review.opendev.org/706741 cell  https://review.rdoproject.org/r/2486811:11
*** morazi has quit IRC11:13
*** leanderthal has joined #rdo11:13
rdogerritrdo-trunk created openstack/murano-dashboard-distgit rpm-master: openstack-murano-ui: failed to build a85d874  https://review.rdoproject.org/r/2487011:14
toskyhi, I have a chicken-and-egg problem with a potential backport for stein11:14
toskyif I try to simply bump cinder-tempest-plugin to 0.3.1, I get an error: https://review.rdoproject.org/r/#/c/24808/11:14
toskythere is a fix to account for the newer version of cinder-tempest-plugin, but of course I can't backport it without having a newer tarball at the same time: https://review.rdoproject.org/r/#/c/24854/11:15
toskyso I'm kind of stuck; should I try to create a patch which works for both the old and the new tarball (which may be hard), bump the version, and then remove the workarounds?11:16
rdogerritEyal proposed openstack/mistral-extra-distgit rpm-master: Add a spec for the mistral-extra library  https://review.rdoproject.org/r/1421611:18
*** dsneddon_ has quit IRC11:18
*** jlibosva has quit IRC11:20
*** jlibosva has joined #rdo11:20
rdogerritEyal proposed openstack/mistral-extra-distgit rpm-master: Add a spec for the mistral-extra library  https://review.rdoproject.org/r/1421611:31
*** jcapitao is now known as jcapitao|lunch11:42
rdogerritEyal proposed openstack/mistral-extra-distgit rpm-master: Add a spec for the mistral-extra library  https://review.rdoproject.org/r/1421611:45
rdogerritChandan Kumar proposed rdo-jobs master: Added standalone tempest api and scenario tests  https://review.rdoproject.org/r/2470111:47
rdogerritSagi Shnaidman created config master: Add rhel8 and centos8 containers jobs for vexxhost  https://review.rdoproject.org/r/2487211:47
sshnaidmjpena, chkumar|rover ^11:47
chkumar|roversshnaidm, nodeset missing11:47
chkumar|roversshnaidm, we need a patch in rdo-jobs for the nodeset11:48
rdogerritSagi Shnaidman proposed config master: Add rhel8 and centos8 containers jobs for vexxhost  https://review.rdoproject.org/r/2487211:48
sshnaidmchkumar|rover, it's in parent job11:48
sshnaidmchkumar|rover, ah, I see what you mean..11:48
jpenachkumar|rover: did all these container jobs fail around the same time? I see 10:46 UTC11:48
*** dsneddon_ has joined #rdo11:49
chkumar|roverjpena, few of the jobs in master and train got impacted at undercloud install11:49
jpenachkumar|rover: we were updating the SSL cert in the registry around that time, which required a brief restart11:49
rdogerritMerged openstack/ironic-ui-distgit train-rdo: openstack-ironic-ui-3.5.4-1  https://review.rdoproject.org/r/2484011:49
chkumar|roverjpena, total 6 3 in master and 3 in train11:50
rdogerritMerged openstack/ironic-ui-distgit stein-rdo: openstack-ironic-ui-3.4.1-1  https://review.rdoproject.org/r/2483911:50
pandajpena: it works! \o/ well, at least the server work in the way intended, but I'm still getting an avalanche of errors. but THANKS.11:51
chkumar|roverjpena, thanks :-)11:51
pandachkumar|rover: he received too many thanks today, they don't even affect him anymore.11:53
*** dsneddon_ has quit IRC11:54
chkumar|roverpanda, :-)11:56
jpena:D11:56
rdogerritSagi Shnaidman created rdo-jobs master: Add rhel8 vexxhost nodesets  https://review.rdoproject.org/r/2487311:57
rdogerritSagi Shnaidman proposed config master: Add rhel8 and centos8 containers jobs for vexxhost  https://review.rdoproject.org/r/2487211:58
sshnaidmchkumar|rover, seems ok https://review.rdoproject.org/r/#/c/24872/12:03
pandajpena: when I promote a commit_distro hash, how does the server know the mapping commit -> component ? I don't see it in the database, and the promotion is trying to find a None dir in the components/ dir.12:04
jpenapanda: the database should have the component associated with the commit (https://github.com/softwarefactory-project/DLRN/blob/master/dlrn/db.py#L56)12:05
jpenait could be None (if we're not using components), but if we are, we should specify it12:05
pandajpena: oh, I don't see them in any of the commits_2.yaml, but if it's handled this is wonderful news, less work to replicate in the staging environment12:08
*** rpittau is now known as rpittau|bbl12:08
pandajpena: I needed to implement a complex mapping ...12:08
rdogerritMerged rdo-jobs master: Add rhel8 vexxhost nodesets  https://review.rdoproject.org/r/2487312:09
d0ugalIs there a problem with zuul? I am getting CORS errors12:13
sshnaidmd0ugal, 3party or upstream jobs?12:15
rdogerritMerged config master: Add rhel8 and centos8 containers jobs for vexxhost  https://review.rdoproject.org/r/2487212:15
chkumar|roversshnaidm, let me run in check for vexxhost container job and see12:16
d0ugalsshnaidm: like this one; https://review.rdoproject.org/zuul/build/363481005dc042b0b78ef92df13acb6212:18
sshnaidmchkumar|rover, are you preparing a patch? Please include "force_periodic: true" then for each one12:18
chkumar|roveryes12:19
d0ugalsshnaidm: from the browser console... Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://logserver.rdoproject.org/66/24766/1/check/DLRN-rpmbuild/3634810/job-output.json. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).12:19
chkumar|roversshnaidm, I think this one needs support for vexxhost https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-ci-periodic-base/containers-push-registry-login.yaml#L20 ?12:21
sshnaidmd0ugal, I guess there were changes to logserver recently, maybe jpena knows12:21
chkumar|roversorry, it comes from env var12:21
d0ugalsshnaidm: thanks, currently navigating logs.rdoproject manually for now :)12:22
sshnaidmchkumar|rover, I rewrite rdo_registry_url in job definition12:22
rdogerritChandan Kumar created rdo-jobs master: [DNM] vexxhost rhel8 container build  https://review.rdoproject.org/r/2487412:22
chkumar|roveryes got that12:22
chkumar|roversshnaidm, ^612:22
sshnaidmchkumar|rover, add c8 job too12:23
rdogerritUser marios proposed rdo-infra/ci-config master: Add ppc_manifests check and switch to manifest_push to exclude ppc  https://review.rdoproject.org/r/2480712:23
chkumar|roversshnaidm, ok adding to that12:23
sshnaidmchkumar|rover, better to run them on smoke-test-vexxhost project12:23
sshnaidmchkumar|rover, it's where we experiment :)12:23
*** ykarel is now known as ykarel|afk12:24
*** dsneddon_ has joined #rdo12:24
rdogerritChandan Kumar proposed rdo-jobs master: [DNM] vexxhost rhel/centos8 container build  https://review.rdoproject.org/r/2487412:25
*** udesale has joined #rdo12:28
*** raildo has joined #rdo12:28
*** dsneddon_ has quit IRC12:29
rdogerritrdo-trunk created openstack/murano-dashboard-distgit rpm-master: openstack-murano-ui: failed to build a85d874  https://review.rdoproject.org/r/2487512:31
*** jpena is now known as jpena|lunch12:35
rdogerritUser marios proposed rdo-infra/ci-config master: Add ppc_manifests check and switch to manifest_push to exclude ppc  https://review.rdoproject.org/r/2480712:39
*** artom has joined #rdo12:45
*** mgariepy has joined #rdo12:45
rdogerritChandan Kumar proposed rdo-jobs master: [DNM] stein scenario 2 and 4 tempest failure  https://review.rdoproject.org/r/2486912:47
*** morazi has joined #rdo12:48
*** jpich has quit IRC12:49
*** jpich has joined #rdo12:50
*** rlandy has joined #rdo12:50
*** chem has quit IRC12:55
*** ccamel has joined #rdo12:56
*** camelCaser has quit IRC12:56
*** chem has joined #rdo12:57
*** marios is now known as marios|call13:00
*** dsneddon_ has joined #rdo13:03
sshnaidmchkumar|rover, did you create a patch?13:04
*** dsneddon_ has quit IRC13:08
*** amoralej is now known as amoralej|lunch13:10
*** rpittau|bbl is now known as rpittau13:12
d0ugalAnyone got time to review this change? https://review.rdoproject.org/r/#/c/14216/13:14
d0ugalThe depends-on has merged (but I don't think it was working anyway?)13:15
*** holser has joined #rdo13:15
chkumar|roversshnaidm, about c8 build?13:16
chkumar|roversshnaidm, https://review.rdoproject.org/r/#/c/24874/13:17
sshnaidmchkumar|rover, yeah, I see13:17
*** rh-jelabarre has joined #rdo13:17
chkumar|roverit is running here13:17
chkumar|roversshnaidm, I think c8 has some issue, it is going in retry mode13:17
chkumar|roversshnaidm, https://logserver.rdoproject.org/74/24874/2/check/periodic-tripleo-rhel-8-master-containers-build-push-vexxhost/4a85f0d/job-output.txt13:18
chkumar|roversshnaidm, Error: Error copying image to the remote destination: Error writing blob: Error initiating layer upload to /v2/tripleomaster/rhel-binary-nova-compute-ironic/blobs/uploads/ in trunk.registry-vexxhost.rdoproject.org: received unexpected HTTP status: 500 Internal Server Error13:18
sshnaidmchkumar|rover, check out #rhos-ops13:19
chkumar|roversshnaidm, yes13:19
chkumar|roverthanks!13:19
*** jcapitao|lunch is now known as jcapitao13:19
*** artom has quit IRC13:20
*** aedc_ has joined #rdo13:20
*** skramaja has quit IRC13:22
*** aedc has quit IRC13:23
*** jaosorior has joined #rdo13:25
*** ykarel|afk is now known as ykarel13:26
*** aedc_ has quit IRC13:30
*** jpena|lunch is now known as jpena13:30
rdogerritChandan Kumar proposed rdo-jobs master: [DNM] testing https://review.opendev.org/706741 cell  https://review.rdoproject.org/r/2486813:32
*** aedc_ has joined #rdo13:33
*** rfolco has joined #rdo13:42
rdogerritJoel Capitao created rdoinfo master: Pin murano-ui to last working py2 commit  https://review.rdoproject.org/r/2487613:43
*** dsneddon_ has joined #rdo13:43
*** mcornea has joined #rdo13:46
*** dsneddon_ has quit IRC13:48
*** mgariepy has quit IRC13:51
*** mgariepy has joined #rdo13:52
pandajpena: while promoting components, I'm getting a "message": "commit_hash+distro_hash combination not found", there is no commit/distro in the body, but if I gete from the workflow the right commit/distro, they are in the DB13:55
jpenapanda: are you using the short of the long version of the distro hash?13:55
pandajpena: I take the pairs to promote from the db ... and I use only long versions13:56
jpenahm, that's weird13:56
pandajpena: is there a way to get what commit/kdistro he doesn't like ?13:56
jpenapanda: yes, you could add some debugging info in https://github.com/softwarefactory-project/DLRN/blob/master/dlrn/api/dlrn_api.py#L316-L32513:57
pandajpena: ok, open heart surgery it is then :)13:58
rdogerritJoel Capitao proposed rdoinfo master: Pin murano-ui to last working py2 commit  https://review.rdoproject.org/r/2487613:58
pandajpena: yep that's the one14:01
*** ekultails has joined #rdo14:01
pandajpena: "message": "commit_hash: 360d335e94246d7095672c5aa92b59afa380a061, distro_hash: 9e5988125e88f803ba20743be7aa99079dd275f4 combination not found"14:01
*** marios|call is now known as marios14:02
pandajpena: sqlite> select commit_hash, distro_hash from commits;14:02
pandajpena: 360d335e94246d7095672c5aa92b59afa380a061|9e5988125e88f803ba20743be7aa99079dd275f414:02
jpenapanda: is the status == 'SUCCESS? ?14:02
jpenaif it is, I can only think of a mismatch between the DB file that the API is trying to access, and the location where you're running sqlite314:03
jpenathat file is defined in projects.ini14:03
pandajpena: it's SUCCESS for all the commits, I'm not simulating a build failure14:03
*** jlibosva has quit IRC14:03
pandajpena: mmmhh14:03
pandajpena: database_connection="sqlite:///commits.sqlite" does it mean /commits.sqlite or $PWD/commits.sqlite ?14:04
jpenapanda: $PWD/commits.sqlite14:04
pandajpena: after starting the server, the only commits.sqlite that exists in my file system is the one under /tmp/promoter-staging/ in which I run the server on.14:08
pandafind / -name commits.sqlite 2> /dev/null14:08
panda/tmp/promoter-staging/dlrn/commits.sqlite14:08
pandajpena: the single pipeline test still passes :/14:09
jpenapanda: hm, that code path is quite straightforward. It just queries the database for commits, and filters for the commit/distro_hash14:10
pandajpena: maybe you can spot something in the db data14:12
pandajpena: https://review.rdoproject.org/paste/show/215/14:12
*** amoralej|lunch is now known as amoralej14:12
pandathe others commit/distro pairs work, I caqn see promotions on them, but that particular one fails.14:13
jpenaahhh14:14
jpenait's missing the "status: SUCCESS" part14:14
*** lbragstad has quit IRC14:14
pandajpena: SONOVA14:15
pandajpena: I checked twice!14:15
pandajpena: thanks!14:15
jpenano prob :)14:15
pandaaaaaah I cut it from the latest cut-paste14:16
*** dsneddon_ has joined #rdo14:16
*** dsneddon_ has quit IRC14:21
*** lbragstad has joined #rdo14:23
*** dmacpher has joined #rdo14:24
*** aedc_ has quit IRC14:24
mjturekjpena: baha rfolco and I are stumped. We push a container to dockerhub, it pulls down fine. We push it to rdoproject registry, the first blob always fails to pull. Is there anyone that knows enough about the registry to help debug this?14:25
jpenamjturek: dmsimard should also know about the registry, let's see what we can do. What's the container?14:27
mjturekany container we build seems to do it, we've been using centos-binary-base14:28
jpenawhich tag?14:28
mjtureklet me grab an example14:29
mjturekfor example14:30
mjturektrunk.registry.rdoproject.org/tripleomaster/centos-binary-base:67f299a819cfdde7898ce7fd0d25fba25c8a5853_10e135ca_ppc64le14:30
mjturekhttps://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/2004/logs/logs/hash_info.sh14:31
mjturekthat's the most recent centos-binary-base container we built and pushed along with it's hash info14:31
*** Goneri has joined #rdo14:35
*** nweinber has joined #rdo14:35
jpenamjturek: I'm getting this when trying to fetch -> http://paste.openstack.org/show/789373/14:36
jpenait's complaining about the arch mismatch, is that the same for you?14:36
mjturekjpena: nope, we try from ppc64le and "Unexpected EOF" sorry let me get you a paste14:37
*** joelinux__ has joined #rdo14:39
apevecSo today is the last day to comment on Fedora/CentOS "gitforge" proposal, I almost missed that!14:43
apevechas anyone commented?14:43
*** joelinux_ has quit IRC14:43
apevecTBH not sure if they've a hackmd now or is it all on the list?14:43
mjturekjpena sorry had to find the right node, trying to pull now14:44
apevecIIRC openshift registry doesn't have manifests for multiarch?14:47
apevecor am I mixing things up14:47
mjturekjpena: http://paste.openstack.org/show/789374/14:48
mjturekapevec: that is correct14:48
*** d0ugal has quit IRC14:48
mjturekbut we push to a different tag14:48
*** d0ugal has joined #rdo14:48
*** xek has joined #rdo14:50
apevectosky, we can discuss https://review.rdoproject.org/r/24808 here in the open, I didn't quite follow what's wrong there, please add a review comment14:51
apevecI've CCed few of us to have a look14:51
*** d0ugal has quit IRC14:51
toskyapevec: oh, sure; I asked about it here and also on the public list14:51
toskyso all open :)14:51
ykarelme looking14:52
ykarelso looks like tar name changed for some recent version14:52
toskycorrect14:52
toskyand it has been fixed in train and later14:53
toskybut I can't backport the fix as it is without bumping the tag used at the same time -> problem14:53
*** d0ugal has joined #rdo14:53
*** dsneddon_ has joined #rdo14:54
*** jlibosva has joined #rdo14:54
ykareltosky, okk got it after seeing rdoinfo patch14:56
ykarelboth needs each other14:56
toskyyes14:58
*** dsneddon_ has quit IRC14:59
mjturekjpena: any questions??15:00
jpenamjturek: no, just trying to figure out what's going on15:01
ykareltosky, so package build is green in rdoinfo review, we need to update spec such that it have different source for dlrn and cbs15:01
mjturekcool let me know what you see15:01
ykareltosky, like https://github.com/rdo-packages/keystone-tempest-plugin-distgit/blob/7d45d462bec2176b76604315099fd2a74573a1e2/python-keystone-tests-tempest.spec#L12-L1315:02
ykareland usage of tarsources in spec15:02
ykarelwith this cbs build job should pass and dlrn one should fail in -distgit patch15:02
ykareland pass in rdoinfo patch15:02
toskythat's the barba-trick, I guess15:03
*** TrevorV has joined #rdo15:04
*** eharney has joined #rdo15:04
toskyand then remove it after updating rdoinfo15:04
ykareltosky after rdoinfo patch merges, distgit patch will be created to bump release in distgit15:05
ykareland that will fail, so we need to update the review15:05
*** dsneddon_ has joined #rdo15:05
*** ykarel is now known as ykarel|afk15:09
jpenamjturek: that's strange. I'm checking the logs for the blob id that failed, and I only see HEAD calls, but not GET calls15:12
jpenamaybe there's some caching layer?15:12
*** mgariepy has quit IRC15:14
pandajpena: DLRN/components server works with the staging environemnt!!!! \o/15:14
* panda burst into dance15:14
mjturekgo panda!15:15
mjturekjpena: I don't know anything about what registry logs look like unfortunatekly :(15:15
mjturekjpena: so basically a HEAD call gets the info about the blob for the subsequent GET?15:16
jpenamjturek: I'd say so, according to https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/HEAD15:17
jpenamjturek: http://paste.openstack.org/show/789377/ is an example of the logs15:17
mjturekjpena oh :)15:17
*** mgariepy has joined #rdo15:18
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: [WIP] Adapt staging environment for the component pipeline  https://review.rdoproject.org/r/2481815:18
mjturekjpena: interesting15:22
jpenamjturek: looking directly at the registry container logs, I see some GET calls. Let me look a bit more15:22
mjturekokay thank you jpena15:22
mjturekmuch appreciated15:22
jpenamjturek: ok, I see all GET calls were supposed to finish correctly, I get "response completed" everywhere15:27
*** numans_ has quit IRC15:28
rdogerritLuigi Toscano created openstack/cinder-tempest-plugin-distgit stein-rdo: Conditionally chose the name of the generated tarball  https://review.rdoproject.org/r/2487715:30
*** jmlowe has joined #rdo15:33
*** morazi has quit IRC15:34
*** morazi has joined #rdo15:35
ykarel|afktosky, ^^ needs to revert the order in L16,1715:36
*** Eleventh_Doctor is now known as Pharaoh_Atem15:37
ykarel|afkif dlrn, then plugin, if !dlrn then module15:37
*** jcapitao is now known as jcapitao|afk15:42
*** ykarel|afk is now known as ykarel|away15:42
rdogerritLuigi Toscano proposed openstack/cinder-tempest-plugin-distgit stein-rdo: Conditionally chose the name of the generated tarball  https://review.rdoproject.org/r/2487715:44
rdogerritMerged rdo-infra/ci-config master: Modularize Promoter: create initial transition  https://review.rdoproject.org/r/2466715:45
mjturekjpena the ID of the problematic blob is always 23bd9eb8fdc015:46
mjturekjpena anything in the logs specifically about that one?15:46
jpenamjturek: I was checking logs for that blob, specifically. Let me send them to you15:48
mjturekjpena thanks15:48
*** jcapitao|afk is now known as jcapitao15:52
* mjturek looks15:55
*** nweinber has quit IRC15:58
mjturekjpena these are all pulls correct?15:58
jpenamjturek: right. If you see a POST or PUT, that will be an upload15:58
mjturekgotcha15:59
mjturekso I only pulled centos-base15:59
mjturekcould you watch live while I pull centos-base?15:59
mjtureksee if that makes a difference?15:59
jpenamjturek: I'm getting in a call, let's try in ~30 mins16:01
mjtureknp! just ping me jpena and we'll try16:01
*** Trevor_V has joined #rdo16:02
mjturekpanda just to confirm, still having trouble pulling ppc64le containers?16:05
*** TrevorV has quit IRC16:06
rdogerritMerged puppet/puppet-tripleo-distgit train-rdo: Drop dependencies to removed services  https://review.rdoproject.org/r/2486616:10
pandamjturek: as far as I know, yes, but you have to confirm with the people on the frontline, ruck and rover (either weshay or chkumar|rover today)16:14
*** chkumar|rover is now known as raukadah16:15
*** nweinber has joined #rdo16:17
*** udesale has quit IRC16:24
*** rh-jelabarre has quit IRC16:26
rdogerritUser marios proposed rdo-infra/ci-config master: dd ppc_manifests check and switch to manifest_push to exclude ppc  https://review.rdoproject.org/r/2480716:30
*** rh-jelabarre has joined #rdo16:32
*** bandini has quit IRC16:36
*** jmlowe has quit IRC16:39
mjturekpanda: okay as far as we can tell this seems to be an rdoregistry problem? We have trouble pulling from rdoregistry but not from dockerhub (test was pushing a sample container to both and doing a pull from both)16:39
*** lmiccini has quit IRC16:41
*** jdandrea has joined #rdo16:42
*** marios is now known as marios|out16:42
jpenamjturek: I'm ready for the test16:43
mjturekawesome, I'll try to pull centos-binary-base in 30 seconds16:45
mjturekjpena pulling no16:46
mjtureknow*16:46
rdogerritRafael Folco proposed rdo-infra/ci-config master: convert get-hash bash task into a reusable role  https://review.rdoproject.org/r/2476216:48
*** tosky has quit IRC16:48
mjturekjpena: and failed, let me know what you see16:48
jpenamjturek: I see all blobs getting pulled16:48
jpenathe last one is 23bd9eb8fdc010dbd36575046a8c42317f78a9926da949829722cfc815d46cf916:49
jpenahm16:49
jpenainteresting16:49
jpenahttp.response.duration=3.94202ms http.response.status=200 http.response.written=016:49
mjturekhttp://paste.openstack.org/show/789379/16:49
mjtureknot sure how these IDs line up jpena ^16:50
mjturekahhh I see now16:50
jpenaso the last blob is the one that failed, however this "response.written=0" is intriguing16:50
mjturekyeah and that's the problematic blob (the last one)16:50
mjturekjpena what do you think it indicates?16:51
*** marios|out has quit IRC16:51
jpenait's like it didn't send any data, let me confirm that16:51
mjturekcool cool16:53
jpenaa-ha!16:57
jpenamjturek: I think I got it16:57
jpenaI've gone straight to the registry storage16:57
jpenaand tried to find the blob causing issues16:57
*** aufi has quit IRC16:58
jpenait does not seem to be there in the FS. I see other blobs just fine (for example 05d3c81701ccd009c7a8992b5213d966485ff1bc4509d0fb0628ca7c7046bd6b, and its size matches the http.response.written value from the log)16:58
mjturekgeeze16:59
mjturekjpena would watching a live push help at all? I doubt it'17:00
*** avivgt has quit IRC17:00
jpenamjturek: if you could re-push the image, I could see if the blob gets uploaded17:00
mjturekI don't have that exact hash but I will repush that image17:01
mjturekworking on it no17:01
mjtureknow*17:01
mjturekpushing this tag now jpena f47571d32d7ced8d5a04c68f446eff0d01fbf7d4_b5a8f808_ppc64le_mjturek_test17:02
*** chasmo77 has joined #rdo17:04
mjturekhttp://paste.openstack.org/show/789381/ jpena17:04
*** rpittau is now known as rpittau|afk17:07
pandajpena: lol, I'm promoting too fast, I have to sleep for 1 second after every promotion to not have two hashes with the same timestamp17:07
*** nweinber has quit IRC17:08
jpenamjturek: if you now try to pull that same tag, does it fail on the same blob?17:09
*** avivgt has joined #rdo17:10
*** tmazur has quit IRC17:13
mjturektrying now but taking awhile17:15
rdogerritRafael Folco proposed config master: Switch to the new get_hash role  https://review.rdoproject.org/r/2484717:15
*** avivgt has quit IRC17:15
mjturekjpena: odd looks like this http://paste.openstack.org/show/789382/17:23
mjturekso doesn't SEEM to be that blob17:24
mjturekjpena brb lunch17:29
rdogerritMerged rdo-infra/rdo-infra-playbooks master: Added rcn-share role for image server  https://review.rdoproject.org/r/2480517:29
*** nweinber has joined #rdo17:33
*** dtantsur is now known as dtantsur|afk17:34
*** jpich has quit IRC17:35
*** sshnaidm is now known as sshnaidm|afk17:35
*** tesseract has quit IRC17:52
*** gfidente is now known as gfidente|afk17:55
*** openstackstatus has quit IRC17:57
*** openstack has joined #rdo17:59
*** ChanServ sets mode: +o openstack17:59
*** derekh has quit IRC18:00
*** jdandrea has quit IRC18:07
jpenamjturek: I have to leave now. I haven't been able to find anything relevant in the logs :-/- Let's continue tomorrow (or today if dmsimard can dedicate some time to it)18:09
*** jpena is now known as jpena|off18:09
*** jcapitao has quit IRC18:11
*** jmlowe has joined #rdo18:12
*** jaosorior has quit IRC18:16
*** nweinber has quit IRC18:16
*** alexmcleod has quit IRC18:27
*** migarcia is now known as migarcia|away18:29
*** rascasoft has quit IRC18:36
*** ramishra has quit IRC18:37
ekultailsHey folks, for the RDO repositories, is "current-tripleo" simply symlinked to "current-passed-ci"? Or are there situations where they would be different? https://trunk.rdoproject.org/centos7-train/18:44
ekultailsI noticed that on here, at the bottom, it doesn't talk about the current-tripleo repo: https://www.rdoproject.org/what/trunk-repos/18:45
rdogerritRafael Folco created config master: DNM: Test molecule job w/ noop change  https://review.rdoproject.org/r/2487818:45
mjtureksounds fair thanks jpena|off18:54
*** morazi has quit IRC18:59
*** jmlowe has quit IRC19:00
*** jdandrea has joined #rdo19:01
*** jmlowe has joined #rdo19:03
*** amoralej is now known as amoralej|off19:04
*** amoralej|off is now known as amoralej19:04
amoralejekultails, current-passed-ci is symlink to current-tripleo-rdo which is last current-tripleo validated with other non-tripleo jobs19:05
amoralejso, it may be different in some cases19:06
amoralejthis is the pipleline used to promote from current-tripleo to current-tripleo-rdo (==current-passed-ci)19:06
amoralejhttps://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-train-current-tripleo/19:06
amoralejekultails, ^19:06
*** amoralej is now known as amoralej|off19:07
*** eharney has quit IRC19:07
*** chem has quit IRC19:07
*** chem has joined #rdo19:08
ekultailsamoralej: Great, thanks for the explanation!19:09
*** eharney has joined #rdo19:09
*** ralonsoh has quit IRC19:10
*** jmlowe has quit IRC19:11
mjturekdmsimard: if you do have some time to check out what jpena|off and I were discussing, let me know! Obviously doesn't need to be today19:15
*** artom has joined #rdo19:17
*** raildo_ has joined #rdo19:17
*** raildo has quit IRC19:18
*** jmlowe has joined #rdo19:20
rdogerritwes hayutin created rdo-infra/ci-config master: add heat stats, per request of DF  https://review.rdoproject.org/r/2487919:21
ekultailsI put up a review up to add current-tripleo-rdo as a choice in tripleo-repos: https://review.opendev.org/#/c/706962/19:25
*** igordc has joined #rdo19:35
*** jmlowe has quit IRC19:38
*** stevebaker has joined #rdo19:39
*** jmlowe has joined #rdo19:40
rdogerritSorin Sbarnea (zbr) created config master: WIP: Bump molecule to 3.0a5  https://review.rdoproject.org/r/2488019:41
*** numans has joined #rdo19:48
*** phillw is now known as phillw-werner19:50
*** avivgt has quit IRC19:50
*** phillw-werner is now known as phillw19:51
*** jdandrea has quit IRC19:51
*** phillw is now known as phillw-leonel19:51
*** phillw-leonel is now known as phillw19:51
*** jdandrea has joined #rdo19:53
*** igordc has quit IRC19:53
*** phillw is now known as Lodewyk19:57
*** Lodewyk is now known as phillw19:58
rdogerritTristan de Cacqueray created rdo-infra/rdo-infra-playbooks master: Add missing rcn-share group creation  https://review.rdoproject.org/r/2488119:58
*** phillw is now known as blondjaguar20:01
*** blondjaguar is now known as phillw20:01
*** holser has quit IRC20:01
rdogerritSorin Sbarnea (zbr) proposed config master: WIP: Bump molecule to 3.0a5  https://review.rdoproject.org/r/2488020:03
*** morazi has joined #rdo20:18
*** eharney has quit IRC20:22
*** jmlowe has quit IRC20:22
*** jfrancoa has quit IRC20:26
*** mcornea has quit IRC20:29
*** mcornea has joined #rdo20:29
*** nweinber has joined #rdo20:33
*** jlibosva has quit IRC20:34
*** jmlowe has joined #rdo20:36
rdogerritMerged rdo-infra/rdo-infra-playbooks master: Add missing rcn-share group creation  https://review.rdoproject.org/r/2488120:41
*** jfrancoa has joined #rdo20:42
*** paramite has quit IRC20:47
*** maciejjozefczyk has quit IRC20:53
*** jmlowe has quit IRC20:54
rdogerritSteve Baker created rdo-infra/ansible-role-tripleo-ci-reproducer master: Don't install docker on centos-8  https://review.rdoproject.org/r/2488221:03
*** jtomasek_ has quit IRC21:15
*** rfolco has quit IRC21:23
*** holser has joined #rdo21:33
*** xek has quit IRC21:36
*** rfolco has joined #rdo21:37
*** Goneri has quit IRC21:46
*** nweinber has quit IRC22:07
*** phillw is now known as gavman22:11
*** dpawlik has quit IRC22:14
*** dmsimard is now known as dmsimard|off22:17
*** jfrancoa has quit IRC22:20
*** mcornea has quit IRC22:26
*** slaweq has quit IRC22:29
*** gavman is now known as phillw22:33
*** openstackgerrit has quit IRC22:46
*** nweinber has joined #rdo22:51
*** Trevor_V has quit IRC22:56
*** holser has quit IRC22:57
*** nweinber has quit IRC22:59
*** eharney has joined #rdo23:02
rdogerritrdo-trunk created openstack/nova-distgit train-rdo: openstack-nova-20.1.1-1  https://review.rdoproject.org/r/2488323:05
*** raildo_ has quit IRC23:07
*** artom has quit IRC23:09
*** shadower has quit IRC23:11
*** jdandrea has quit IRC23:13
*** apevec has quit IRC23:14
*** holser has joined #rdo23:26
*** rh-jelabarre has quit IRC23:37
*** holser__ has joined #rdo23:45
*** holser has quit IRC23:49
*** holser__ has quit IRC23:55
*** holser has joined #rdo23:58
*** slaweq has joined #rdo23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!