Tuesday, 2019-04-23

« Monday, 2019-04-22 Index Wednesday, 2019-04-24 »
*** sanjayu_ has quit IRC00:33
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ https://review.opendev.org/602248, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.opendev.org/56429100:42
*** rlandy|ruck has quit IRC01:46
aakarshvinaykns, weshay|rover I sort of put together an automation to run perf tests on baremetal in the downstream qe ci as well through browbeat.  using01:54
aakarshlmk if you guys are looking to do that01:54
aakarshthe work needs some polishing but it's in a state where it can run with some minor tweaks01:55
*** dsneddon has quit IRC01:55
*** dsneddon has joined #oooq01:57
*** ykarel|away has joined #oooq02:26
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ https://review.opendev.org/602248, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.opendev.org/56429102:42
*** vinaykns has quit IRC02:52
*** ykarel|away is now known as ykarel03:06
*** udesale has joined #oooq04:22
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ https://review.opendev.org/602248, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.opendev.org/56429104:42
*** ratailor has joined #oooq04:56
*** ykarel is now known as ykarel|afk05:01
*** ykarel|afk has quit IRC05:06
*** raukadah is now known as chandankumar05:11
*** jaosorior has joined #oooq05:17
*** ykarel|afk has joined #oooq05:23
*** ykarel|afk is now known as ykarel05:24
*** marios has joined #oooq05:28
*** ykarel_ has joined #oooq05:28
*** ykarel has quit IRC05:31
*** ccamacho has quit IRC05:46
*** quiquell|off is now known as quiquell|rover05:49
*** matbu has joined #oooq06:11
*** matbu has quit IRC06:18
*** matbu has joined #oooq06:20
*** ykarel_ is now known as ykarel06:31
*** udesale has quit IRC06:35
*** udesale has joined #oooq06:37
*** quiquell|rover is now known as quique|rover|brb06:42
hubbot1FAILING CHECK JOBS on stable/pike: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 @ https://review.opendev.org/602248, stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.opendev.org/56429106:42
*** udesale has quit IRC06:44
*** udesale has joined #oooq06:44
*** jfrancoa has joined #oooq06:54
*** saneax has joined #oooq07:04
*** quique|rover|brb is now known as quiquell|rover07:12
*** arxcruz|off|23 is now known as arxcruz07:13
*** ccamacho has joined #oooq07:13
*** tosky has joined #oooq07:17
*** udesale has quit IRC07:18
*** udesale has joined #oooq07:19
*** ccamacho has quit IRC07:27
*** ccamacho has joined #oooq07:27
*** fmount has quit IRC07:32
*** amoralej|off is now known as amoralej07:34
*** fmount has joined #oooq07:34
*** bogdando has joined #oooq07:41
*** bogdando has quit IRC07:41
*** dsneddon has quit IRC07:42
*** jpena|off has joined #oooq07:43
*** jpena|off is now known as jpena07:43
*** ykarel is now known as ykarel|lunch07:45
*** dsneddon has joined #oooq07:50
*** dtantsur|afk is now known as dtantsur07:55
*** dsneddon has quit IRC07:55
*** dsneddon has joined #oooq08:08
*** dsneddon has quit IRC08:14
*** derekh has joined #oooq08:28
quiquell|roverpanda: did you know where do we have the gerrit config for this ? https://ci.centos.org/job/tripleo-quickstart-extras-gate-master-delorean-full-featureset052/3246/console08:29
quiquell|roverykarel|lunch: ^ Â?08:29
*** apetrich has joined #oooq08:30
*** dsneddon has joined #oooq08:30
quiquell|roverok found it08:30
mariosquiquell|rover: any ideas ? you mentioned something about the images yesterday but which images were you referring to? ( o/ (OSP15/RHEL8 https://tree.taiga.io/project/tripleo-ci-board/us/952 ) any idea why at https://code.engineering.redhat.com/gerrit/#/c/167540/18 i'm still hitting08:30
marioshttps://sf.hosted.upshift.rdu2.redhat.com/logs/40/167540/18/check/tripleo-ci-rhel-8-standalone-rhos-15/f1a73d5/job-output.txt.gz#_2019-04-23_07_59_59_567573 even though we merged that yesterday08:30
marioshttps://code.engineering.redhat.com/gerrit/#/c/168410/1/nodepool/elements/virt-customize/rhel8-cloud.yaml08:31
quiquell|rovermarios: first we have to be sure that the RPM is installed08:34
mariosquiquell|rover: which rpm libselinux-python? it isn't thats the error08:34
quiquell|rovermarios: not always the case08:34
quiquell|rovermarios: it could be inside a virtualenv08:34
mariosquiquell|rover: i thought adding it in https://code.engineering.redhat.com/gerrit/#/c/168410/1/nodepool/elements/virt-customize/rhel8-cloud.yaml would install it08:34
quiquell|rovermarios: without site packages flag08:34
mariosquiquell|rover: they;re doing virt-customize on the guest image to inject things like those packages08:35
*** dsneddon has quit IRC08:35
mariosquiquell|rover: it's all in that file https://code.engineering.redhat.com/gerrit/#/c/168410/1/nodepool/elements/virt-customize/rhel8-cloud.yaml08:35
mariosquiquell|rover: i mean thats the entire "element"08:35
mariosquiquell|rover: i think apevec was somehow involved in that element i'll ask him to check too08:36
mariosthanks man i will try to dig more i asked you cos you mentioned something about the image yesterday i guess you meant the rpm08:36
quiquell|rovermarios: you can add collect logs to that job08:37
quiquell|rovermarios: so we have more info thogh08:37
mariosquiquell|rover: ack08:37
quiquell|rovermarios: use chandankumar new role08:37
quiquell|rovermarios: it can also bey something relaated to python308:37
quiquell|rovermarios: or virtualenvs08:37
mariosquiquell|rover: sure but this is happening like 20seconds into the job08:37
mariosquiquell|rover: so not much to collect08:37
quiquell|rovermarios: do you hava rhel8 node around ?08:37
quiquell|rovermarios: sure there is rpm list and all and yum repos config08:38
mariosquiquell|rover: ack ok08:38
quiquell|rovermarios: we can poke at a RHEL8 node if you have one08:38
mariosquiquell|rover: no i dont and plan to do that but it is diferent. i mean this is specific problem of that 'fake' cloud-rhel-8 element we're trying to use the libselinux problem i mean08:39
mariosquiquell|rover: really we want to be using this https://code.engineering.redhat.com/gerrit/167633 & depends-on08:40
mariosquiquell|rover: which  already has this selinux stuff installed08:40
quiquell|rovermarios: We can try to mount that qcow2 and test on it08:42
marios    image_url: http://download-node-02.eng.bos.redhat.com/rhel-8/nightly/RHEL-8/latest-RHEL-8/compose/BaseOS/x86_64/images/rhel-guest-image-8.0-1801.x86_64.qcow208:42
quiquell|rovermarios: you can ask at inernal infra for nodepool logs08:42
mariosquiquell|rover: that one you mean?08:42
hubbot1FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.opendev.org/56429108:42
quiquell|rovermarios: maybe there are some errors there and images haven't being rebuild with the extra package08:42
quiquell|rovermarios: yep that one08:43
quiquell|rovermarios: but ask internal infra first about nodepool logs08:43
quiquell|rovernodepool-builder08:43
quiquell|rovermaybe this package does not exists at rhel8 and it fails at building new one08:43
mariosquiquell|rover: ack whats the irc chan for internal infra?08:44
mariosplease thanks08:44
quiquell|rover#rhos-ops08:44
quiquell|roverat internal IRC08:44
mariosquiquell|rover: ty noted08:44
quiquell|rovermarios: let me know of anything also inparallel try to startup with libvirt or something that qcow208:45
quiquell|rovermarios: so we can poke around08:45
quiquell|roverand manually try to install those packages08:45
quiquell|roverand even run the role08:45
quiquell|roverhumm wait08:45
quiquell|roverquickvm08:45
*** ykarel|lunch is now known as ykarel08:52
pandaykarel: where's your patch with the build containres setup ?09:01
ykarelpanda, you mean https://review.opendev.org/#/c/653408/?09:01
*** jbadiapa has joined #oooq09:02
pandaykarel: yep, +W09:04
pandaykarel: thanks.09:04
ykarelpanda, Thanks, so u progressed with rhel container builds?09:04
ykarellast u said there were issues with repos09:04
pandaykarel: yes, to the point in which they fai, because of the hardcoded rpm setup config09:05
pandaykarel: the internal repo is http only, so I had to inject a specific configuration for the docker daemon09:05
ykarelpanda, okk good09:05
pandas/repo/registry09:05
*** dsneddon has joined #oooq09:05
ykarelokk09:05
quiquell|roverpanda, ykarel: to fix ci.centos https://review.rdoproject.org/r/#/c/20418/09:06
ykarelquiquell|rover, ack09:07
*** dsneddon has quit IRC09:10
ykarelquiquell|rover, commented09:14
quiquell|roverykarel: fixed09:19
mariosquiquell|rover: thanks for help. posted https://code.engineering.redhat.com/gerrit/#/c/168504/09:20
ykarelquiquell|rover, +109:20
quiquell|roverykarel: is there any way to fix this ourself https://bugs.centos.org/view.php?id=16026  ?09:23
*** ratailor_ has joined #oooq09:25
ykarelquiquell|rover, you filed ^^ to fix the git.opendev.org clone issue?09:26
quiquell|rovernope09:26
quiquell|roverthis is different09:26
quiquell|roverI think09:26
quiquell|roverwes post stuff at etherpad09:27
quiquell|roverbut maybe it's not needed09:27
*** ratailor has quit IRC09:27
quiquell|rovermarios, ykarel, Tengu: This rings any bell ? http://logs.openstack.org/46/654446/1/check/tripleo-ci-centos-7-scenario009-multinode-oooq-container/eb913e4/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz09:29
quiquell|roverpanda: ^09:29
mariosquiquell|rover: llooking09:29
quiquell|rover'message': u'401 Client Error: Unauthorized for url: https://registry-1.docker.io/v2/tripleostein/etcd/manifests/latest',09:29
mariosquiquell|rover: yah not seen that before ...09:29
quiquell|roverack09:30
quiquell|roverwill open a LP09:30
ykarelquiquell|rover, no such image in stein09:30
ykarelmay be specifc to some patch09:31
*** jaosorior has quit IRC09:31
quiquell|roverhttp://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario009-multinode-oooq-container&branch=stable%2Fstein09:31
quiquell|roverall of them are failing09:31
quiquell|roverdifferent patches09:31
quiquell|roversame error09:32
quiquell|rovermaybe they have Depends-On09:32
quiquell|roverlet me check09:32
ykarelokk then some real issue09:32
Tenguquiquell|rover: 401 - missing auth?09:32
ykarelgood to check what push etcd image in tripleomaster09:32
ykareland why it missing in tripleostein09:32
ykarelhttps://hub.docker.com/r/tripleomaster/etcd/tags09:32
quiquell|roverbtw whene did we make that job voting ?09:33
ykarelvs https://hub.docker.com/r/tripleostein/etcd/tags09:33
quiquell|roverit was non-voting as I see09:33
*** amoralej is now known as amoralej|mtg09:33
TenguI think mwhahaha was the one working on etcd integration09:33
Tengubut not 100% sure09:33
quiquell|roverTengu: ack thanks09:33
quiquell|roverwait... I am suffering a big brain fart here09:34
quiquell|roverna is non-voting09:36
quiquell|roverall good09:36
quiquell|roverno need for LP09:36
quiquell|roverykarel: is still fs035 stein running at testproject ?09:42
ykarelquiquell|rover, need to check09:43
ykarelquiquell|rover, yes running and at overcloud deploy step09:44
ykarelhttps://review.rdoproject.org/zuul/stream/8688961ef1ce42a0b17d2ea1438941f8?logfile=console.log09:44
quiquell|rovercool09:44
quiquell|rovermarios: can you merge this https://review.rdoproject.org/r/#/c/20371/  ?09:44
*** dsneddon has joined #oooq10:06
*** amoralej|mtg is now known as amoralej10:10
*** dsneddon has quit IRC10:12
mariosquiquell|rover: looking10:13
*** jaosorior has joined #oooq10:14
chandankumarquiquell|rover: ykarel is fs027 passing now in periodic?10:18
ykarelyes, considering master and stein both promoting10:19
chandankumarack, cool!10:19
mariosykarel: looks like gitweb works now10:26
mariosykarel: ah not quite10:26
mariosykarel: it works on merged changes10:26
mariosykarel: but not still in review :/10:26
ykarelmarios, yup just saw a change and getting 404: https://opendev.org/openstack/karbor/commit/fc2f1a519f7ec4b54347bda23829e1f9b19675a110:27
mariosykarel: but works here https://review.opendev.org/#/c/652102/10:27
mariosfor example10:27
quiquell|roverwell we don't need gitweb anymore10:27
ykarelmarios, hmm there it goes to git.openstack.org10:27
ykarelwhich redirecting to opendev.org10:28
ykarelchandankumar, fyi redirect is working correctly from git.openstack.org ^^10:28
ykarelmarios, for gitweb, i remember gitea bug was only for refs/changes10:29
ykarelchanges which are in progress10:29
quiquell|roverykarel: do you remember where is the config for nodepool dib upstream ?10:30
quiquell|roverthe dib elements and all10:30
ykarelmarios, same i added to commit message in https://review.opendev.org/#/c/654219/10:30
ykarelquiquell|rover, from my memory i think it was in project-config10:30
ykarelbut need to recheck10:30
chandankumaryou mean git.openstack.org/cgit to opendev.org/cgit?10:31
chandankumarykarel: ^^10:31
ykarelchandankumar, yes in opendev.org no cgit10:31
chandankumarok10:32
ykarelquiquell|rover, i see some elements at project-config:- https://github.com/openstack/project-config/tree/master/nodepool/elements10:32
quiquell|roverhttps://opendev.org/openstack/project-config/src/branch/master/nodepool/nodepool.yaml#L163-L18410:32
quiquell|roverfound it10:32
quiquell|roverykarel: thanks10:32
ykarelokk10:32
kopecmartinarxcruz, hi, when you have some time - https://review.opendev.org/#/c/648121/10:33
*** sshnaidm|afk is now known as sshnaidm10:40
hubbot1FAILING CHECK JOBS on stable/ocata: tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001 @ https://review.opendev.org/56429110:42
ykarelquiquell|rover, fs035 passed and stein promoting:- Promoting the container images for dlrn hash 83526f8a80946ab8f4fd546c60b32491d88fff79 on stein to current-tripleo10:47
quiquell|roverykarel: cool! thanks a lot ykarel10:48
*** amoralej is now known as amoralej|lunch10:57
*** jaosorior has quit IRC11:00
*** jaosorior has joined #oooq11:02
*** dsneddon has joined #oooq11:07
*** ykarel is now known as ykarel|afk11:13
*** dsneddon has quit IRC11:13
arxcruzsshnaidm: panda can you guys please, PELASE review https://review.opendev.org/#/c/631773/ ? I'm already feeling bad about the guy and his patience rechecking everytime... :D11:19
pandaarxcruz: is this used also in os_tempest ?11:27
arxcruzpanda: not yet11:27
pandaarxcruz: ok, I'll come back when this is translated to os_tempest then11:30
mariosbiab11:30
*** marios has quit IRC11:31
* panda stops to hear the screams of a thousand owls suffering together11:31
*** apetrich has quit IRC11:32
*** jpena is now known as jpena|lunch11:33
*** quiquell|rover is now known as quique|rover|lun11:35
*** quique|rover|lun is now known as quique|rover|eat11:36
*** apetrich has joined #oooq11:36
*** dsneddon has joined #oooq11:38
*** dsneddon has quit IRC11:44
*** hubbot1 has quit IRC11:55
*** panda is now known as panda|lunch11:57
*** quique|rover|eat is now known as quiquell|rover11:59
*** dsneddon has joined #oooq12:01
*** rlandy has joined #oooq12:06
*** rlandy is now known as rlandy|ruck12:07
quiquell|roverrlandy|ruck: o/ looks like stein promoted12:08
rlandy|ruckquiquell|rover: hey there12:08
rlandy|ruckquiquell|rover: hope we are in for a better day today :)12:08
quiquell|roverrlandy|ruck: and found why the f28 is not working but does not make sense12:08
quiquell|roverrlandy|ruck: I think it looks like12:08
quiquell|roverrlandy|ruck: do you know why weshay|rover set scenrio009 as voting just at THT ?12:10
weshay|roverquiquell|rover should vote on tht, python-tripleoclient, and tripleo-common12:11
quiquell|roverweshay|rover: look like is broken at stein12:11
rlandy|ruckat least promoter is working again12:11
quiquell|roverweshay|rover: I am looking into it https://bugs.launchpad.net/tripleo/+bug/182598212:11
openstackLaunchpad bug 1825982 in tripleo "Stein tripleo-ci-centos-7-scenario009-multinode-oooq-container Unauthorized for url: https://registry-1.docker.io/v2/tripleostein/etcd/manifests/latest" [Critical,Triaged]12:11
quiquell|roverrlandy|ruck: have being working all day, what happend ?12:11
rlandy|ruckquiquell|rover: you didn't see any issues with container push?12:12
rlandy|ruckok - then we fuxed it12:12
rlandy|ruckfixed12:12
rlandy|ruckit was another opendev link12:12
quiquell|roverrlandy|ruck: ack, well fixed some of them at ci-config today12:12
quiquell|roverrlandy|ruck: reproducer works too now12:12
quiquell|roverweshay|rover: have run f28 at reproducer works and works fine12:13
quiquell|roverweshay|rover: issue looks related to RDO dib for f2812:13
weshay|roverquiquell|rover cool.. did you recreate a periodic?12:13
*** dsneddon has quit IRC12:13
weshay|roverI saw ur bug12:13
quiquell|roverweshay|rover: I don't know why python3-setuptools installed stuff get wiped out12:13
quiquell|roverweshay|rover: upstream is not the case :-(12:14
* weshay|rover looks again because I wasn't 100% sure what you found12:14
quiquell|roverjpena|lunch: has hold the RDO nodes for me12:14
quiquell|roverI am poking around12:14
quiquell|roverweshay|rover: looks like revert of reinstall works https://review.rdoproject.org/r/#/c/20419/12:15
*** marios has joined #oooq12:16
rlandy|ruckquiquell|rover, btw we removed the ci.centos gate jobs12:16
rlandy|ruckonly fs052 left12:16
quiquell|roverrlandy|ruck: we don't need them ?12:17
rlandy|ruckquiquell|rover: they have been failing for months12:17
weshay|roverya.. the system does not have resources to have jobs pass12:17
quiquell|roverack fixed 52 stuff with opendev though12:17
weshay|rovertrying to just have standalone - nonvoting and see how that goes12:17
quiquell|roverack step by step12:18
quiquell|roverso we have to fix scenario009 to merge emilien backports to stein12:18
quiquell|roverhttps://review.opendev.org/#/c/654444/12:19
quiquell|roverTengu: you said that Alex was working on etcd stuff ?12:19
Tenguquiquell|rover: I think so - but I might be wrong on that one.12:20
quiquell|roverwell clearly this is not working skopeo inspect docker://docker.io/tripleostein/etcd12:20
quiquell|roverhumm well12:20
quiquell|roveris same error if container is not there12:20
*** jbadiapa has quit IRC12:22
Tenguquiquell|rover: so a git blame on deployment/etcd/etcd-container-puppet.yaml does show Alex - but he's not alone on that.12:22
Tengumaybe he just updated things in the templates. Can't say more -.-'12:22
quiquell|roverthe container is not there https://hub.docker.com/r/tripleostein/etcd12:23
quiquell|roverlooks like we miss some backports from master12:24
Tenguquiquell|rover: so container wasn't built/pushed, you probably need to ping whoever is responsible for the container delivery.12:24
quiquell|roverWell we are :-)12:24
quiquell|roverbut master have it allright12:24
weshay|roverquiquell|rover hrm.. not looking great is it http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-scenario009-multinode-oooq-container12:24
quiquell|roverweshay|rover: etcd container is missin at stein12:25
weshay|roverquiquell|rover looks like master is good12:25
quiquell|roveryep12:25
weshay|roverstein is the issue12:25
Tenguquiquell|rover: https://hub.docker.com/r/tripleostein/centos-binary-etcd12:25
quiquell|roveryep12:25
Tenguquiquell|rover: name's wrong.12:25
weshay|roverah cool12:25
Tenguthat's all :)12:25
weshay|roverbackport12:25
quiquell|roverahhh12:25
* Tengu loves playing with URI12:25
quiquell|roverhumm master is just etcd12:25
Tengumaster's a fool12:25
quiquell|roverok will fix it12:25
quiquell|roverhehe12:25
quiquell|roverok master have it too skopeo inspect docker://docker.io/tripleomaster/centos-binary-etcd:current-tripleo12:26
Tengucool. so that's an easy fix hopefully :)12:26
weshay|roverquiquell|rover we can expand the gate coverage of scenario009 if Tengu and crew ack... it would be smart considering it's openshift on openstack12:27
weshay|roverbut that's all we initially had ack for12:27
quiquell|roverlet's just fix it to merge the backports12:28
quiquell|roverThen we decide at community meeting or the like12:28
weshay|roverquiquell|rover Tengu anything need to go on https://etherpad.openstack.org/p/tripleo-meeting-items12:30
*** jpena|lunch is now known as jpena12:31
Tenguwhy me? what did I do?12:31
Tengu:)12:31
*** amoralej|lunch is now known as amoralej12:32
mariosam i in the wrong call12:32
marioshttps://bluejeans.com/587845809712:32
mariosrfolco: ?12:32
marioscommunity call?12:32
* marios checks 12:32
mariosoh in 1 hour12:32
marios:/12:32
rlandy|ruckweshay|rover: quiquell|rover: anything we want to raise at tripleo meeting?12:33
quiquell|roverrlandy|ruck: maybe what weshay|rover said about expand gate coverage of scenario00912:34
weshay|roverrlandy|ruck quiquell|rover have we confirmed upstream patches and depends-on are being built into the jobs?12:35
rlandy|rucklet's check12:35
quiquell|roverweshay|rover: at least at RDO is working I have test one12:35
arxcruzmarios: málaka this different timezones!12:40
arxcruzs/this/these12:40
mariosarxcruz: kindly forgive me good sir12:41
marios(malaka)12:41
weshay|roverchandankumar arxcruz kopecmartin FYI.. I am now tracking tempest bugs (lp, bz ) in the cockpit...12:44
weshay|roverhttp://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=112:44
arxcruzweshay|rover: i saw it ;)12:44
weshay|roverlook for "TripleO Bugzilla Tempest Bugs",  Upstream Tempest bugs, Untriaged Tempest Bugzilla12:44
weshay|roverarxcruz chandankumar kopecmartin so the bugziila bug process does not change, however anything that *may* be a tempest issue or require the tempest squads eyes to fix will have a tempest tag in lp12:45
kopecmartinok, sounds good12:46
arxcruzweshay|rover: ack12:46
weshay|roverif anything looks incorrect let me know :)12:46
rlandy|rucklooks up picking up right skip now12:50
rlandy|rucklike12:50
ykarel|afkrlandy|ruck, was there some issue with Depends-On being built? i commented on one of your review merged12:50
ykarel|afki was trying to understand if that was causing some issue12:51
*** ykarel|afk is now known as ykarel12:51
rlandy|ruckykarel: ack - depends=on was not being built with upstream changes12:51
ykarelrlandy|ruck, with opendev.org or openstack.org?12:51
rlandy|ruckafter the opendev.org move12:52
ykarelrlandy|ruck, i mean Depends-On with opendev.org is not working or with openstack.org is not working?12:52
rlandy|ruckhttps://review.opendev.org/#/c/653627/12:52
rlandy|ruckykarel: ^^ was the original report12:52
ykarelrlandy|ruck, than it's openstack.org not working while opendev.org Depends-on working as per change https://review.opendev.org/#/c/653627/3..4//COMMIT_MSG12:53
ykareland as per openstack-discuss mail due to some zuul bug openstack.org depends-on will not work12:54
ykarelrlandy|ruck, from mail snippet:- First, any commit Depends-On footers for unmerged changes will need to use https://review.opendev.org urls instead of https://review.openstack.org urls. This is due to a limitation in Zuul and how it sees the Gerrit server. Any Depends-On footers that  use https://review.openstack.org are currently non-functional and they will not enforce the dependency relationship.12:54
ykarelhttp://lists.openstack.org/pipermail/openstack-discuss/2019-April/005423.html12:55
rlandy|ruckykarel: this looks to be working : ogs.openstack.org/27/653627/4/check/tripleo-ci-centos-7-standalone/92720eb/logs/undercloud/home/zuul/tripleo-heat-installer-templates/environments/standalone/standalone-overcloud.yaml.txt.gz12:55
rlandy|rucknow at least12:55
ykarelrlandy|ruck, yes because now Depends-On is using opendev.org12:56
ykarelin PS412:56
rlandy|ruckas long as it's working12:56
chandankumarweshay|rover: looks nice, thanks!12:57
*** smalleni_ has joined #oooq12:57
weshay|roverrlandy|ruck quiquell|rover fyi.. something is continuing to cause timeouts on ovb deployments12:58
weshay|roverI no longer think it's infra12:58
weshay|roverbut obviously not sure12:58
rlandy|ruckweshay|rover: I'll look into it12:58
rlandy|ruckour failures are coming down12:59
*** mjturek has joined #oooq13:01
*** fultonj has joined #oooq13:03
*** fultonj has quit IRC13:03
sshnaidmrlandy|ruck, rfolco I don't find where nodepool instances are running in internal SF, I don't find them neither on rhos-d not on rhos-ocp clouds13:05
sshnaidmrlandy|ruck, rfolco do you know which cloud they run on?13:05
*** jaosorior has quit IRC13:08
*** mjturek has quit IRC13:12
rlandy|rucksshnaidm: I would guess the same cloud as rhos-d but on a different tenant13:12
sshnaidmrlandy|ruck, well, we need these nodes to be on the same tenant with ovb stacks for running ovb jobs..13:13
sshnaidmrlandy|ruck, it won't work if they're on different tenants13:13
rlandy|rucksshnaidm: unless you separate the undercloud and the nodepool node13:14
rlandy|rucklike we do for baremetal13:14
*** jbadiapa has joined #oooq13:15
sshnaidmrlandy|ruck, yeah, and then need to have undercloud in the stack, and it will require to rewrite current ovb role..13:15
*** vinaykns has joined #oooq13:20
*** panda|lunch is now known as panda13:23
*** jrist- is now known as jrist13:25
*** ratailor_ has quit IRC13:25
*** sshnaidm is now known as sshnaidm|afk13:26
*** quiquell|rover is now known as quique|rover|lun13:26
*** quique|rover|lun is now known as quique|rover|eat13:26
*** rlandy|ruck is now known as rlandy|ruck|mtg13:30
chandankumarweshay|rover: rlandy|ruck perf meeting time13:30
mariosrfolco: is the meeting now community13:32
mariosand it is at https://bluejeans.com/587845809713:32
mariosrfolco: cos thats what th calendar says13:33
rfolcomarios, ah its wes'bj13:33
mariosrfolco: ack joining13:33
*** skramaja has joined #oooq13:33
rfolcoci community mtg at https://bluejeans.com/411356779813:33
*** skramaja has quit IRC13:47
*** quique|rover|eat is now known as quiquell|rover13:47
*** Vorrtex has joined #oooq13:50
*** mjturek has joined #oooq13:50
*** sshnaidm|afk is now known as sshnaidm13:55
*** Goneri has joined #oooq13:57
*** jaosorior has joined #oooq14:00
*** saneax has quit IRC14:03
pandarfolco: can you remove me from the invitation for the two meetings today ?14:03
pandachandankumar: you have the UA meeting now to attend14:03
chandankumarpanda: thanks, /me needs to find the invite14:04
rfolcopanda, done14:06
mariosquiquell|rover: that seem right to you jobs are stuck in queued at https://review.opendev.org/#/c/654952/14:08
mariosquiquell|rover: is the queue very long?14:08
mariosquiquell|rover: ah 5.6 hours apparently grafana14:08
quiquell|rovermarios: queue  is ok14:08
mariosok thanks14:08
quiquell|roverwell gate14:09
mariosquiquell|rover: i mean that was posted 4 hours ago14:09
mariosshould have run by now?14:09
mariosquiquell|rover: wondering if it is stuck i need recheck or just wait14:09
quiquell|roverrecheck is not going to work there14:09
*** dsneddon has joined #oooq14:10
mariosquiquell|rover: ok why you see some problem?14:11
quiquell|roverThis is not same as being queued at gates14:13
quiquell|roverthis look related to not enough powerhorse at upstream zuul14:13
*** dsneddon has quit IRC14:18
chandankumarkopecmartin: arxcruz http://logs.openstack.org/06/654406/9/check/openstack-tox-docs/55bf487/html/install/basic_deployment/os_tempest.html i am not sure what is wrong there in rendering14:28
*** dsneddon has joined #oooq14:29
*** rlandy|ruck|mtg is now known as rlandy|ruck14:30
*** dsneddon has quit IRC14:35
*** dsneddon has joined #oooq14:37
* chandankumar headed home, will back soon14:40
mariosweshay|rover: do we have etherpad for ptg or using that https://etherpad.openstack.org/p/tripleo-ptg-train14:41
mariosquiquell|rover: ^14:41
mariosquiquell|rover: weshay|rover: do we want a tripleo-ptg-train-ci-squad etherpad14:41
mariosfor session notes etc will serve as log14:41
*** dsneddon has quit IRC14:42
mariosquiquell|rover: seems to be general problem 'queued' at zuul status14:44
mariosquiquell|rover: for this one too https://review.opendev.org/65178914:44
*** udesale has quit IRC14:45
*** udesale has joined #oooq14:46
*** ccamacho has quit IRC14:46
rlandy|ruckmarios: what is the issue?14:47
rlandy|ruckmarios: the gate queue or check?14:48
mariosrlandy|ruck: both, in one is gate the other is check14:48
mariosrlandy|ruck: here https://review.opendev.org/654952 - stuck in zuul check ... "queued" also this stuck in gate https://review.opendev.org/65178914:48
mariosrlandy|ruck: ah the 789 one is now running again14:49
marioshttps://review.opendev.org/654952 still stuck14:50
mariosrlandy|ruck: so that one is check ^14:50
rlandy|ruckmarios: gate job looks ok14:50
rlandy|ruckjust that there is traffic14:50
rlandy|rucklooking at check14:51
mariosrlandy|ruck: hm we don't have check queue time at grafana dashboard14:51
mariosrlandy|ruck: where do you check the check queue length?14:51
rlandy|ruckmarios: mostly we don;t track check times14:51
rlandy|ruckmy guess is that this job will start shortly14:51
rlandy|ruckthe one above has just started14:52
mariosrlandy|ruck: its been like 4+ hours14:52
rlandy|ruckthe ques don't look stuck - just loaded14:52
mariosrlandy|ruck: ive resisted the recheck till now so i don't lose my place in the queue :!14:52
rlandy|ruckmarios: remember people had to recheck stuff after yesterday's move to opendev14:52
mariosrlandy|ruck: me looking at http://zuul.openstack.org/status for 65495214:53
mariosrlandy|ruck: well it was posted today14:53
mariosrlandy|ruck: k... gonna recheck it...14:53
rlandy|rucksays 6:58am14:54
rlandy|rucklooks like today14:54
mariosrlandy|ruck: 17:53 < marios> rlandy|ruck: well it was posted today14:54
rlandy|ruckso it's no so far behind14:54
rlandy|ruckcorrect14:54
mariosrlandy|ruck: i rechecked...14:54
mariosoh well..14:54
rlandy|ruckmarios:I don;t think recheck will help you :(14:54
mariosyeah14:55
mariosi guess i'll see results tomorrow ;)14:55
*** ykarel is now known as ykarel|away15:08
marioshttps://review.opendev.org/#/c/653951/2 docs review please thanks rfolco weshay|rover sshnaidm about emit-release http://logs.openstack.org/51/653951/2/check/openstack-tox-docs/90eca1b/html/ci/emit_releases_file.html15:09
marios& related https://review.opendev.org/#/c/652095/ commenting the functions15:12
* marios adds trown to those reviews 15:12
*** dsneddon has joined #oooq15:12
*** ccamacho has joined #oooq15:17
rlandy|ruckweshay|rover: been looking at the timeouts - 0.7% - only one today15:17
rlandy|ruckin introspection15:18
rlandy|ruckquiquell|rover: ^^15:18
rlandy|rucknot sure that this is a major problem15:18
rlandy|ruckthoughts?15:18
*** dsneddon has quit IRC15:19
* marios shutdown sequence 15:21
weshay|roverya.. /me sees just plain old failures today15:22
weshay|roverafter poking a bit more15:22
mariosweshay|rover: added some notes there for now https://etherpad.openstack.org/p/tripleo-ptg-train-ci-squad15:23
*** aakarsh has quit IRC15:27
*** jfrancoa has quit IRC15:30
weshay|rovercan someone paste me a skeopeo command?15:33
weshay|rovermarios add it to the main tripleo pad15:33
weshay|rovermarios https://etherpad.openstack.org/p/tripleo-ptg-train15:34
*** dsneddon has joined #oooq15:35
mariosweshay|rover: ack i was planning on doing that after discussing with you. but we probably also want that https://etherpad.openstack.org/p/tripleo-ptg-train-ci-squad so we can keep our session's notes? assuming folks will do that like we had each team with its own pad. anyway ack will do tomorrow15:35
weshay|roverya.. ur right15:35
* weshay|rover added what you had to the main pad, but we can use the ci pad to refine things15:36
mariosweshay|rover: ack thx15:36
weshay|roverquiquell|rover can you join my blue15:37
weshay|roverrlandy|ruck ^15:38
quiquell|roversure15:38
*** dsneddon has quit IRC15:42
*** ccamacho has quit IRC15:46
*** ccamacho has joined #oooq15:47
*** marios has quit IRC15:47
*** dsneddon has joined #oooq15:50
rlandy|rucksshnaidm: I'd like to take standalone master job out of changes to projects15:52
rlandy|ruckruns and fails - no real testing15:52
sshnaidmrlandy|ruck, ok15:52
rlandy|ruckweshay|rover: you seen luca around today15:52
rlandy|ruckweshay|rover: don;t see him anywhere on irc15:53
chandankumararxcruz: kopecmartin weshay|rover sshnaidm https://review.opendev.org/#/c/654406/ please have a look on os_tempest docs15:53
weshay|roverchandankumar aye.. getting there :)15:53
*** sshnaidm is now known as sshnaidm|afk15:54
*** ykarel|away has quit IRC15:55
quiquell|roverok droping now15:55
quiquell|roverrlandy|ruck, weshay|rover: about f28 there is something fishy at rdo fedora-28 DIB will continue tomorrow looking at it15:55
*** dsneddon has quit IRC15:56
rlandy|ruckquiquell|rover: weshay|rover; ok - maybe we'll pick up that investigation this afternoon15:56
arxcruzchandankumar: you have my -1 :)15:57
quiquell|roverrlandy|ruck: let me paste the IPs with hold nodes from RDO15:57
quiquell|roverrlandy|ruck, weshay|rover: give me your pub keys15:58
weshay|roverhttps://github.com/weshayutin.keys15:58
quiquell|roverweshay|rover: can you access zuul@38.145.33.18815:59
weshay|roverquiquell|rover aye.. what is it?16:00
chandankumararxcruz: thanks, done16:00
weshay|roverstandalone dpeloyment16:00
weshay|roverquiquell|rover clark is going to run an updated report for the last 30days16:00
quiquell|rovercool16:00
quiquell|roverthat will be good16:00
rlandy|ruckquiquell|rover: key sent16:01
weshay|roverquiquell|rover what is that?16:02
quiquell|roveryou can see that python3-setuptool rpm is installed16:02
quiquell|roverbut the content is deleted16:02
weshay|roverahhh16:02
quiquell|roverholded RDO node with the fedora28 failure16:02
quiquell|roverpython3-setuptools issue16:02
weshay|roversee.. I knew this shit was crazy16:02
quiquell|roverThis could be the fix but breaks reproducer https://review.opendev.org/#/c/654950/16:03
rlandy|ruckI think we should apply it16:03
quiquell|roverWeir stuff not reinstalling python3-setuptools works fine with upstream dib f28 images16:03
rlandy|ruckand try fix reproducer another way16:03
quiquell|roverrlandy|ruck: we will break f28 jobs at reproducer if we apply that16:03
quiquell|roverrlandy|ruck: we can bypass reinstall checking versions I think16:04
quiquell|rovercould be a fix16:04
quiquell|roveryou decide merge the revert to unblock fedora28 if you want16:04
quiquell|roverHave to drop16:04
quiquell|roverread you tomorrow16:04
*** quiquell|rover is now known as quiquell|off16:04
rlandy|ruckweshay|rover: can we bj about this?16:05
quiquell|offrlandy|ruck: I can jump there for a few to explain16:05
quiquell|offso you decide16:05
rlandy|ruckwe need to get done here16:06
weshay|roverVery nice guys16:07
weshay|roverand gals16:07
weshay|rover12/06/2018 http://paste.openstack.org/show/736797/  43% use of all upstream resources16:07
weshay|rover04/23/2019 http://paste.openstack.org/show/749651/  25% use of all upstream resources16:07
*** jbadiapa has quit IRC16:11
weshay|roverFYI https://docs.google.com/document/d/1iPYYlXjftbAhxr-zhIVpmGzYrjwzQYDDW6ay_4GsZ7w/edit16:16
*** ykarel|away has joined #oooq16:16
weshay|roverchandankumar how much more time is in your day?16:16
weshay|roverrfolco oh wow you took panda off the meeting16:17
weshay|roverrfolco we should have some overlap :)16:17
rfolcoweshay|rover, he asked for16:17
weshay|roverpanda you still with us?16:17
chandankumarweshay|rover: I will be lurking around irc for 30 more mins :-)16:17
weshay|roverrfolco chandankumar ok.. let's move up this meeting to get chandankumar out of here16:18
rfolcoweshay|rover, move to when16:18
weshay|roverrfolco chandankumar I rfolco just moved it16:18
weshay|roverchandankumar rfolco lets join16:18
weshay|roverhttps://bluejeans.com/u/whayutin/16:18
*** kopecmartin is now known as kopecmartin|off16:23
pandaso it's stein pronounced "stin" like "frankenstin" ?16:25
*** dtantsur is now known as dtantsur|afk16:28
*** dsneddon has joined #oooq16:29
*** dsneddon has quit IRC16:37
*** aakarsh has joined #oooq16:38
vinayknsweshay|rover: hey, I'm looking to have oooq deployment which has new qdrouter image in osp1316:41
vinayknsI'm not sure, what extra stuff that I need to do to get the updated image in my osp13 deployment...16:42
chandankumarweshay|rover: https://tree.taiga.io/project/tripleo-ci-board/us/867?milestone=22626716:43
*** dsneddon has joined #oooq16:45
*** tosky has quit IRC16:46
chandankumarweshay|rover: periodic jobs https://tree.taiga.io/project/tripleo-ci-board/us/97216:46
*** udesale has quit IRC16:47
*** altlogbot_1 has quit IRC16:50
*** altlogbot_1 has joined #oooq16:51
*** jpena is now known as jpena|off16:52
*** dsneddon has quit IRC16:53
weshay|roverpanda rfolco chandankumar https://docs.google.com/document/d/15WOk-ZwiDI4f3dwqDEJbKXpRiY61gFfIKBFH4CgZ8-s/edit?ts=5cbf2a6a#heading=h.eviqxsrl36bf16:54
*** altlogbot_1 has quit IRC16:54
*** altlogbot_0 has joined #oooq16:55
*** derekh has quit IRC16:57
*** jbadiapa has joined #oooq17:05
*** dsneddon has joined #oooq17:10
*** dsneddon has quit IRC17:14
*** dsneddon has joined #oooq17:15
vinayknsweshay|rover: ping.17:45
rlandy|ruckweshay|rover: ping re: 2019-04-23 16:01:37 | Exception: Not found image: docker://docker.io/tripleorocky/centos-binary-zaqar:ae5ac2c882d7eaf07708cf6ff858961f6ac6abc5_9c673bd517:50
rlandy|ruck2019-04-23 16:01:37 | time="2019-04-23T16:01:37Z" level=fatal msg="Error reading manifest ae5ac2c882d7eaf07708cf6ff858961f6ac6abc5_9c673bd5 in docker.io/tripleorocky/centos-binary-zaqar: manifest unknown: manifest unknown"17:50
weshay|roverrlandy|ruck from which job?17:53
*** ccamacho has quit IRC17:54
*** amoralej is now known as amoralej|off17:56
rlandy|ruckweshay|rover; fs001 rocky branch17:58
rlandy|ruckthe container is missing17:58
rlandy|ruckhttp://38.145.34.55/centos7_rocky.log17:58
rlandy|ruck2019-04-23 10:08:20,389 7121 INFO     promoter TASK [Check if dockerhub has all images] *17:59
weshay|roverrlandy|ruck is it on the rdo reg?17:59
rlandy|ruckchecking17:59
rlandy|ruckweshay|rover: ack18:00
rlandy|ruckrdo reg has it docker does not18:01
rlandy|ruckno record of it being pushed either18:01
rlandy|ruckhttps://github.com/openstack/tripleo-common/blob/stable/rocky/container-images/overcloud_containers.yaml.j218:03
* weshay|rover looks18:08
weshay|roverrlandy|ruck hrm18:08
rlandy|ruckweshay|rover: this is a new failure  - as of today18:09
rlandy|ruckmaybe it's a container push hitch18:09
weshay|roverrlandy|ruck let's see if it was built18:09
weshay|rovershould we chat?18:09
weshay|roverrlandy|ruck http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-rocky-containers-build-push/47f8930/logs/build.log.txt.gz18:11
weshay|roverzaqar is there18:11
rlandy|ruckcentos-binary-zaqar not18:12
weshay|roverhttps://console.registry.rdoproject.org/registry#/images/tripleorocky/centos-binary-zaqar18:12
weshay|rover$ sudo docker pull trunk.registry.rdoproject.org/tripleorocky/centos-binary-zaqar:ae5ac2c882d7eaf07708cf6ff858961f6ac6abc5_9c673bd518:12
weshay|roverperhaps we need to repush that hash of containers to docker.io18:13
rlandy|ruckI think so18:13
rlandy|ruckunless we wait for the next push18:13
weshay|roverthe failure was in upstream so I think we have to try and fix it18:13
weshay|roverrlandy|ruck which upstream job is failing w/ that atm18:14
weshay|rover?18:14
rlandy|ruckweshay|rover: ovb rocky-branch18:15
* weshay|rover refreshes18:15
weshay|roverhttp://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&fullscreen&panelId=20718:15
weshay|roverhttps://review.opendev.org/#/c/614632/18:15
weshay|roverrlandy|ruck ah.. see it http://logs.rdoproject.org/52/654952/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch/14bd44b/logs/undercloud/home/zuul/install-undercloud.log.txt.gz18:18
weshay|roverrlandy|ruck https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/container-push/manual_push.sh18:20
weshay|roverI don't think I've use that before18:20
weshay|roverhave you18:20
rlandy|ruckweshay|rover: I have not but I was looking at it when I asked18:20
rlandy|ruckI think we just go to the server18:21
rlandy|rucksource the secret and run18:21
rlandy|ruckI think rfolco has18:21
weshay|roverya.. rlandy|ruck let's tmate / chat18:21
rlandy|ruckweshay|rover: k - let's close out this and f2818:22
ykarel|awayrlandy|ruck, weshay|rover mmm so opendev transition caused this image issue18:33
rlandy|ruckykarel|away: how so?18:33
ykarel|awayrlandy|ruck, https://review.rdoproject.org/r/#/c/20406/18:34
ykarel|awayrlandy|ruck, updated url is wrong18:34
rlandy|ruckykarel|away: branch/master18:34
rlandy|ruckweshay|rover: https://review.rdoproject.org/r/#/c/20406/1/ci-scripts/container-push/container-push.yml18:35
ykarel|awayrlandy|ruck, for clearity paste https://git.openstack.org/cgit/openstack/tripleo-common/plain/container-images/overcloud_containers.yaml.j2?id=40fa70be3338b0ad84ff58b2be8bc19254a24b27 in browser18:35
ykarel|awayand notice the redirect18:35
ykarel|awaythere may be some other issue as well, but atleast ^^ needs to be fixed18:36
rlandy|ruckykarel|away: fixing18:36
ykarel|awayack18:38
* ykarel|away leaving18:49
ykarel|awayi think the issue should be fixed with just fixing the url ^^, issue detected with zaqar as in newer release image name is zaqar-wsgi18:52
ykarel|awayanyway will catchup tomorrow18:52
* ykarel|away out18:52
*** ykarel|away has quit IRC18:57
*** dtrainor_ is now known as dtrainor19:01
-openstackstatus- NOTICE: the zuul scheduler is being restarted now in order to address a memory utilization problem; changes under test will be reenqueued automatically19:07
rlandy|ruckweshay|rover: https://review.rdoproject.org/r/2042719:24
rlandy|ruckweshay|rover: quiquell|off's revert is the right fix ... https://review.rdoproject.org/zuul/stream/272dbf1ab15e4fe4816d76bf8bddae97?logfile=console.log20:05
weshay|roverso weird20:06
weshay|roverk20:06
rlandy|ruckweshay|rover: promoter ... oh dear ... we're back to the multiple promotions running concurrently20:13
weshay|roverugh20:13
rlandy|ruckI think it's because we stopped and started the cron20:13
weshay|roverreally?20:13
rlandy|ruckweshay|rover: check logs master, rocky20:13
rlandy|rucklet's stop and restart20:14
rlandy|ruckback on tmate20:14
weshay|roverhrm.. not sure if I closed20:14
weshay|roverI did20:14
weshay|roverstupid20:14
weshay|roverrlandy|ruck btw.. we need stein next, not master20:15
weshay|roverhttp://rhos-release.virt.bos.redhat.com:3030/rhosp20:15
weshay|roverrlandy|ruck give me a ssh when you have it20:15
rlandy|ruckweshay|rover: join me pls on tmate20:15
rlandy|rucksame as before20:15
weshay|roverah k20:15
*** aakarsh has quit IRC20:30
*** Vorrtex has quit IRC20:44
*** Goneri has quit IRC20:59
*** aakarsh has joined #oooq21:06
*** mjturek has quit IRC21:12
*** rfolco has quit IRC21:24
*** vinaykns has quit IRC21:26
*** tosky has joined #oooq21:28
*** smalleni_ has quit IRC21:30
*** vinaykns has joined #oooq22:12
*** amoralej|off has quit IRC22:27
*** radez has quit IRC22:28
*** dalvarez has quit IRC22:29
*** radez has joined #oooq22:44
*** tosky has quit IRC23:13
rlandy|ruckweshay|rover: want to exclude tripleo-quickstart-promote-stein-current-tripleo-delorean-minimal from stein as well?23:14
rlandy|rucknvm - running tempest now23:15
*** rlandy|ruck has quit IRC23:40
« Monday, 2019-04-22 Index Wednesday, 2019-04-24 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!