Wednesday, 2023-01-04

akahat|rovero/04:32
marioso/06:09
*** ysandeep|out is now known as ysandeep06:20
ysandeepakahat|rover, good morning o/06:25
chandankumarysandeep: marios akahat|rover Good morning and Happy new year o/06:26
ysandeepakahat|rover, You need to update promotion-testing-hash-* files for periodic job to pick it up. https://review.rdoproject.org/zuul/status#46525 will not work 06:26
ysandeepchandankumar, happy new year06:27
marios\o chandankumar happy new year06:27
*** soniya is now known as soniya2906:32
chandankumarthank you ysandeep: akahat|rover rlandy|out for taking care of growvols bug :-)06:32
ysandeepThat's a weird bug, we are now saying newer lvm2/device-mapper is culprit, but I had bad results with older lvm2/device-mapper as well.06:37
ysandeephttps://logserver.rdoproject.org/37/28537/66/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-zed/2f30180/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz06:37
ysandeepovercloud package list: https://logserver.rdoproject.org/37/28537/66/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-zed/2f30180/logs/overcloud-controller-0/var/log/extra/package-list-installed.txt.gz 06:38
marios:/ so we are still chasing that one then 06:38
chandankumarysandeep: may be we can first get passing job package exclude list patch merged and then start removing packages packages one by one or combination?06:41
chandankumarto find the actual culprit06:42
mariosakahat|rover: pojadhav|ruck: sent invite for sync call at usual time please check inbox 06:44
ysandeepchandankumar, with https://review.opendev.org/c/openstack/tripleo-quickstart/+/868605/13/config/release/tripleo-ci/CentOS-9/master.yml last 8/8 job pass node provisioning06:45
ysandeeplet me reduce the diff to just device-mapper/lvm2 to see if that enough(as per steve's comment).. but last week i hit issues with older device-mapper/lvm2 too.06:45
marioscould it be something really nasty like combination of kernel and lvm for example06:46
ysandeepyes could be06:46
mariosysandeep: arent we already testing that per akahat https://review.opendev.org/c/openstack/tripleo-quickstart/+/869099/3/config/release/tripleo-ci/CentOS-9/master.yml06:46
chandankumarnot sure selinux-policy package06:46
mariosysandeep: i mean 08:45 < ysandeep> let me reduce the diff to just device-mapper/lvm2 to see if that enough(as per steve's comment).. but last week i hit issues with older device-mapper/lvm2 too.06:46
ysandeepmarios, yes I mean to update akahat|rover patch, his patch won't work with periodic job06:47
ysandeep>> <ysandeep> akahat|rover, You need to update promotion-testing-hash-* files for periodic job to pick it up. https://review.rdoproject.org/zuul/status#46525 will not work 06:47
mariosysandeep: ah ok 06:47
pojadhav|ruckmarios, accepted !06:49
marios\o06:51
mariosakahat|rover: ysandeep: chandankumar: can you check https://review.opendev.org/c/openstack/tripleo-quickstart/+/869121 rabi posted revert and content providers look good there06:53
marios(when you have time please thanks)06:53
chandankumardone!06:54
ysandeepdouble +wed :D06:55
mariosthanks :)06:56
akahat|roverysandeep, o/ hey.. thanks i'll update the patch.. 07:01
ysandeepakahat|rover, nw, I already posted it07:02
akahat|rovermarios, accepted invite. thanks!07:02
marioso/07:02
* akahat|rover lunch07:30
*** ysandeep is now known as ysandeep|afk07:42
Tenguhello there07:57
Tenguchandankumar: heya! fyi, I'll work a bit on tripleo-lab in order to support the standalone + external compute. That should help me understanding why the modifications I'm trying to get in are apparently failing. I think I may have an idea, but it's really weird.07:58
chandankumarTengu: hello o/ Happy new year :-)08:27
Tenguchandankumar: HNY :)08:27
chandankumarTengu: I was checking the current logs(not fully), I think few tasks are getting skipped that's why compute is not getting deployed 08:28
Tenguchandankumar: apparently the "target_host" points to "undercloud" instead of "subnode-1"08:29
Tenguguess it's not properly passed.08:29
Tenguthat's why08:29
Tenguso it's apparently pure ansible being dumb (with my help)08:29
Tenguit *should* work.08:29
chandankumarok08:30
chandankumarah https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/866864/25/playbooks/deploy-standalone-compute.yml#4408:30
chandankumarthat's why you are defining target host here08:30
Tenguyeah08:31
chandankumarby the way the what is the idea08:31
Tengutrying to. last log showed "undercloud" as the value instead of "subnode-1".08:31
Tenguchandankumar: which idea?08:31
chandankumarTengu: sorry on this one "I think I may have an idea, but it's really weird."08:32
Tenguah, that. well, the param not being passed from the play level.... sounds weird.08:32
Tengubasically, everything is launched from the undercloud, ON the undercloud :/.08:32
Tenguthe only ansible trace we see on the subnode-1 is the first play, when it gathers facts; and when collect-logs role is launched.08:33
chandankumarTengu: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1fb/866864/24/check/tripleo-ci-centos-9-standalone-external-compute-target-host/1fbb44c/logs/undercloud/usr/share/ansible/tripleo-inventory/02-computes08:37
chandankumarcan we change this localhost: to subnode-1 ip or hostname08:37
Tenguchandankumar: it's not supposed to use that inventory 08:38
Tenguchandankumar: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1fb/866864/24/check/tripleo-ci-centos-9-standalone-external-compute-target-host/1fbb44c/logs/undercloud/home/zuul/tripleo-deploy/edpm-inventory/02-computes08:38
chandankumaryes correct as per the chnage08:39
pojadhav|ruckmarios, https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status08:40
Tenguchandankumar: I must say, I'm a bit surprised by the overally behavior.... 08:43
chandankumarthere is nothing on subnode-1208:44
Tenguchandankumar: even with the --limit, it seems to not do the intended thing.08:44
chandankumarTengu: Do you want to hold a node on rdo side via tp and debug that?08:46
chandankumarwhat if we put the ip of subnode-1 here https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1fb/866864/24/check/tripleo-ci-centos-9-standalone-external-compute-target-host/1fbb44c/logs/undercloud/home/zuul/tripleo-deploy/edpm-inventory/02-computes08:47
chandankumarand before running the playbook we can run some adhoc command 08:47
chandankumarand then run the playbook08:47
Tenguchandankumar: that's the enxt step yeah - injecting the ansible_host in the subnode inventory entry.08:48
* chandankumar running a tp and holding the node08:49
Tengufor now I want to see if we use the correct node.08:49
akahat|roverysandeep|afk, marios excluding lvm2 and libblockdev didn't worked: https://review.rdoproject.org/zuul/status#46525 I think we need to add more packages.08:52
akahat|roverlvm2 and device-mapper *08:53
chandankumarTengu: ok08:55
Tenguchandankumar: and, as said, I'm working in order to add that support in tripleo-lab - that will help me better understanding the whole thing.08:55
Tenguchandankumar: also, we may add some more tests in the validate-compute, for instance by ensuring we DO have a hypervisor not named "standalone.localdomain" or the like.08:56
chandankumarTengu: yup, sounds good .08:56
mariosthanks akahat|rover so lets see with ysandeep|afk ongoing tests too then 08:56
chandankumarLet me restart the tempest work there also so that we have proper validation in place there08:57
mariospojadhav|ruck: dasm: fyi added task into backlog https://issues.redhat.com/browse/TRIPLEOCI-1342 (via chat rhos-ops just now)08:57
Tengusure08:57
mariosnot sure who or when but just captured it for now pojadhav|ruck dasm 08:57
pojadhav|ruckmarios, ack09:09
* pojadhav|ruck lunch09:09
*** soniya29 is now known as soniya29|afk09:20
marioschandankumar: bhagyashris: ysandeep|afk: did we already move jobs to rhos1? i didn't think we did? (d/stream node_failures may be due to missing rhos-1 flavor)09:39
mariosdpawlik: ^^ 09:40
chandankumarmarios: nope, we have not moved any jobs to rhos-0109:57
chandankumarmarios: I tried to run jobs on rhos-01 last week but got hit by node failure09:57
*** ysandeep|afk is now known as ysandeep09:59
mariosk thanks for confirm chandankumar 10:04
dpawlikchandankumar: ack10:08
dpawlikchandankumar: you can try now10:08
dpawlikwe changed the flavor, job should start now10:08
chandankumardpawlik: ok10:17
*** soniya29|afk is now known as soniya2910:56
* akahat|rover coffee10:59
*** ysandeep is now known as ysandeep|afk11:07
*** rlandy|out is now known as rlandy11:14
mariosakahat|rover: o/ around can you join us please https://meet.google.com/vzm-nrah-qqf11:31
chandankumarTengu: I think switching to ip will work11:32
*** dviroe|out is now known as dviroe11:32
*** dviroe is now known as dviroel11:32
chandankumarTengu: https://paste.openstack.org/raw/bmXceV5YmRZYcdvJyJD7/11:33
chandankumar--limit does not work11:33
chandankumarrlandy: dviroel good morning & Happy new year o/11:33
rlandychandankumar: sec - rr syncing - will talk to you in a few11:34
dviroelchandankumar: hi o/11:34
rlandyakahat|rover: if you are around - pls join11:37
akahat|roverrlandy, yes. joining11:37
bhagyashrismarios, hey kindly +w on https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/43830911:43
chandankumarTengu: left comment on the patch11:45
pojadhav|ruckrlandy, https://code.engineering.redhat.com/gerrit/c/openstack/sf-config/+/43831311:49
pojadhav|ruckrlandy, https://code.engineering.redhat.com/gerrit/c/openstack/sf-config/+/43831011:49
mariosrlandy: https://code.engineering.redhat.com/gerrit/c/openstack/sf-config/+/43831011:52
pojadhav|ruckrlandy, marios : https://code.engineering.redhat.com/gerrit/c/testproject/+/436232/11#message-ec2b3a0c582010d3867c8fc912fca84ba66675ad12:04
rlandyakahat|rover: https://72bac32aa1e395a959c2-fe1bae3c47bea9de602eec2d1e53cbdb.ssl.cf1.rackcdn.com/869020/4/check/tripleo-ci-centos-9-content-provider/59909ec/logs/undercloud/var/log/extra/package-list-installed.txt12:15
rlandyhttps://b2b9d08c5b85917c3495-18cd7f889ae1379763e7ec0a1ec70a8d.ssl.cf2.rackcdn.com/869121/1/gate/tripleo-ci-centos-9-content-provider/ae8d9f4/logs/undercloud/var/log/extra/package-list-installed.txt12:15
rlandyqemu-img.x86_64                                  17:7.2.0-2.el9                        @quickstart-centos-appstreams 12:15
rlandyqemu-img.x86_64                                  17:7.1.0-6.el9                        @quickstart-centos-appstreams 12:15
rlandyhttps://b2b9d08c5b85917c3495-18cd7f889ae1379763e7ec0a1ec70a8d.ssl.cf2.rackcdn.com/869121/1/gate/tripleo-ci-centos-9-content-provider/ae8d9f4/logs/container-builds/86794729-b591-48ae-b45a-050377a223ac/base/os/nova-base/nova-libvirt/nova-libvirt-build.log12:17
rlandyjq: error: Could not open file /usr/share/qemu/firmware/50-edk2-ovmf-cc.json: No such file or directory12:17
rlandyjq: error: syntax error, unexpected end of file (Unix shell quoting issues?) at <top-level>, line 1:12:17
rlandy.mapping["nvram-template"] =                             12:17
rlandyjq: 1 compile error12:17
* pojadhav|ruck needing tea break.. back in few mins..12:18
chandankumardviroel: I got the kubeconfig for rhos-1 cluster and I am able to install ansiblee operator12:18
chandankumarand run sample playbook12:18
* marios food 12:18
chandankumarnext thing I need to deploy a full control plane on that operator12:18
chandankumarfor getting creds https://code.engineering.redhat.com/gerrit/c/testproject/+/438178/2/playbooks/cluster_creds.sh#412:18
chandankumarI passed the rhos-01 namespace it gave me the creds12:19
Tenguchandankumar: oh, you mean the inventory file is borked, and "all" doesn't work?12:22
chandankumaryes all is not working12:22
Tengupfrtt.12:22
*** ysandeep|afk is now known as ysandeep12:27
bhagyashrisrlandy, marios ysandeep|afk fyi, added all the required reviews link to start the rhos18 on rhel9 integration line in the review hackmd https://hackmd.io/FGMoCiRfSNa8puA1BpTQ-Q#Jan-4-2023 as their dependency   12:27
reviewbotDo you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks.12:27
dviroelchandankumar: looking12:28
dviroelchandankumar: hive says that there is no claim for rhos-1, you might be using another clusterclaim12:28
chandankumardviroel: I have created the claim, it is still in initialized state12:30
chandankumardviroel: without claim, I can get the kubeconfig for a namespace12:30
dviroelchandankumar: "stage-1-claim" 12:30
dviroelon your change12:30
dviroel"bash /home/zuul/src/code.engineering.redhat.com/testproject/playbooks/cluster_creds.sh stage-1-claim > ~/.kube/stage-1-claim-kubeconfig"12:31
dviroelchandankumar: only rhos-01-01-bk9fs-master-0 is on rhos-112:32
rlandyakahat|rover: can you try exclude that qemu rpm so it uses the old one as a workaround to unblock gate?12:32
rlandymarios: ^^12:33
rlandyin release files12:33
rlandychandankumar: dviroel: sorry - reading back - was on rr issues12:33
chandankumardviroel: http://pastebin.test.redhat.com/108686112:34
chandankumarrlandy: dviroel if you can come on a call, then I can show the same12:35
chandankumarlet me know when free12:39
soniya29arxcruz, rlandy, please add/edit today's agenda for tempest meeting - https://hackmd.io/xWJLuBHGSI2YGX7UlkwRbg?both12:40
ysandeepmarios, hey o/ I saw your comment on https://review.opendev.org/c/openstack/tripleo-quickstart/+/868605/13/config/release/tripleo-ci/CentOS-9/master.yml#177 to only try with kernel/device-mapper/lvm2.. akahat|rover already tried that here: https://review.opendev.org/c/openstack/tripleo-quickstart/+/869099/6/config/release/tripleo-ci/CentOS-9/promotion-testing-hash-master.yml , didn't help12:40
arxcruzsoniya29: i don't have anything 12:40
arxcruzshould we cancel it?12:40
rlandysoniya29: arxcruz: ok with canceling12:40
soniya29yeah, we can, i also dont have anything new and martin is also on pto12:41
ysandeepakahat|rover, looks like only wallaby test where we excluded systemd is procedding further12:41
akahat|roverysandeep, yes. https://review.opendev.org/c/openstack/tripleo-quickstart/+/869099/6/config/release/tripleo-ci/CentOS-9/promotion-testing-hash-wallaby.yml#14412:41
soniya29alright, i will cancel the meeting :)12:41
akahat|roverthis is looks promissing12:41
rlandybhagyashris: thanks - will review later12:41
akahat|roverysandeep, i'll exclude same packages in all release12:41
akahat|roverand then retry 12:41
akahat|roverafter job passes12:41
ysandeepakahat|rover, ack, for 1 release please only exclude systemd12:42
dviroelchandankumar: your clusterclaim is in the wrong namespace, thats why it is still PENDING -> Initialized. Must be on "oooci" namespace.12:42
akahat|roverysandeep, sure.. got your point. :)12:42
dviroelchandankumar: you still can reach credentials of that cluster, in the way you did with "jsonpath='{.spec.clusterMetadata.adminKubeconfigSecretRef.name}')"12:43
ysandeepdviroel, hey o/ merged your flavor change PR few hours ago.12:43
dviroelchandankumar: but you don't have the control of who is claiming the cluster12:44
dviroelysandeep: thanks man, hive is already creating new clusters on the new flavor12:44
chandankumardviroel: thanks 12:44
chandankumardviroel: it worked now12:44
dviroelchandankumar: \o/12:45
ysandeepdviroel, I have a query - what was the reason to delete some files there like install-config-sno_secret_osp_03.yaml12:45
chandankumardviroel: the job is running in rhos-1 cloud12:45
dviroelysandeep: we were not using them anymore. I merged those files when Hive Inventory wasn't working. If you check those deleted files, you will see FIP info there.12:46
dviroelysandeep: this info lives on ClusterDeploymentCustomization now: https://github.com/openshift/release/blob/master/clusters/hive/pools/openstack-k8s-operators/openstack-k8s-operators-cdc-vexx-01.yaml12:47
dviroelysandeep: and are consumed by the ClusterPool here: https://github.com/openshift/release/blob/9cd4cc1fa636452d52f63ae62d9c50663e8114d6/clusters/hive/pools/openstack-k8s-operators/openstack-k8s-operators-4-10-vexxhost-sno_clusterpool.yaml#L2312:47
dviroelchandankumar: i see a node there12:47
ysandeepdviroel, ahh okay, thanks!12:47
rlandychandankumar: can meet now if you want12:49
chandankumarrlandy: sure12:49
chandankumarrlandy: dviroel https://meet.google.com/gkq-weqq-xzo?authuser=0&hl=en12:50
chandankumarrlandy: dviroel just joined12:57
*** bhagyashris is now known as bhagyashris|afk13:17
akahat|roverysandeep, marios rlandy updated patch with requested package exclude by ysandeep. https://review.opendev.org/c/openstack/tripleo-quickstart/+/869099, https://review.rdoproject.org/r/c/testproject/+/4652513:32
akahat|roveri've rekicked tp.. 13:33
akahat|rovermarios, rlandy https://bugs.launchpad.net/tripleo/+bug/2001626, Bogdan working on the issue, he has some patches up for it.13:34
dasmo/13:40
mariosthanks akahat|rover can you please also notify mailing list since its new gate blocker13:40
akahat|rovermarios, rlandy i'm leaving to home.. i'll be back online after some time.. I've updated the CIX cards with the latest info13:40
akahat|rovermarios, yes. i've already done.13:40
dviroelrlandy: while I was writing the 3rd solution: it ends up in the same situation as the 1st. Even if we have a isolated tenant, we plan to run nodepool nodes there, so we need to care about leaking FIPS there. In the end, where nodepool runs, we need to have a FIP cleanup solution. 13:40
dasm08:57 <marios> pojadhav|ruck: dasm: fyi added task into backlog13:40
mariosakahat|rover: can you please update the xfs card with latest? 13:41
marioslooking13:41
dasmmarios: thanks. not need to announce though :)13:41
mariosack thanks akahat|rover looks like you did13:41
mariosdasm: o/ 13:41
akahat|rovermarios, yeah.. see you in a bit.13:41
*** ysandeep is now known as ysandeep|away13:49
*** pojadhav- is now known as pojadhav|ruck13:58
pojadhavrlandy, marios : i can rhos17.1 line running smoothly atleast build image/containers and standalone jobs...14:06
rlandydviroel: correct - but we would not impact all jobs14:25
rlandyjust the ones that require have14:25
rlandyie: if FIPs caused an issue, other jobs would keep running ie: less impact14:25
rlandychandankumar: hi  - I thought I added the EDPM team meeting to my calendar but I don;t see it14:27
rlandycan you remind me of the time/day?14:27
dviroelrlandy: new vexx flavor worked, job is passing again14:27
rlandydviroel: \o/14:28
chandankumarrlandy: OSP NG: External Data Plane Weekly Status: Timing Monday, 2:30 PM UTC14:29
rlandychandankumar: ah 0 I have it - it was just not scheduled this week14:30
chandankumarrlandy: yup14:30
mariosrlandy: firefox finally gave up... restarting it and rejoining14:40
*** dviroel is now known as dviroel|lunch14:54
* pojadhav lunch15:08
* pojadhav sorry... dinner 15:08
Tengumarios: that's why I use a dedicated chromium instance for meet/calendar/gchat - the rest is running in firefox :).15:11
mariosTengu: :) yeah i mean i also have chrome just for google stuff (mail/calendar/chat etc also slack and also radio(s)/sounds) but for some reason sharing doesn't work for me with chrome & meet15:12
mariosTengu: so i use ffox for it 15:12
mariosTengu: maybe should try chromium instead15:12
rlandyI have the same setup as Tengu 15:15
rlandychrome runs G stuff15:15
rlandyfirefox runs the rest15:15
mariosme too :)15:19
chandankumarsee ya people tomorrow!15:19
rlandysee you tomorroew15:19
mariosexcept i cant get meet to work properly on chrome so have ff for it 15:19
marioso/ ch15:19
marioschandankumar: o/15:19
Tengumarios: chromium + sharing works fine here at least.15:23
Tengusee you tomorrow chandankumar 15:23
*** dviroel|lunch is now known as dviroel16:06
rlandylunch brb16:30
* marios off in few have a good rest of day oooq o/ 16:34
*** marios is now known as marios|out16:39
dasmmarios|out: o/16:41
Tenguok, bogdando patch is correcting the content-provider issue.16:46
Tengucould validate it with my EDPM work.16:46
Tenguheh, we may need to get it in Zed before master16:46
Tenguhttps://review.opendev.org/c/openstack/tripleo-common/+/86920616:46
Tengurlandy, marios|out wdyt? zed before master, or content-provider-zed on master as non-voting until we get it in master + zed?16:47
rlandyTengu: master first16:48
rlandyhas more jobs16:48
rlandythen zed16:48
Tengurlandy: so we must switch content-provider-zed to non-voting, as well as the dependents...16:49
Tengusince content-provider-zed takes its code from the stable/zed branch. which is unpatched.16:49
Tengu:)16:49
rlandyif you did zed first?16:50
Tenguthat's a nice Ouroboros situation.16:50
rlandyyou would need to make master non-voting16:50
rlandyon a job that changes master ....16:50
rlandyonly zed and master run16:50
rlandyoh and wallaby16:51
Tenguwallaby doesn't seem affected?16:51
rlandyif you look at the dependencies ...16:51
Tenguso, if we're to get master in first, we must set content-provider-zed and undercloud-upgrade to non-voting16:52
rlandyyeah 16:52
rlandysmaller list than everything that depends on master16:52
Tenguok16:52
Tengusoooo it's tripleo-upgrades-master-pipeline apparently?16:53
rlandytemp comment that out?16:55
Tenguprobably easier/better than hacking through tripleo-ci16:55
marios|outreally got to go Tengu will check in morning... but if we have content provider broken then what is providing the container (regardless if it is non voting)16:55
Tenguas a dedicated patch so that it's easier to revert?16:55
Tengumarios|out: np - will figure out a way16:56
marios|outoh i see now .. you mean content provider zed non voting to merge the master fix16:57
Tengumarios|out: or https://review.opendev.org/c/openstack/tripleo-common/+/869240 (cc rlandy )16:58
Tengulet's see if it does the intended thing, then I can rebase bogdan's fix onto it.16:58
Tenguand we should get out of that chicken'n'egg issue.16:58
marios|outTengu: yeah maybe this one is better... but can be combined into bogdan patch and followed by partial revert which is rebased onto that16:59
Tengumarios|out: I hage partial reverts..16:59
marios|out:)16:59
marios|outanyway got to go now... will pickup in ~13 hours ;)16:59
marios|outhave a good one o/ 16:59
Tenguok - repushed as:  comment / bogdan's / revert-comment17:01
Tenguthat should do it.17:01
akahat|roverWhat if we non-vote all zed jobs?17:01
rlandyTengu: I'd vote for https://review.opendev.org/c/openstack/tripleo-common/+/869240 if it works17:02
rlandysimpler17:02
Tenguerrr.17:03
Tenguthat one won't work17:03
Tenguit will fail due to the very issue bogdan is correcting in the patch that's following this comment-out.17:03
Tenguso I'll need to amend his patch and do the partial revert....17:03
TenguI hate that.17:03
Tenguok, signing off. Hopefully the CI will be green tomorrow so that Marios or anyone else may be able to merge it.17:19
akahat|roverrlandy, https://review.rdoproject.org/r/c/testproject/+/46525 -> we got one master job passing.. other failed on undercloud install.. rerunning 17:23
rlandyakahat|rover: thanks - let's try settle on the list of exclude and merge those up tomorrow morning17:38
rlandyTengu: sorry - meeting reading back17:38
rlandytomorrow is fine17:39
akahat|roverrlandy, may be we need some more green runs..17:50
akahat|roverin my morning i'll keep poking it.. and will get merge tomorrow.17:51
rlandythank you17:53
* dviroel out21:09
*** dviroel is now known as dviroel|ourt21:09
dasmdviroel|ourt: o/21:09
* dasm => offline22:14
dasmo/22:14
*** dasm is now known as dasm|off22:14
*** rlandy is now known as rlandy|out23:19

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!