Wednesday, 2020-10-28

*** Goneri has quit IRC01:03
*** dsneddon has quit IRC02:29
*** dsneddon has joined #oooq02:33
*** dsneddon has quit IRC02:38
*** dsneddon has joined #oooq02:48
*** dsneddon has quit IRC02:59
*** dsneddon has joined #oooq03:01
*** dsneddon has quit IRC03:06
*** dsneddon has joined #oooq03:07
*** dsneddon has quit IRC03:12
*** dsneddon has joined #oooq03:13
*** apetrich has quit IRC03:13
*** dsneddon has quit IRC03:19
*** udesale has joined #oooq03:25
*** dsneddon has joined #oooq04:17
*** ykarel has joined #oooq04:24
*** dsneddon has quit IRC04:41
*** dsneddon has joined #oooq04:41
chandankumarsshnaidm|afk: Hello, please have a look https://review.opendev.org/#/c/759985/ job freeze issue when around, thanks!04:50
*** dsneddon has quit IRC04:59
*** dsneddon has joined #oooq05:04
*** dsneddon has quit IRC05:09
*** dsneddon has joined #oooq05:26
*** ratailor has joined #oooq05:46
*** ratailor has quit IRC05:55
*** ratailor has joined #oooq05:58
*** saneax has joined #oooq06:08
*** ysandeep|away is now known as ysandeep|ruck06:08
*** ratailor_ has joined #oooq06:25
*** ratailor has quit IRC06:28
*** marios has joined #oooq06:54
*** dsneddon has quit IRC06:59
*** dsneddon has joined #oooq06:59
*** sshnaidm|afk is now known as sshnaidm|rover07:00
ysandeep|ruckakahat, hi , Have you posted a patch to switch tripleo-ci-centos-8-undercloud-upgrade-ussuri to non-voting for the time being? if not I will post it07:12
*** dsneddon has quit IRC07:18
*** dsneddon has joined #oooq07:18
chandankumarsshnaidm|rover: Hello07:22
sshnaidm|roverchandankumar, hey07:22
chandankumarsshnaidm|rover: need help on this review https://review.opendev.org/#/c/759985/07:22
chandankumarsshnaidm|rover: I already rebased it on https://review.opendev.org/#/c/759892/507:22
sshnaidm|roverchandankumar, yeah, looking07:22
chandankumarand also tried removed zuuld. reference but now hitting c7 content provide issue07:23
sshnaidm|roverchandankumar, why do you change old templates there?07:24
chandankumarsshnaidm|rover: by seeing akahat and rfoloc change I was confused, then I thought of using old template, making the changes there, then we donot we need to update the layout in branch project side07:26
chandankumarsshnaidm|rover: Do we need to use the new template?07:26
sshnaidm|roverchandankumar, it doesn't make sense to create two same templates07:26
chandankumarsshnaidm|rover: ok switching back to new one07:27
sshnaidm|roverchandankumar, I know it's more work, but it's safer to do controlled change07:27
chandankumarsshnaidm|rover: please have a look at https://review.opendev.org/#/c/759788/ and logs here https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_578/759849/3/check/tripleo-ci-centos-8-content-provider-train/578d681/job-output.txt 'dict object' has no attribute 'results'07:29
chandankumarsshnaidm|rover: I am not sure about the correct condition here https://review.opendev.org/#/c/759788/6/roles/container-build/tasks/non_tripleo_containers.yml@5807:30
ykarelsshnaidm|rover, isn't async task did not complete within the requested time - 5700s fixed in content provider jobs?07:40
ykareli still see in some patches07:40
ykarelalso noticed train/ussuri content provider jobs running in master patches, is that intentionally being run?07:41
sshnaidm|roverykarel, not fixed yet07:41
ykarellike in https://review.opendev.org/#/c/759772/, https://review.opendev.org/#/c/759771/07:41
sshnaidm|roverykarel, they have to if there are upgrades jobs in template07:41
ykarelsshnaidm|rover, but upgrade runs from N-1 to N release07:42
sshnaidm|roverykarel, yeah, train job seems redundant there, need to optimize07:42
ykarelsshnaidm|rover, okk07:42
ykarelnot sure if how provider/consumer helps in upgrade jobs07:43
sshnaidm|roverykarel, what do you mean?07:43
sshnaidm|roverthe goal is not to download from docker.io07:44
ykarelsshnaidm|rover, for multiple jobs it should help, but if there is single upgrade job running a content provider just for it07:44
*** ykarel has quit IRC07:44
*** ykarel has joined #oooq07:45
* ykarel got disconnected07:45
sshnaidm|roverykarel, yeah, we thought about building in the job itself, but it's very time consuming to build containers twice and do upgrade07:45
akahatysandeep|ruck, hey.. yes it's because of the duplicate jobs in the template and patches are up for it.07:45
sshnaidm|roverwe would hit timeouts07:45
akahathope so it will resolve the issue.07:46
ysandeep|ruckakahat, ack thanks!07:46
ykarelsshnaidm|rover, i meant build once for N-1 and for N release fetch from provider, let's see how it goes. after async timeout issue is fixed it should get better07:47
ykarelthat can be follow once we have some stats on current approach07:48
sshnaidm|roverykarel, yeah, but two parallel jobs would be faster and no need to define a special job for it, just reusing existing07:49
akahatchandankumar, sshnaidm|rover +w needed: https://review.opendev.org/#/c/759867/07:50
ykarelsshnaidm|rover, ack agree on parallel jobs, the less the jobs, more better for gate in case of inconsistent failures in jobs, once these failures are resolved results will look better :)07:51
sshnaidm|roverykarel, that's right07:53
*** amoralej|off is now known as amoralej08:11
*** yolanda has joined #oooq08:19
*** frenzy_friday has joined #oooq08:20
Tengusshnaidm|rover: heya! patch merged for https://bugs.launchpad.net/bugs/190167608:25
openstackLaunchpad bug 1901676 in tripleo "Undercloud install is failing with Error message: Validation ['r', 'o', 'p', 'd', '.', 'i', 'y', 'k', 'c', 'e', 'l', 's', '-', 'n', 'u', 'a', 'm'] not found in /usr/share/ansible/validation-playbooks." [Critical,Fix released] - Assigned to Cédric Jeanneret (cjeanner)08:25
Tengupromotion should be fine now.08:25
sshnaidm|roverTengu, great, thanks08:30
*** ykarel_ has joined #oooq08:49
*** ykarel has quit IRC08:52
*** jpena|off is now known as jpena08:54
*** tosky has joined #oooq08:57
*** ysandeep|ruck is now known as ysandeep|lunch09:00
*** jfrancoa has joined #oooq09:05
*** openstack has quit IRC09:21
*** openstack has joined #oooq09:22
*** ChanServ sets mode: +o openstack09:22
*** jfrancoa has quit IRC09:27
*** ysandeep|lunch is now known as ysandeep|ruck09:45
*** apetrich has joined #oooq09:47
*** ykarel_ is now known as ykarel09:51
chandankumarpojadhav: zbr arxcruz bhagyashris https://review.opendev.org/#/q/topic:new-ci-job+status:open+branch:stable/queens please have a look when free, thanks!10:08
chandankumarsshnaidm|rover: Do we want to move these jobs also to https://opendev.org/openstack/tripleo-common/src/branch/stable/queens/zuul.d/layout.yaml#L13 content provider?10:09
zbrwtf does SUCCESS in 8h 41m 25s means....10:11
chandankumarzbr: which job is it?10:13
*** marios has joined #oooq10:14
arxcruzmarios: i forgot to post the testproject, i'll try today, working on the quay push containers10:15
arxcruzsorry, too many info at same time10:15
mariosack thanks arxcruz10:16
arxcruzchandankumar: marios what's the job that shows the list of containers build for rocky, queens and stein ?10:21
arxcruzI got periodic-tripleo-ci-build-containers-ubi-8-push-{ussuri, victoria, train}10:22
chandankumararxcruz: getting the link10:22
arxcruzbut i don't see rocky, queens and stein jobs10:22
sshnaidm|roverchandankumar, I think we can not to hurry with queens for now10:23
chandankumararxcruz: periodic-tripleo-centos-7-stein-containers-build-push10:24
sshnaidm|roverchandankumar, let's polish the master first10:24
chandankumararxcruz: periodic-tripleo-centos-7-rocky-containers-build-push10:24
chandankumararxcruz: https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable4-510:24
chandankumararxcruz: periodic-tripleo-centos-7-queens-containers-build  -> https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable6 hope it helps10:25
chandankumarsshnaidm|rover: ok, sure10:26
arxcruzchandankumar: thanks, i was clicking in the wrong link to check the logs lol10:27
zbrchandankumar:  i seen it on https://review.opendev.org/#/c/759892/510:27
*** ysandeep|ruck is now known as ysandeep|brb10:31
arxcruzokay... queens is a problem since it's using kolla and not generating the list of containers...10:32
chandankumararxcruz: take a look at content provider job10:33
chandankumarqueens one10:33
pojadhavchandankumar, ack !10:36
*** ysandeep|brb is now known as ysandeep|ruck10:54
*** slaweq has joined #oooq10:57
marioschandankumar: arxcruz: fyi remote:   https://review.opendev.org/760106 DNM testing testing is this this on tripleoclient victoria11:02
chandankumarmarios: ack!11:09
chandankumarsshnaidm|rover: can we update this patch https://review.opendev.org/#/c/759884/ it is blocking c7 content provider job when gating repo is used11:14
*** udesale_ has joined #oooq11:26
*** udesale has quit IRC11:29
*** dmellado has quit IRC11:38
*** dmellado has joined #oooq11:39
*** udesale_ has quit IRC11:45
*** ykarel has quit IRC11:47
*** udesale has joined #oooq11:48
*** ykarel has joined #oooq11:49
*** rlandy has joined #oooq12:07
*** ykarel_ has joined #oooq12:16
*** pojadhav is now known as pojadhav|afk12:24
chandankumararxcruz: does rocky and stein containers pushed on quay.io?12:24
rlandychandankumar: ^^ yep ... when testing stein landed up with unauthorized12:25
*** jpena is now known as jpena|lunch12:25
rlandywhich could mean the containers don't exist12:25
chandankumarah12:25
rlandychandankumar: or we need auth to pull - which would be unusual12:25
rlandydepends how tenant is set up12:26
chandankumarrlandy: https://quay.io/organization/tripleorocky containers not there12:26
arxcruzchandankumar: stein is being pushed right now12:26
arxcruzhttps://quay.io/organization/tripleostein12:26
arxcruzrocky is the next12:26
chandankumararxcruz: ok then rechecking the rlandy patch12:26
rlandychandankumar: arxcruz: ok - so I'll try the test again12:26
arxcruzstein is 18/104 containers12:27
rlandychandankumar: let's wait until all the stein containers are there12:27
arxcruzchandankumar: wait, stein still being pushed12:27
rlandyotherwise we'll hit another error12:27
chandankumararxcruz: sure,12:27
rlandychandankumar: any other task you need me to pick up?12:27
* chandankumar waits for arxcruz's signal12:27
chandankumarrlandy: akahat and I had taken care of queens12:27
chandankumarrlandy: we need one job in rdo side which will push the ceph related containers to rdo registry12:28
rlandychandankumar: k - checking other repos then for stein/rocky12:28
rlandychandankumar: how often?12:28
chandankumarrlandy: let me create a card drop all the stuff there12:28
rlandywhen ceph changes>12:28
chandankumarrlandy: weekky would be fine12:29
chandankumar* weekend would be fine12:29
chandankumarakahat: ping12:29
akahatchandankumar, pong12:30
chandankumarakahat: any branch left for porting?12:30
akahatchandankumar, ussuri/queens/master done.12:30
chandankumarakahat: c7 train done?12:30
chandankumarlet me check rfolco patches12:31
akahatchandankumar, rfolco is taken care of it.12:31
rlandychandankumar: k .. will need to speak with weshay|ruck about what we want to do with repos and stable/stein branches12:31
rlandylike: https://github.com/openstack/tripleo-heat-templates/blob/stable/stein/zuul.d/layout.yaml12:31
rlandyand stable/rocky12:32
rlandywe removed the tests from running on those branches12:32
rlandybut the templates still exist12:32
chandankumarrlandy: yes12:32
akahatchandankumar, sshnaidm|rover marios rfolco weshay|ruck Anyone joining content provider meeting?12:32
*** rfolco has joined #oooq12:34
mariosakahat: skipping today still have day 3 of ptg12:35
arxcruzchandankumar: rlandy there are 104 containers being pushed, those are containers that were successfully built, not sire if this is the total amount though12:35
arxcruzchandankumar: rlandy if some container is missing let me know12:35
mariosakahat: well, i am skipping i don't know if the meeting is skipped ...12:35
mariosto be clear12:35
akahatmarios, okay. np12:36
rlandyarxcruz: sounds about the right number12:37
chandankumarrlandy: sshnaidm|rover weshay|ruck Validate ovb jobs on psi public - https://projects.engineering.redhat.com/browse/RHOSZUUL-27412:37
arxcruzrlandy: chandankumar 48 of 10412:37
chandankumarI am just sharing the link12:38
chandankumarnicholas is working on that12:38
rlandychandankumar: ack sshnaidm|rover and I have been working with nicholas on that12:38
chandankumarcool, thanks :-)12:39
weshay|ruckmarios, need your review https://review.opendev.org/#/c/759912/12:43
weshay|ruckakahat, https://review.opendev.org/#/c/759912/12:44
chandankumarrlandy: https://projects.engineering.redhat.com/browse/TRIPLEOCI-24812:49
weshay|ruckakahat, https://review.opendev.org/#/c/759912/12:50
rlandychandankumar: ok - will look a that after PTG12:50
rlandychandankumar: fyi ... for after we merge the rocky/stein check changes https://review.rdoproject.org/r/#/c/30489/12:50
rlandyhmmm ... has europe changes the clocks? platform meeting is an hour later12:51
ysandeep|ruckrlandy, yes day light saving12:52
rlandyk - we're still an hour out in the US12:52
chandankumarrlandy: akahat please take care of these patches https://review.opendev.org/#/q/topic:new-ci-job+status:open+owner:%22Chandan+Kumar+(raukadah)+%253Cchkumar%2540redhat.com%253E%22 from tomorrow12:53
weshay|ruckchandankumar, rlandy sshnaidm|rover DO NOT WORK ON PSI PUBLIC this week12:53
chandankumarweshay|ruck: yup12:54
weshay|ruckchandankumar, I've had a chat w/ nhicher and he's ok w/ delaying that effort until we're out of docker.io and ptg fun12:54
chandankumargreat, thanks :-)12:55
weshay|rucknot all things can be done at the same time :)12:55
akahatweshay|ruck, updated with depends on: https://review.opendev.org/#/c/757836/12:55
weshay|ruckthanks12:55
akahatchandankumar, I will charge for it.12:56
weshay|ruckchandankumar, sshnaidm|rover think we're missing a feature in content-provider for upgrade jobs12:56
chandankumarrfolco: for fixing c7 content provider, we need this patch https://review.opendev.org/#/c/759884/12:56
chandankumarweshay|ruck: what is that?12:56
rfolcochandankumar, ok will add a depends-on12:57
weshay|ruckchandankumar, build-test-packages needs to be run twice on the content-provider... and provide a repo for the deployment, and one for the upgrade12:57
rfolcoto my 10000000 patches12:57
arxcruzchandankumar: rlandy stein containers pushed12:57
arxcruzstarting now rocky12:57
rfolco:(12:57
weshay|ruckarxcruz, nice12:57
chandankumarweshay|ruck: ok12:57
*** pojadhav|afk is now known as pojadhav12:58
chandankumarweshay|ruck: rlandy https://review.opendev.org/#/c/760078/ mass deletion12:58
ysandeep|ruckrlandy, weshay|ruck I was preparing for Cix call, But i have less background about https://bugs.launchpad.net/tripleo/+bug/190015812:58
openstackLaunchpad bug 1900158 in tripleo "ci.centos jobs in master and ussuri are failing introspection - failed to run hardware-detect utility" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy)12:58
chandankumarit is blocking patches related to quay.io12:58
mariosPTG day 3 starting in 3 mins :D \o/12:59
ysandeep|rucki was trying to check logs but - https://ci.centos.org/job/tripleo-quickstart-promote-ussuri-current-tripleo-delorean-minimal/ didn't triggered after 20th12:59
sshnaidm|roverweshay|ruck, it's what provider does12:59
ysandeep|ruckrlandy, if you know status of this, or what the next step we should check- that would be very helpful13:00
rlandyysandeep|ruck: there was a launchpad created13:01
rlandywith a link to a merged change by rpittau.13:02
rlandypls check if you have his change includde13:02
rlandyhe was very helpful13:02
rlandyfeel free to ping him with questions13:02
rlandyysandeep|ruck: do you see the LP bg  otherwise will search for it13:03
ysandeep|ruckrlandy, this one right https://bugs.launchpad.net/tripleo/+bug/190015813:03
openstackLaunchpad bug 1900158 in tripleo "ci.centos jobs in master and ussuri are failing introspection - failed to run hardware-detect utility" [Critical,Triaged] - Assigned to Ronelle Landy (rlandy)13:03
weshay|ruckrlandy, join ptg13:03
weshay|ruckwe're talking about ansible 2.1013:03
rlandyweshay|ruck: we switched rooms, right?13:04
*** amoralej is now known as amoralej|lunch13:07
arxcruzi'll brb +- 1hr13:08
*** TrevorV has joined #oooq13:16
rfolcochandankumar, https://review.opendev.org/#/c/759884/ is w-1, so c7 content provider patches are all BLOCKED13:18
rfolcoakahat, fyi ^13:18
*** ratailor_ has quit IRC13:23
*** sshnaidm|rover has quit IRC13:25
*** jpena|lunch is now known as jpena13:27
*** ykarel_ has quit IRC13:31
*** sshnaidm|rover has joined #oooq13:39
*** sshnaidm|rover is now known as sshnaidm|mtg13:44
weshay|rucksshnaidm|mtg, /me thinks ur right.. if upgrade always has two content providers.. we're good13:59
weshay|ruckand it's just the tags.. we need to fix..14:00
weshay|ruckjust going to loop marios into the context14:00
*** amoralej|lunch is now known as amoralej14:02
mjturekrfolco: could you help me figure out why buildah isn't getting used here https://logserver.rdoproject.org/ci.centos.org/tripleo-upstream-containers-build-master-ppc64le/2798/logs/logs/build.log14:03
mjturekwe define the use_buildah var here https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/tripleo-upstream-builders.yml#L5114:05
mjturekmaybe there's a better way to do that? Like pass the var here https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/tripleo-upstream-builders.yml#L210 ?14:06
rlandychandankumar: what do you think about the comment on https://code.engineering.redhat.com/gerrit/#/c/215917/29/zuul.d/rhos-release.yaml?14:08
arxcruzchandankumar: rlandy tripleo rocky containers pushed14:09
rlandyshould I change the name of the rhos-release container build job14:09
rlandyarxcruz: k - let me recheck14:09
chandankumarchecking14:09
chandankumarrlandy: rename it to rhos-release-rhel-8-rpmbuild14:12
rlandychandankumar: ack - ok14:13
rlandyarxcruz: stein containers are complete as well?14:13
chandankumararxcruz: cool, thanks, rechecking the patch14:14
rlandychandankumar: rocky patch https://review.opendev.org/#/c/760147/14:14
arxcruzrlandy: yes14:15
rlandyarxcruz: perfect - thanks - both under test14:15
rfolcomjturek, sure, give me a sec to take a look14:16
arxcruzrlandy: let me know if some container is missing, as I said, i get the list from the build jobs not sure if some is missing or not14:16
rlandyk14:16
mjturekthanks rfolco :)14:16
*** holser has quit IRC14:20
*** holser has joined #oooq14:20
ysandeep|ruckrlandy, when you have time, introspection is still failing :( https://artifacts.ci.centos.org/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-826/console.log , Where we need to check if https://github.com/redhat-cip/hardware/pull/150 is included?14:24
rfolcomjturek, question, do you still build w/ kolla ?14:25
mjturekrfolco nope we're trying to build with buildah14:26
mjturekthe use_buildah var had been enough for that in the past https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/tripleo-upstream-builders.yml#L5114:26
rfolcomjturek, different level of tools14:26
rfolcoolder branches in tripleo ci use kolla -> buildah, newer like master use ticb -> buildah14:27
mjturekaaaaaaaaaah14:27
rfolcotcib (tripleo container build) is the new build method that replaces kolla, thats why I ask14:28
rfolcohttps://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/build-containers.yaml#L201-L21014:28
rfolcouses ubi8 as base container and builds on top of that14:28
rfolcomjturek, in your current job, I'd suggest to switch use_buildah: true14:29
rfolcomjturek, without ""14:29
mjturekwill do rfolco14:29
rfolcomjturek, but I recommend switching to tcib14:29
mjturekalright will do, simple as use_kolla: false?14:29
rfolcomjturek, try that, not sure if anything hidden in the base/parent job definitions14:30
mjturekfair enough rfolco14:30
mjturekthanks!14:30
rfolcoyw mjturek let me know if any questions14:31
mjturekwill do rfolco14:31
rlandyysandeep|ruck: we need to check the ipa image14:31
rlandyto see if that is includde properly14:31
rlandygetting14:31
ysandeep|ruckweshay|ruck, sshnaidm|mtg fyi.. i joined cix but it got cancelled , some folks were out due to holiday in brno14:33
sshnaidm|mtgysandeep|ruck, yeah, it should be canceled today14:33
weshay|ruckysandeep|ruck, ya.. thank you14:33
rlandyysandeep|ruck: following up on #tripleo14:35
ysandeep|ruckrlandy, thanks14:35
rlandyOMG - the check queue is huge14:37
*** sanjayu_ has joined #oooq14:40
*** saneax has quit IRC14:42
*** Goneri has joined #oooq14:48
weshay|rucksshnaidm|mtg, fyi http://dashboard-ci.tripleo.org/d/Z4vLSmOGk/cockpit?orgId=1&fullscreen&panelId=23114:49
weshay|ruckysandeep|ruck, ^14:49
weshay|ruckthat's rdocloud not vexx btw14:50
*** Goneri has quit IRC14:52
*** ykarel is now known as ykarel|away15:07
sshnaidm|mtgweshay|ruck, all these stacks are in DELETE_FAILED because some glitches on rdo-cloud, our scripts can't remove them15:16
sshnaidm|mtgweshay|ruck, unfortunately I can't get help from rdo admins as well15:16
sshnaidm|mtgweshay|ruck, should I submit another ticket?15:16
weshay|ruckk.. so it goes..15:16
sshnaidm|mtgnot sure if someone is looking at them at all15:17
weshay|rucksshnaidm|mtg, I'll ping alan and kforde15:17
sshnaidm|mtgweshay|ruck, apevec is not online as I see15:17
weshay|rucksshnaidm|mtg, just to confirm.. you reran the scripts.. and we've deleted everything we can15:17
sshnaidm|mtgweshay|ruck, yeah, they run automatically15:17
weshay|ruckI know they run automatically..15:17
weshay|ruckI'll jump on toolbox and confirm before I esacalate15:18
weshay|ruckthanks15:18
sshnaidm|mtgweshay|ruck, http://paste.openstack.org/show/799478/15:18
sshnaidm|mtgweshay|ruck, I'm already there15:18
sshnaidm|mtg# ls -l /var/log/clean_stacks_rdo.log15:19
sshnaidm|mtg-rw-------. 1 root root 366930 Oct 28 15:10 /var/log/clean_stacks_rdo.log15:19
sshnaidm|mtgin /var/log/clean_stacks_rdo.log you can see the last log15:19
sshnaidm|mtg ls -l /var/log/clean_failed_stacks.log15:20
sshnaidm|mtg-rw-------. 1 root root 17993 Oct 28 14:43 /var/log/clean_failed_stacks.log15:20
rlandymigi: chandankumar: wrt https://code.engineering.redhat.com/gerrit/#/c/215917/29/zuul.d/rhos-release.yaml renaming rhos-release job, I think I will need to do this in another step15:23
rlandyper the depends-on  - which is the other way around15:23
* rlandy puts in another patch to rename rhos-release job15:24
chandankumarrlandy: ack15:24
rlandyysandeep|ruck: pls review https://code.engineering.redhat.com/gerrit/#/c/215942/15:26
arxcruzchandankumar: https://review.opendev.org/#/c/759913/ i don't see any rocky or stein job consuming the containers15:27
ysandeep|ruckrlandy, will do , i already checked previous patchset, i am okay with the approach15:29
chandankumararxcruz: https://review.opendev.org/#/c/760147/15:39
arxcruzchandankumar: ack15:40
chandankumarrlandy: https://review.opendev.org/#/c/759793/ +w it15:41
weshay|rucksshnaidm|mtg, you have an open ticket?15:41
sshnaidm|mtgweshay|ruck, on my way15:41
rlandyok15:41
*** ykarel|away has quit IRC15:41
weshay|rucksshnaidm|mtg, k.. cool paste me the ticket # and I'll email all the folks15:41
weshay|rucksshnaidm|mtg, thanks15:41
sshnaidm|mtgweshay|ruck, RITM078002915:44
weshay|rucksshnaidm|mtg, thanks15:45
chandankumarrlandy: thanks!15:48
rfolcoany known issues with openstack-tox-molecule ?15:59
rfolcohttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a9e/759429/3/check/openstack-tox-molecule/a9e06de/job-output.txt15:59
rfolcofailing to open config/release/newton.yml16:00
weshay|ruckrfolco, that test should be updated to remove newton16:01
weshay|ruckya.. that the images16:01
rfolcohmm ok, https://images.rdoproject.org/newton/rdo_trunk/current-tripleo/overcloud-full.tar16:01
weshay|ruckon the rdo project16:01
rfolco40416:01
weshay|ruckrfolco, ya.. I nuked it16:02
weshay|ruckit's the wrong dir anyway16:02
weshay|ruckchange the test16:02
rfolcook16:02
weshay|ruckonly test supported releases16:02
weshay|ruckrfolco, thanks16:02
rfolcoweshay|ruck, thank you16:02
chandankumarrfolco: my friend16:03
chandankumarrfolco: https://review.opendev.org/760078 will fix molecule issue16:03
rfolcook great chandankumar16:04
rfolcothanks for fixing it16:04
rfolcoI'll just rebase16:04
chandankumarrfolco: :-)16:05
arxcruzsshnaidm|mtg: chandankumar weshay|ruck shall we increase the timeout for the container-build job? i'm getting a lot of timeouts, and rechecks wouldn't work16:05
rfolcochandankumar, what about c7 content provider fix ?16:05
weshay|ruckarxcruz, ya.. we're seeing the same.. I think so16:05
rfolcochandankumar, https://review.opendev.org/#/c/759884/ is blocking all c7 patches16:06
rfolcosshnaidm|mtg, ^16:06
chandankumarsshnaidm|mtg: https://review.opendev.org/#/c/759884/ please update this patch to unblock c7 content provider16:06
rfolcoakahat, please do not recheck c7 provider patches until this isn't ready ^ (add depends-on)16:07
rfolcoits waste of time, rechecks will fail for c7 content provider16:07
weshay|ruckFYI... if container builds fail w/ manifest not found https://github.com/containers/buildah/issues/272716:13
*** migi is now known as migi_pto16:14
weshay|ruckrdo zuul may be down16:14
ysandeep|ruckweshay|ruck, sshnaidm|mtg moving ussuri to vexx helped, ussuri promoted few mins ago16:14
weshay|ruck++16:15
arxcruzweshay|ruck: https://review.opendev.org/#/c/760177/16:19
arxcruzincreasing timeout from 95min to 12016:19
chandankumarysandeep|ruck: sshnaidm|mtg: marios: zbr arxcruz weshay|ruck rlandy rfolco akahat jpena I will be on pto on tomorrow will back by next wed, see ya :-)16:22
jpenaenjoy your pto chandankumar !16:23
mariosenjoy chandankumar16:23
arxcruzchandankumar: I'll miss you16:23
*** chandankumar is now known as raukadah16:23
arxcruzsafe travel16:23
raukadahthanks all :-)16:23
ysandeep|ruckchandankumar, safe journey16:23
rlandyraukadah: safe travels16:23
weshay|ruck0/16:24
*** ysandeep|ruck is now known as ysandeep|away16:29
rfolcoI don't think sshnaidm|mtg patch would solve c7 content provider issue https://85243a2a53e02e239382-cf02a1947fec3ea7a230e93d6a5b1fec.ssl.cf5.rackcdn.com/759736/1/check/tripleo-ci-centos-7-content-provider/013c03e/logs/undercloud/home/zuul/workspace/build.log16:35
rfolcobrb16:38
mariosweshay|ruck: fyi when you have 2 mins please check https://review.opendev.org/#/c/760113/16:55
*** amoralej is now known as amoralej|off17:06
*** dtantsur is now known as dtantsur|afk17:10
*** tosky has quit IRC17:20
*** marios is now known as marios|out17:25
*** udesale has quit IRC17:34
*** sshnaidm|mtg is now known as sshnaidm|rover17:46
*** ykarel|away has joined #oooq17:47
rfolcoweshay|ruck, c7 provider should use kolla for queens, tcib for train ?17:48
*** marios|out has quit IRC17:53
*** sanjayu_ has quit IRC17:57
*** ykarel|away has quit IRC17:59
*** jpena is now known as jpena|off18:05
weshay|ruckrfolco, there is no tcip c7 for train18:07
weshay|ruckc8 train and up18:07
weshay|ruckfor tcip18:07
weshay|ruckkolla for c7 train18:07
weshay|ruckand queens18:07
rfolcoyeah figured that out...18:07
rfolcothe problem with content provider c7 is that both are not using buildah18:08
rfolcoqueens and train18:08
rfolcoso I am proposing this https://review.opendev.org/76019818:08
rfolcoweshay|ruck, we set use_buildah only in branchful https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/undercloud-jobs.yaml#L196-L23018:09
rfolcoweshay|ruck, so the default is false, https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers/defaults/main.yaml#L1318:09
weshay|ruckbuildah is ok18:09
weshay|ruckfor c7 train18:09
rfolcoc7 train fails here https://85243a2a53e02e239382-cf02a1947fec3ea7a230e93d6a5b1fec.ssl.cf5.rackcdn.com/759736/1/check/tripleo-ci-centos-7-content-provider/013c03e/logs/undercloud/home/zuul/workspace/build.log18:09
rfolcothe only clear indication for me is buildah, I might be wrong18:10
rfolcolack of buildah18:10
*** jbadiapa has quit IRC18:37
*** dsneddon has quit IRC18:39
*** dsneddon has joined #oooq18:40
sshnaidm|roverrfolco, file:///etc/yum.repos.d/gating.repo, line: 118:42
sshnaidm|roverrfolco, it means gating.repo is not there18:42
sshnaidm|roverwhen we copy it need to ensure we have it18:42
rfolcosshnaidm|rover, the build fails for all c7 content provider18:42
sshnaidm|roverrfolco, logs?18:43
rfolcosshnaidm|rover, https://zuul.opendev.org/t/openstack/build/e13aaf40c5cf49518a6a01a9a28cf4c718:43
rfolcosshnaidm|rover, I suspect its the use_buildah flag missing18:43
rfolcowe only added that to the branchful job, so I am proposing this https://review.opendev.org/#/c/760198/18:44
rfolcosshnaidm|rover, let me know if this makes sense ^18:44
rfolcoI've seen c7 provider failing in all my c7 train patches18:45
sshnaidm|roverrfolco, I see the same problem, seems for me it's gating.repo issue18:48
sshnaidm|roverrfolco, it worked till today, right?18:49
rfolcosshnaidm|rover, I can't tell it ever worked18:49
rfolcosshnaidm|rover, not sure, I don't think so18:49
*** sshnaidm_ has joined #oooq18:57
*** sshnaidm|rover has quit IRC19:00
*** sshnaidm_ is now known as sshnaidm|rover19:06
rlandyarxcruz: I think we have an issue with the rocky/stein containers ...19:28
rlandy/quay.io/tripleostein/centos-binary-cinder-api:28900aa1633e91bfbbd1c821f32cb470f8835ace_8d9f186119:28
*** sshnaidm|rover is now known as sshnaidm|afk19:28
rlandyhttps://quay.io/repository/tripleostein/centos-binary-cinder-api?tab=tags19:28
rlandythe only tag that exists is current-tripleo19:28
rlandylooking for the hash19:29
arxcruzrlandy: yes, only current-tripleo19:29
rlandyhttps://f9ec75f4b8cfafc048e0-c67b93b429db84769e417a4684ceffa6.ssl.cf5.rackcdn.com/759926/1/check/tripleo-ci-centos-7-standalone/fce0dd7/logs/undercloud/home/zuul/standalone_deploy.log19:29
arxcruzweshay|ruck: was to push only current-tripleo right ?19:29
rlandyarxcruz: hmmm the test looks for the hash19:29
* rlandy looks to see if we can use the human readable form19:30
rlandylooks like it's resolving the hash19:30
rlandy            tag: 28900aa1633e91bfbbd1c821f32cb470f8835ace_8d9f186119:31
rlandyhttps://f9ec75f4b8cfafc048e0-c67b93b429db84769e417a4684ceffa6.ssl.cf5.rackcdn.com/759926/1/check/tripleo-ci-centos-7-standalone/fce0dd7/logs/undercloud/home/zuul/containers-prepare-parameters.yaml19:32
arxcruzrlandy: I can push the tag as well19:32
arxcruzjust need to change the code19:32
rlandyarxcruz: let me see if I can adjust the tag it's looking for19:32
arxcruzrlandy: i got the list of containers from this https://logserver.rdoproject.org/openstack-periodic-integration-stable4-5/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-stein-containers-build-push/a8af5ee/logs/containers-successfully-built.log19:33
arxcruzand the tak looks the right one19:33
rlandyarxcruz: ack - it's the correct hash19:33
arxcruzs/tak/tag19:33
rlandybut it's push as current-tripleo19:33
rlandyas the tag rather than the hash itself19:33
rlandyhttps://quay.io/repository/tripleostein/centos-binary-cinder-api?tab=tags19:33
rlandyyou've definitely pushed the correct one but it's not the tag it's looking for19:34
arxcruzrlandy: let me check the quay api, otherwise it will be hard to push all this tags19:35
rlandydocker_image_tag: "{{ dlrn_hash|default(dlrn_hash_tag) }}"19:35
rlandyarxcruz: we don;t need them all19:35
rlandywe never go backwards19:35
weshay|ruckarxcruz, correct19:35
rlandybut we would need to push if there was an update19:35
rlandyie: a  promotion19:35
rlandydocker_image_tag: "{{ dlrn_hash|default(dlrn_hash_tag) }}"19:35
rlandydlrn_hash  resolving that19:36
rlandyhttps://f9ec75f4b8cfafc048e0-c67b93b429db84769e417a4684ceffa6.ssl.cf5.rackcdn.com/759926/1/check/tripleo-ci-centos-7-standalone/fce0dd7/logs/undercloud/var/log/extra/dump_variables_hostvars.yaml19:37
rlandyget sresolved19:37
rlandyweshay|ruck: arxcruz: we can fix the problem by changing https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-7/stein.yml#L2419:37
rlandyor pushing the containers with the tag = dlrn_hash19:37
weshay|ruckarxcruz, OH.. you didn't tag them w/ the hash?19:37
rlandyarxcruz: weshay|ruck: choice on the above?19:38
rlandyweshay|ruck: they are tagged current_tripleo19:38
weshay|ruck:(19:38
rlandycurrent-tripleo19:38
rlandyhttps://quay.io/repository/tripleostein/centos-binary-cinder-api?tab=tags19:38
weshay|ruckshould be tagged w/ both..19:38
weshay|ruckarxcruz, ^19:38
arxcruzrlandy: weshay|ruck i'm checking how to tag, give me a few minutes19:38
weshay|ruckarxcruz, ya.. w/ buildah it's pretty easy19:38
weshay|ruckand quick19:38
weshay|ruckto retag19:39
weshay|ruckarxcruz, ya.. so production containers.. need to be tagged w/ both the hash and the human readable19:39
rlandyk ... waiting19:39
rlandywill recheck when you give me the go ahead19:40
weshay|ruckwhen we promote...  we repush w/ dlrn hash tag.. and retag the latest w/ current-tripleo.. so you still have the old dlrn hash tag..19:40
weshay|ruckarxcruz, that allows us not to break any running job19:40
arxcruzweshay|ruck: ack, working on it19:40
weshay|ruckevery job.. uses the dlrn-hash tag..  humans use current-tripleo19:40
rlandyI think at this point human read dlnr hashes as well :)19:41
*** dsneddon has quit IRC19:43
*** dsneddon has joined #oooq19:45
*** dsneddon has quit IRC19:50
*** dsneddon has joined #oooq19:57
*** dsneddon has quit IRC20:02
*** dsneddon has joined #oooq20:03
rlandyweshay|ruck: hmmm .. ok we've taken out too much ...20:18
rlandyopenstack-tox-pep8queued20:19
rlandyopenstack-tox-lower-constraintsqueued20:19
rlandytripleo-ci-centos-7-scenario000-multinode-oooq-container-updates20:19
rlandy^^ only those three jobs triggered20:19
rlandyfrom https://review.opendev.org/76022120:19
weshay|ruckrlandy, what review did you trigger w/?20:19
* weshay|ruck looks20:19
rlandyI should get the standalone upgrade20:19
rlandyI edited the ci file20:19
* rlandy fixes20:19
weshay|ruckwait.. ya20:20
rlandyremoved the templates as well20:22
weshay|ruckrlandy, add.. tripleo-multinode-container-minimal20:23
weshay|ruckto the templates section20:23
*** yolanda has quit IRC20:24
*** yolanda has joined #oooq20:25
rlandyok20:30
weshay|ruckrlandy, ya.. afaict.. it should trigger on anything not included here https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/multinode-jobs.yaml#L16-L4220:32
rlandyhttps://zuul.openstack.org/status hasn't updated yet20:33
rlandystill has patch 2 listed20:34
weshay|ruckrlandy, hrm20:34
weshay|ruckya20:34
rlandyweird20:34
rlandywhy is it still running patchset 2?20:35
weshay|ruckoh ya20:35
weshay|rucknice eyes20:35
rlandywell done20:35
weshay|ruckheh20:35
rlandynow it's on there20:35
rlandythree20:35
weshay|ruckjust flipped20:35
rlandyslooooowwww20:35
weshay|rucksrsly20:35
rlandycurse of our generation - we expect immediate results20:36
weshay|ruckrlandy, oh sush20:37
weshay|ruckrlandy, but wtf.. standalone ?20:38
weshay|ruckI guess for stein that's fine.. but I don't yet see the why20:38
rlandyweshay|ruck: eeekk .. that should still trigger on stein20:38
rlandywell unless we removed stein from thatjob20:38
rlandychecking depends-on20:38
weshay|ruckstein is right.. I think that's fine20:39
*** tosky has joined #oooq20:39
weshay|rucktripleo-ci-centos-7-undercloud-containersqueued20:39
weshay|rucktripleo-ci-centos-7-scenario000-multinode-oooq-container-updateswaiting20:39
weshay|rucktripleo-ci-centos-7-standalonequeued20:39
weshay|rucktripleo-ci-centos-7-standalone-upgrade-steinqueued (non-voting)20:39
weshay|ruckopenstack-tox-pep8queued20:39
weshay|ruckopenstack-tox-lower-constraints20:39
weshay|ruckjust not sure why undercloud-containers20:39
rlandyit's right20:39
rlandyI think that's a reasonable set20:40
rlandynow rocky ...20:40
weshay|ruckbranches: ^(stable/(rocky|stein)).*$20:40
weshay|ruckhttps://review.opendev.org/#/c/757643/16/zuul.d/undercloud-jobs.yaml20:40
weshay|ruckline 11520:40
weshay|ruckcan remove that job20:41
weshay|ruckwell.. for stein20:41
weshay|ruckrocky still needs it20:41
weshay|ruckrlandy, and this https://review.opendev.org/#/c/759446/20:42
weshay|ruckrlandy, you may want to rebase your tripleo-ci patches on top of alex's patch20:43
rlandyok20:44
weshay|ruckrlandy, so we should see... undercloud-containers go away.. and standalone-upgrade go away20:45
rlandyweshay|ruck: I thought we wanted standalone-upgrade20:46
weshay|ruckwe should see.. tox-*, centos-7-standalone and tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates left20:46
rlandyfor stein20:46
weshay|ruckwe said that a while ago, but alex wants it to go bye bye20:46
weshay|ruckso..20:46
weshay|ruckhe wins20:46
rlandylol - ... rebasing20:46
weshay|ruckthanks20:46
weshay|ruckrlandy++20:58
weshay|ruckrlandy, so rocky would be the same.. except for no standalone.. so just containers-multinode and containers-multinode-updates20:59
rlandyweshay|ruck: k - on that patch21:02
rlandyat least now we know the model for non-ci repos21:02
rlandyI like quay.io - it's kind of slick looking21:03
rlandyclean and modern21:04
*** rfolco has quit IRC21:06
rlandyweshay|ruck: https://review.opendev.org/#/c/760244/ - pls check zuul if this is the correct triggering21:19
rlandyif so, will remove the DNM and the test change21:19
holserfolks can anyone explain why https://zuul.opendev.org/t/openstack/build/9e0aeac259784e6880d9c9e5309dd5a1 failed21:22
holserI am testing tripleo-common, but failure is in client21:22
weshay|ruckrlandy, k21:23
weshay|ruckrlandy, we need to get someone on https://opendev.org/openstack/puppet-openstack-integration/src/branch/master/zuul.d/tripleo.yaml21:26
weshay|ruckthat is the parent job for all the puppet repos21:26
weshay|ruckhttps://opendev.org/openstack/puppet-openstack-integration/src/branch/master/zuul.d/tripleo.yaml#L3321:26
weshay|ruckholser, 2020-10-27 17:38:27.756548 | ubuntu-bionic | {3} tripleoclient.tests.v1.overcloud_deploy.test_overcloud_deploy.TestDeployOvercloud.test_config_download_timeout [0.040716s] ... FAILED21:28
holserweshay|ruck I saw that21:28
rlandyweshay|ruck: k - I can take it21:29
rlandylet me juts un DNM my rocky stein work and that can merge then21:29
weshay|ruck++21:29
weshay|ruckthank you21:29
weshay|ruckrlandy, I'll assume that each puppet repo that takes that job as the parent will use a content provider and we don't have to touch each puppet repo21:30
weshay|ruckholser, I don't see anything that would make a slow infra system ( which is the case today ) affect that test https://opendev.org/openstack/python-tripleoclient/src/branch/master/tripleoclient/tests/v1/overcloud_deploy/test_overcloud_deploy.py#L1532-L157221:32
weshay|ruckholser, looks fairly stable.. what happens when you run locally? https://zuul.openstack.org/builds?job_name=tripleo-cross-tripleoclient-py3721:32
holserweshay|ruck playing with tests right now21:33
weshay|ruckk21:33
rlandyweshay|ruck: k ... so that's puppet on all branches?21:33
weshay|ruckrlandy,  just focus on master for now21:34
rlandyk21:34
weshay|ruckrlandy, let's see if the other puppet repos play nicely w/ it21:34
weshay|ruckrlandy, example 760246,1 in zuul now21:34
weshay|ruckpuppet-keystone-tripleo-standalone21:34
*** dsneddon has quit IRC21:35
weshay|ruckif we fix that parent, maybe it will run w/ a content-provider21:36
rlandywow - there is a lot of triggering there21:36
rlandyso add:21:39
rlandy            vars: &undercloud_consumer_vars21:39
rlandy              consumer_job: true21:39
rlandy              remove_tags:21:39
rlandy                - build21:39
rlandy            dependencies:21:39
rlandy              - tripleo-ci-centos-8-content-provider21:39
rlandyand the content provider job call21:39
*** slaweq has quit IRC21:48
* weshay|ruck looks21:49
weshay|ruckrlandy, ya.. I think that will do it21:50
weshay|ruckrlandy, we can enlist alex's review too21:50
weshay|ruckthank you!21:51
rlandyk21:53
*** rfolco has joined #oooq21:57
*** rfolco has quit IRC22:03
rlandyweshay|ruck: https://review.opendev.org/760249 Add content provider jobs for tripleo based jobs22:03
weshay|ruckrlandy++22:04
*** slaweq has joined #oooq22:04
rlandyweshay|ruck: hmmm ... didn't trigger any tripleo jobs22:06
rlandytripleo-based22:06
weshay|ruckrlandy, why add undercloud-containers?22:06
rlandyno content provider22:06
weshay|ruckoh .. u didnt22:07
rlandyit was there22:07
rlandylet me change a relevant file22:09
weshay|ruckya22:09
weshay|rucksorry22:09
weshay|ruckrlandy, testing w/ 76024622:09
weshay|ruckya.. didn't pick up content-provider22:10
rlandyhmmm22:11
rlandywhy though?22:11
rlandynot defined in that repo?22:11
weshay|ruckrlandy, lets try a change in puppet-integration itself22:13
*** frenzy_friday has quit IRC22:13
rlandyweshay|ruck: just did22:13
rlandysee patch 222:13
*** slaweq has quit IRC22:14
rlandyzuul still stuck on patch one22:14
weshay|ruckrlandy, ah k22:14
weshay|ruckya22:14
weshay|ruckCOME ON ZUUL22:14
weshay|ruckit's like playing casino games22:14
weshay|ruck4 apples!!!22:15
rlandyugh - seriously22:15
weshay|rucktoo slow22:16
weshay|ruckrlandy, let's look tomorrow22:16
weshay|ruckrlandy, get out of here22:16
arxcruzweshay|ruck: rlandy sorry being late it's being tagged properly22:17
arxcruzi had to read the quay.io api documentation, which is not so good in some aspects22:17
arxcruzhttps://quay.io/repository/tripleostein/centos-binary-cinder-backup?tab=tags22:17
rlandyarxcruz: thanks - will recheck22:17
arxcruzyou can see some tags already being tagged22:17
arxcruzrlandy: wait a little bit, i'm still running the script to tag22:18
rlandyhttps://quay.io/repository/tripleostein/centos-binary-aodh-api?tab=tags22:18
rlandyarxcruz: I see - no worries - ...22:18
rlandyI need ti run out in about 15 0 will recheck when I get back22:18
arxcruzrlandy: it will be done i think in 15 min22:19
rlandyweshay|ruck: zuul registered patch 2 but no content provider22:19
weshay|ruckno tripleo job22:20
rlandyah ...22:20
rlandyhttps://opendev.org/openstack/puppet-openstack-integration/src/branch/master/zuul.d/layout.yaml22:20
rlandywe need to find where this template is called22:21
weshay|ruckadd tripleo-puppet-standalone to line 21?22:21
rlandyhttps://opendev.org/openstack/puppet-openstacklib/src/branch/master/.zuul.yaml#L822:22
rlandythat is the only reference I see22:22
arxcruzrlandy: i gotta go sleep, but please, you can run in 15 minutes22:22
rlandyweshay|ruck: yeah - I'll add it just to test the trigger22:22
weshay|ruckk22:23
arxcruzit will take a while until the job get to the point to pull the images so it will end in time22:23
weshay|ruckthen get out of here ;)22:23
weshay|ruckarxcruz, good night :)22:23
rlandyarxcruz: no worries - go to sleep - I'll leave you email if there is a problem22:23
rlandygood night22:23
weshay|ruckand thanks22:23
rlandylet's see what happens here: https://review.opendev.org/76025222:27
rlandytripleo-ci-centos-8-content-provider22:27
rlandyweshay|ruck: ^^22:27
rlandywoohoo22:27
rlandywe have him now22:28
rlandyweshay|ruck: but child jobs will no inherit22:29
rlandythose vars are per template22:29
*** rfolco has joined #oooq22:30
* rlandy has to go to warehouse job - back later22:33
*** rlandy is now known as rlandy|bbl22:33
*** dsneddon has joined #oooq22:34
*** rfolco has quit IRC22:35
*** dsneddon has quit IRC22:39
*** dsneddon has joined #oooq22:40
weshay|ruckrlandy|bbl, yup.. ok we'll tackle them tomorrow22:45
*** dsneddon has quit IRC22:45
*** dsneddon has joined #oooq22:46
*** dsneddon has quit IRC22:51
*** dsneddon has joined #oooq22:52
*** TrevorV has quit IRC23:10
*** dsneddon has quit IRC23:23
*** dsneddon has joined #oooq23:25
*** holser has quit IRC23:58

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!