Tuesday, 2019-11-19

*** ysandeep has joined #oooq00:06
*** ysandeep has joined #oooq00:06
*** tosky has quit IRC00:07
*** dsneddon_ has joined #oooq00:09
*** dsneddon_ has quit IRC00:14
*** ysandeep has quit IRC00:24
*** rlandy|rover has quit IRC00:30
*** ysandeep has joined #oooq00:33
*** dsneddon_ has joined #oooq00:46
*** weshay_ has joined #oooq00:48
*** jschlueter has quit IRC01:04
*** redrobot has quit IRC01:07
*** jtomasek has quit IRC02:06
*** jtomasek has joined #oooq02:13
*** caowei99 has joined #oooq02:16
*** ysandeep has quit IRC03:04
*** apetrich has quit IRC03:08
*** dsneddon_ has quit IRC03:38
*** ysandeep has joined #oooq03:39
*** ykarel|away has joined #oooq03:53
*** ykarel|away is now known as ykarel03:58
*** bhagyashris has joined #oooq04:02
*** ykarel has joined #oooq04:02
*** skramaja has joined #oooq04:08
*** sanjayu__ has joined #oooq04:08
*** dsneddon_ has joined #oooq04:10
*** weshay has quit IRC04:18
*** weshay has joined #oooq04:19
*** epoojad1 has joined #oooq04:29
*** caowei99 has quit IRC04:43
*** caowei99 has joined #oooq04:48
*** bhagyashris has quit IRC04:51
*** ratailor has joined #oooq04:53
*** epoojad1 has quit IRC04:58
*** caowei99 has quit IRC05:02
*** dsneddon_ has quit IRC05:06
*** dsneddon_ has joined #oooq05:14
*** dsneddon_ has quit IRC05:20
*** ykarel_ has joined #oooq05:21
*** ykarel has quit IRC05:24
*** ysandeep has quit IRC05:36
*** bhagyashris has joined #oooq05:38
*** ysandeep has joined #oooq05:43
*** dsneddon_ has joined #oooq05:48
*** surpatil has joined #oooq06:00
*** caowei99 has joined #oooq06:08
*** openstackstatus has joined #oooq06:08
*** ChanServ sets mode: +v openstackstatus06:08
*** epoojad1 has joined #oooq06:15
*** marios has joined #oooq06:19
*** akahat has joined #oooq06:51
*** dsneddon_ has quit IRC07:03
*** udesale has joined #oooq07:06
*** dsneddon_ has joined #oooq07:13
*** jfrancoa has joined #oooq07:18
*** apetrich has joined #oooq07:34
*** dtrainor has quit IRC07:35
*** ykarel_ has quit IRC07:44
*** ykarel_ has joined #oooq07:44
*** ykarel_ is now known as ykarel07:48
*** ccamacho has joined #oooq07:48
*** surpatil has quit IRC07:49
arxcruzmarios: the scen000 still a wip07:55
arxcruzenabling only keystone07:55
*** surpatil has joined #oooq07:57
mariosarxcruz: k thanks, i also looked at the task in taiga but there was no moreinfo there08:07
*** amoralej|off is now known as amoralej08:18
*** tesseract has joined #oooq08:18
*** tosky has joined #oooq08:26
*** sanjayu__ has quit IRC08:27
*** bogdando has joined #oooq08:34
*** ysandeep has quit IRC08:41
*** ysandeep_ has joined #oooq08:41
*** jpena|off is now known as jpena08:42
*** caowei99 has quit IRC08:45
*** caowei99 has joined #oooq08:45
*** ykarel is now known as ykarel|lunch08:48
*** dsneddon_ has quit IRC09:06
*** ykarel|lunch is now known as ykarel|pto09:09
*** d0ugal has quit IRC09:18
*** sanjayu__ has joined #oooq09:21
*** d0ugal has joined #oooq09:23
*** marios has quit IRC09:29
*** dsneddon_ has joined #oooq09:30
*** ratailor_ has joined #oooq09:30
*** ratailor has quit IRC09:32
*** dsneddon_ has quit IRC09:36
*** derekh has joined #oooq09:38
*** ykarel|pto has quit IRC09:43
*** ysandeep_ has quit IRC09:55
*** ysandeep has joined #oooq09:55
*** dsneddon_ has joined #oooq10:05
*** d0ugal has quit IRC10:07
*** d0ugal has joined #oooq10:07
*** sshnaidm|afk is now known as sshnaidm|ruck10:10
*** d0ugal has quit IRC10:30
*** marios has joined #oooq10:36
*** marios has quit IRC10:39
*** marios has joined #oooq10:39
*** d0ugal has joined #oooq10:40
*** rfolco has joined #oooq10:43
*** caowei99 has quit IRC10:46
*** caowei99 has joined #oooq10:49
*** ratailor_ has quit IRC10:58
*** ratailor has joined #oooq10:59
mariosfmount:11:02
mariossorry fmount by mistake ^11:02
*** dsneddon_ has quit IRC11:08
rfolcothank god it was just a fmount11:16
*** udesale has quit IRC11:17
*** caowei99 has quit IRC11:20
*** caowei99 has joined #oooq11:20
*** jschlueter has joined #oooq11:35
*** caowei99 has quit IRC11:48
*** dtantsur|afk is now known as dtantsur11:49
*** akahat has quit IRC11:57
chandankumarweshay: component ci meeting time12:00
weshayaye12:00
*** sanjayu__ has quit IRC12:03
*** saneax has joined #oooq12:04
mariosrfolco: o/12:08
*** epoojad1 is now known as epoojad1|afk12:11
*** ratailor has quit IRC12:15
*** epoojad1|afk has quit IRC12:16
rfolcochandankumar, will need more time for the bug triage12:28
rfolcowill be late12:28
rfolcowait... marios isn't our component mtg after community call?12:29
*** jpena is now known as jpena|lunch12:31
mariosrfolco: well, i saw an invite for later on yeah but i thought that one was with 'everyone' i.e. mburns et al12:31
mariosrfolco: lets talk 5 mins now at least for quick sync on that stuff you put out12:31
mariosrfolco: see if there is somethign i can do there otherwise i continue digging in my other stuff12:31
*** caowei99 has joined #oooq12:32
*** ysandeep has quit IRC12:32
pandamarios: the job in 23698 fails because of how the command string is split in the subproces call12:33
mariospanda: ack thanks i am working on the hashes unchanged now will look at that next12:34
rfolcomarios, where12:37
*** rlandy has joined #oooq12:54
*** rlandy is now known as rlandy|rover12:54
mariosrfolco:++12:56
rfolcomarios, does it mean death ?12:56
mariosrfolco: haha no man just nice one getting progress there12:57
*** udesale has joined #oooq12:58
*** surpatil has quit IRC12:59
rfolco:013:02
rlandy|roversshnaidm|ruck: hi - anything to catch up on?13:02
sshnaidm|ruckrlandy|rover, hey13:02
rfolcochandankumar, o/13:03
rlandy|roverhttps://ci.centos.org/view/rdo/view/promotion-pipeline/ surprisingly green13:03
chandankumarrfolco: \o13:03
sshnaidm|ruckrlandy|rover, buildimage again, rerunning it13:03
rlandy|roversshnaidm|ruck: I already kicked it13:03
rfolcochandankumar, wanna triage13:03
sshnaidm|ruckrlandy|rover, and ovb 001 for rhel8 train failed, looking now13:03
sshnaidm|ruckrlandy|rover, will rerun it too13:03
rlandy|roversshnaidm|ruck: also rekicked13:03
sshnaidm|ruckrlandy|rover, oh, great13:03
rlandy|roversshnaidm|ruck: working on jobs to replace fs00213:04
sshnaidm|ruckrlandy|rover, be aware of https://review.opendev.org/#/c/677237/13:04
rfolcochandankumar, ok if you are busy, I'll do it offline :)13:04
sshnaidm|ruckrlandy|rover, ack, I'll do updates to cockpit finally13:04
chandankumarrfolco: sorry I need to leave now, better offline , I will take a look on tempes tbugs13:04
*** chandankumar is now known as raukadah13:05
*** dsneddon_ has joined #oooq13:05
rfolcoraukadah, np13:05
rlandy|roversshnaidm|ruck: are we expecting fallout from https://review.opendev.org/#/c/677237?13:05
rlandy|roverweshay: pls review https://review.rdoproject.org/r/#/c/23756/ and let me know if these jobs are correct to replace fs00213:09
*** dsneddon_ has quit IRC13:09
sshnaidm|ruckrlandy|rover, it touches a lot13:11
sshnaidm|ruckrlandy|rover, ci passed, but maybe everything13:12
mariospanda: fyi when you next have some time thanks https://review.rdoproject.org/r/#/c/23037/11/ci-scripts/dlrnapi_promoter/test_get_latest_hashes.py - main problem is that may fail (race) when promotion happens as that is executed (so hashes will change)... but it does exercise the functions. let me know what you think13:12
*** skramaja has quit IRC13:16
*** hamzy has quit IRC13:19
rfolcomarios, what var name I should set in testproject to get current-tripleo tag ? hash?13:22
mariosrfolco: not sure what you mean13:24
rfolcomarios, I think periodic job will get tripleo-ci-testing by default, isn t it ?13:24
mariosrfolco: variable to the job definition in testproject, https://review.rdoproject.org/r/#/c/23749/2/.zuul.yaml .. ah yeah periodic jobs should be using tripleo-ci-testing13:25
mariosrfolco: so force_periodic should cover you but is it failing/not bringing ci-testing?13:25
rfolcomarios, its not bringing right tag... and I need current-tripleo instead13:26
rfolconeed to enforce it13:26
mariosrfolco: trying to find the example *(yeah i see what you mean now) but not sure it will work13:27
mariosrfolco: so like that right https://review.rdoproject.org/r/#/c/23141/8/.zuul.yaml13:27
mariosrfolco: but i remember someone (wes? not sure) saying it won't work or we shouldn't do that13:28
mariosrfolco: maybe cos periodic overrides it?13:28
rfolcomarios, periodic use tripleo-ci-testing, I need current-tripleo13:28
rfolcomarios, don't recall13:29
mariosrfolco: periodic use ci-testing ... they promote to current-tripleo13:29
rfolcomarios, yes but in our case we need stable current-tripleo plus component13:29
mariosrfolco: k so not sure if that example helps... maybe you can try with dlrn_hash_tag13:30
mariosrfolco: community call13:30
*** jpena|lunch is now known as jpena13:31
rfolcocommunity call13:31
rfolcohttps://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw?both13:31
rfolcohttps://meet.google.com/bqx-xwht-wky13:32
rfolcorlandy|rover, sshnaidm|ruck arxcruz raukadah weshay marios panda zbr|ooo13:32
* weshay is on another call atm13:32
rfolcok13:33
rlandy|roverraukadah: hi - where are we with fs039 train?13:33
*** caowei99 has quit IRC13:34
*** dsneddon_ has joined #oooq13:36
*** bhagyashris has quit IRC13:37
*** dsneddon_ has quit IRC13:41
*** epoojad1 has joined #oooq14:11
*** epoojad1 has quit IRC14:46
*** jaosorior has joined #oooq14:49
*** bhagyashris has joined #oooq14:49
*** dtrainor has joined #oooq14:50
panda marios ping me if you want help with the test, it would be easire face to face. But if you prefer I'll comment in the review with a 60 pages detailed guide.15:06
weshayrlandy|rover, he's traveling atm15:08
*** bhagyashris has quit IRC15:08
rlandy|roverakc - ok15:11
mariospanda: ack thanks for comment i saw that... i am looking at the other thing now (manifest push switch )15:11
mariospanda: will revisit and we take from there. i can book a flight to ireland tomorrow morning15:12
rlandy|roverssbarnea: ping re: tox-linters15:14
mariospanda: few comments in review would be nice if you have some time though then i might iterate before you speak for example15:14
weshayrlandy|rover, I believe sorin is also out this week.15:16
weshayrlandy|rover, what can I help w/?15:16
rlandy|roverweshay: I am working on the tox-linters failure ...15:17
rlandy|roverit look slike the jobs comes from zuul-jobs15:17
weshayrlandy|rover, the one that was mentioned in rhos-delivery?15:17
*** saneax has quit IRC15:17
rlandy|roverand runs on the  runc-centos node15:17
rlandy|roverweshay: ack15:17
rlandy|roverweshay: now I think that is the case because run-ccentos is the default nodeset15:18
weshaylooks like it's failing in pre.. if retry failed15:18
rlandy|roverweshay: I need to switch the nodeset15:19
weshayrlandy|rover, to what?15:19
rlandy|roverbut I can't find the jobs definition15:19
rlandy|roverdib-centos-715:20
rlandy|roverhttps://sf.hosted.upshift.rdu2.redhat.com/logs/91/185791/1/check/tox-linters/8f13824/job-output.txt passes though15:21
rlandy|roverthose jobs aren;t even mine15:21
rlandy|roverI never added them15:21
rlandy|roverthey just live on our tenant15:21
weshayya.. probably just inherited15:22
weshayrlandy|rover, it only failed once15:22
weshay2019-11-18 21:19:18.352110 | runc-centos | MODULE FAILURE: sudo: a password is required15:23
weshayI think was the initial issue15:23
mariospanda: do you mean just the extra space is the problem? https://review.rdoproject.org/r/#/c/23698/3/ci-scripts/dlrnapi_promoter/dlrnapi_promoter.py@16315:24
rlandy|roverweshay: also that rhos-15 job is completely outdated15:24
rlandy|roverweshay: marios added that job15:25
rlandy|roverhttps://code.engineering.redhat.com/gerrit/gitweb?p=openstack/tripleo-ci-internal-config.git;a=commit;h=b7537244fa92cee35ef2d206a786199d5f08f62d15:25
rlandy|roverbut who maintains it????15:25
rlandy|roverthis philosophy that any that exists downstream lands on my plate is stupid15:26
rlandy|roverweshay: basically what's the point of having these jobs run?15:27
mariosrfolco: o/ need me to check somethign?15:28
marioserr sorry rfolco meant rlandy|rover re 17:25 < rlandy|rover> weshay: marios added that job15:28
weshayrlandy|rover, so.. there is an ideal and then the pragmatic reality15:28
weshayI'll address the pramatic reality atm15:28
rlandy|roverthere is no gain in running rhos-15 jobs now15:29
mariosrlandy|rover: do you need me to check something? which job?15:29
rlandy|roveragainst changes to anything in tripleo-common15:29
weshayour team has not had any requirement to stand up and maintain check jobs on the internal tripleo repos15:29
mariosrlandy|rover: is that d/stream tripleo-common and the rhel8 standalone (we did that a while ago but it was mostly POC? weshay ?)15:29
weshayand I have not pushed us there either because we have too much to do already15:29
weshayrlandy|rover, now.. ideally.. there can be a lot of patches submitted internally that lag going upstream15:30
weshayor never make it15:30
weshayupstream15:30
rfolcomarios, job is executing tempest \o/15:30
weshayand ideally we'd have some amount of coverage w/ check jobs on code.eng for tripleo15:30
mariosrfolco: nice !15:31
weshayrfolco, oooh.. the patched nova job?15:31
mariosrfolco: so it was just a "lucky" timing15:31
weshayor container update?15:31
mariosweshay: patched15:31
weshayI knew you guys would get it :)15:31
mariosweshay: err whats container update then ? you mean container build?15:31
mariosweshay: i did not rfolco did15:31
pandamarios: yes, the subprocess is splitting the argumetns as "-e", "manifest_push=false", "playbook". I think they should be split as "-e manifest_push=true", "playbook" because -e is an argument with a parameter.15:31
rfolcohold until we get a success and check logs15:31
weshaymarios, pretty sure I know which one.. it's all good15:32
mariosweshay: with https://review.opendev.org/#/c/69483615:32
rfolcofor now what I have is console15:32
rfolcohttps://review.rdoproject.org/zuul/stream/7a6e65f0478c446ebc3a2ef96e1974d3?logfile=console.log15:32
*** rlandy|rover is now known as rlandy|rover|mtg15:32
weshaysee it.. thanks15:33
weshayrlandy|rover|mtg, /me still in mtgs15:34
*** ysandeep has joined #oooq15:34
mariospanda: yes but will removing the initial extra " " solve that. isn't clear to me15:34
mariospanda: "-e manifest_push=true" already has a space in it15:35
mariospanda: so it will get split into "-e" "manifest_push=true"15:35
*** bhagyashris has joined #oooq15:36
*** dsneddon_ has joined #oooq15:37
mariospanda: maybe do the split first and then insert the -e ... :/15:38
mariospanda: will try it let me know if you have better ideas15:38
pandamarios: I think I have it now15:38
pandamarios: remove the double space and you should be fine15:38
mariospanda: you mean the initial extra " "15:39
pandamarios: the double space is intepredet as "" in the subprocess call, which is, althouhg empty, the first positional argument, which is interpreted by ansible-playbook as the playbook15:39
pandamarios: yes15:39
pandamarios: so the commadn is sayin "I don't know what arguament is "path/to/playbook"" because it thinks it already has its playbook15:40
mariospanda: ack ok thanks15:41
mariospanda: that's what i asked and thought initially but your replies earlier confused me15:41
*** dsneddon_ has quit IRC15:42
weshayrfolco, http://logs.rdoproject.org/49/23749/2/check/periodic-tripleo-ci-centos-7-standalone-master/7a6e65f/logs/undercloud/etc/yum.repos.d/15:48
*** sshnaidm|ruck has quit IRC15:49
rfolcoweshay, we need to rename delorean.repo... let me check if component repo wasn't actually used15:49
*** sshnaidm has joined #oooq15:50
weshayya.. it's not there15:50
*** sshnaidm is now known as sshnaidm|ruck15:50
rfolcoweshay, the file is not there but might have used the component repo, confirming15:51
*** bhagyashris has quit IRC15:55
rfolcoweshay, definitely didn't work :(15:57
rfolcohttp://logs.rdoproject.org/49/23749/2/check/periodic-tripleo-ci-centos-7-standalone-master/7a6e65f/logs/undercloud/home/zuul/containers-prepare-parameters.yaml.txt.gz15:57
rfolcowill investigate after tc call15:57
weshayrfolco, you didn't update the release in the zuul change15:57
rfolcoweshay, I included a depends-on in the testproject change15:58
rfolcoweshay, the tqe change has the release file modified15:59
weshayrfolco, that's the source change for the release.. but it's still just running a vanilla periodic on master15:59
weshayrfolco, remember.. the quickstart.sh --release would need to point to the component release file15:59
rfolcoweshay, thats why I changed promotion-hash-master file, not created a new one15:59
weshayoh. /me looks16:00
rfolcoI am ignoring the new component-master.yml in the patch16:00
rfolconot using that one16:00
rfolcojust for the poc16:00
rfolcootherwise we need to change the logic16:00
weshayrfolco, ya.. so name collision16:00
weshayhttp://logs.rdoproject.org/49/23749/2/check/periodic-tripleo-ci-centos-7-standalone-master/7a6e65f/logs/undercloud/home/zuul/repo_setup.sh.txt.gz16:00
*** udesale has quit IRC16:00
rfolcocorrect we need to rename delorean.repo16:00
mariosweshay: rfolco do we have a sync callnow?16:03
mariosweshay: its on the calendar but i am foreveralone16:03
rfolcono :(16:03
rfolco@tc call16:03
marios"Component Testing CI Project16:03
* weshay sends flowers to marios 16:03
weshaynothing on my cal for component sync16:03
rfolcomarios, I don't have it in my cal either16:04
rfolco:(16:04
mariosweshay: rfolco ack ok thanks maybe i created a duplicate or something like that16:05
*** ysandeep has quit IRC16:08
rlandy|rover|mtgsshnaidm|ruck: for repo-setup, where is default list of repos we would build a gating repo for?16:13
sshnaidm|ruckrlandy|rover|mtg, not sure I understand16:13
weshayrlandy|rover|mtg, is that in to-build?16:14
weshaythe role16:14
rlandy|rover|mtgyes16:14
rlandy|rover|mtgwhere is that list?16:15
weshayrlandy|rover|mtg, https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/check-to-build-or-not-to-build/defaults/main.yml16:15
sshnaidm|ruckto-build just enables running of build-test-package role16:15
rlandy|rover|mtgsshnaidm|ruck: yeah - I am looking if we could include a nova change if we ran tripleo testing on a nova patch16:16
sshnaidm|ruckrlandy|rover|mtg, well, to_build is actually for images, not for packages..16:17
sshnaidm|ruckrlandy|rover|mtg, it's in repo-setup role16:17
weshayrlandy|rover|mtg, so for16:18
weshay2019-11-19 15:18:46.468714 | TASK [bindep : Install distro packages from bindep]16:18
weshay2019-11-19 15:18:46.908356 | runc-centos | MODULE FAILURE: sudo: a password is required16:18
weshayshouldn't the image have passwdless sudo?16:18
rlandy|rover|mtgsshnaidm|ruck: that is where I started looking but I didn;t see the list16:19
rlandy|rover|mtgweshay: runc-centos probably doens;t have sudo16:19
rlandy|rover|mtgif we switch to another node type we may get it16:20
rlandy|rover|mtgI was testing that out16:20
sshnaidm|ruckrlandy|rover|mtg, https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/defaults/main.yml#L21:L2116:20
rlandy|rover|mtgproblem is where the job is defind16:20
sshnaidm|ruckrlandy|rover|mtg, nova is not there, so should build16:21
rlandy|rover|mtgsshnaidm|ruck: ok - so this is a default list - we can override it per config file16:21
sshnaidm|ruckrlandy|rover|mtg, it's list for *skip*16:21
*** rlandy|rover|mtg is now known as rlandy|rover16:22
weshayrlandy|rover, the bottom of that is.. line 17 https://code.engineering.redhat.com/gerrit/gitweb?p=openstack/tripleo-ci-internal-config.git;a=blob;f=zuul.d/projects.yaml;h=524936923a026c5e7d9d03ceb54c21c2ce700c64;hb=HEAD#l1716:22
rlandy|roverweshay: think we should chat about downstream tox-linters failure16:22
weshayrlandy|rover, ok16:22
weshayavailable16:22
rlandy|roverI am aware that is the problem16:23
weshayk16:23
weshaymeet?16:23
rlandy|roverack16:23
rlandy|roverweshay: https://meet.google.com/aza-iude-oud16:24
*** amoralej is now known as amoralej|off16:27
pandamarios: \o/16:28
mariospanda: yeah the job passed that time https://review.rdoproject.org/r/23698 if you mean that?16:29
mariospanda: otherwise \o/ why are we celebrating tho16:29
pandamarios: yeah taht the job passed16:30
pandamarios: SHIP IT16:30
*** bogdando has quit IRC16:39
*** jpena is now known as jpena|brb16:46
*** jaosorior has quit IRC16:48
*** tesseract has quit IRC16:51
*** chem is now known as chem|eod16:57
*** marios is now known as marios|out17:01
*** jaosorior has joined #oooq17:02
*** chem|eod has quit IRC17:04
*** chem|eod has joined #oooq17:08
rlandy|roverweshay: ok - so the tox-linters jobs looks ok on my first test on the upstream node - going to recheck the tripleo-common job17:10
weshayk cool17:10
rlandy|roverhttps://sf.hosted.upshift.rdu2.redhat.com/logs/09/185109/7/check/tox-linters/8da211b/17:11
*** marios|out has quit IRC17:14
rlandy|roverlunch - brb17:19
*** dtantsur is now known as dtantsur|afk17:20
*** ccamacho has quit IRC17:23
*** jpena|brb is now known as jpena17:26
rfolcojpena, can you please create a symlink for https://trunk-staging.rdoproject.org/centos7/component/compute/consistent/delorean.repo <-- component.repo17:27
rfolcojpena, just for a poc17:27
jpenarfolco: sure17:27
rfolcoreason is delorean.repo is conflicting with the normal repo, so it gets overwritten17:28
jpenarfolco: done. I have stopped the cron job, to avoid overwriting the consistent symlink17:29
rfolcojpena, nice!! thank you very much :)17:30
rlandy|roverhttps://sf.hosted.upshift.rdu2.redhat.com/logs/13/171213/10/check/tox-linters/70a6ef7/job-output.txt17:35
rlandy|roverweshay: ^^ same deal with the upstream node17:35
* weshay looks17:35
rlandy|rovermoving the tox job to non-voting while we look into it17:35
rlandy|rover2019-11-19 17:20:49.963741 |17:36
rlandy|rover2019-11-19 17:20:49.963948 | TASK [tox : Run tox without tests]17:36
rlandy|rover2019-11-19 17:30:13.318929 | upstream-centos-7 | ERROR: unknown environment 'linters'17:36
rlandy|roveroh wait17:36
weshay2019-11-19 17:30:13.318929 | upstream-centos-7 | ERROR: unknown environment 'linters'17:36
weshayya17:36
weshaydifferent17:36
rlandy|roveranyways17:36
weshayI think that is a tox.ini problem17:36
rlandy|rovermaking it non-voting17:36
weshayI bet they didn't have it cherry picked17:37
weshaysec17:37
*** holser has joined #oooq17:38
*** dsneddon_ has joined #oooq17:38
*** jfrancoa has quit IRC17:40
weshayrlandy|rover, sec17:41
weshayrlandy|rover, it's missing envlist = pep8,py37,py36,py2717:41
weshaylinters17:41
weshayrlandy|rover, hrm..https://opendev.org/openstack/tripleo-common/src/branch/stable/rocky/tox.ini17:42
rlandy|roverweshay: http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-config.git/tree/tox.ini?17:42
weshayrlandy|rover, so.. 5 min on this.. but it's interesting17:43
*** dsneddon_ has quit IRC17:43
weshayrlandy|rover, https://meet.google.com/zwv-dxni-oxz?authuser=117:43
rlandy|roverweshay: https://code.engineering.redhat.com/gerrit/185808 Do not stop tripleo-common jobs on check failures yet17:47
weshayrlandy|rover,17:50
weshay remotes/origin/rhos-16.0-trunk-patches17:50
weshay  remotes/origin/rhos-17.0-trunk-patches17:50
*** chem|eod has quit IRC17:53
*** holser has quit IRC17:54
*** chem|eod has joined #oooq17:59
*** derekh has quit IRC18:01
rlandy|roverweshay: rhos-15 branches look good18:05
rlandy|rovertesting 16 now18:05
sshnaidm|ruckrlandy|rover, do we have something about master upgrades job? Or it's in progress by matbu? https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-standalone-upgrade-master18:05
weshaycool18:05
rlandy|roversshnaidm|ruck: nothing on master atm18:05
rlandy|roveronly train18:05
rlandy|roveri don;t think we expect master to pass atm18:06
rlandy|roverif it is - great18:06
sshnaidm|ruckrlandy|rover, so we should ignore it18:06
rlandy|roversshnaidm|ruck: which job is failing?18:06
sshnaidm|ruckbecause I see it was passing18:06
sshnaidm|ruckrlandy|rover, see my link above18:06
rlandy|roverif it was passing before and failing now, LP it18:06
rlandy|roversshnaidm|ruck: since it passed at one point, I think we should at least investigate18:07
rlandy|roveris it in criteria?18:07
rlandy|roverchecking18:07
rlandy|roverit's not in criteria18:08
rlandy|roverso not critical outage18:08
rlandy|roverweshay: tox-lintersopenstack-tripleo-commonrhos-16.0-trunk-patchescheck185625,41842019-11-19T17:56:02SUCCESS18:09
sshnaidm|ruckrlandy|rover, yeah, it seems failing on /var/lib/openvswitch/ovnsb.db issue18:09
rlandy|roversshnaidm|ruck: so it's real18:09
sshnaidm|ruckrlandy|rover, and it was fixed recently18:09
weshayrlandy|rover, sshnaidm|ruck added to etherpad.. upgrade bug for train.. https://bugs.launchpad.net/tripleo/+bug/185318318:10
openstackLaunchpad bug 1853183 in tripleo "undercloud upgrades on train is failing in step 1" [Critical,Triaged]18:10
weshayrlandy|rover++18:10
weshayrlandy|rover, well done!18:10
rlandy|roverthanks for the help18:10
rlandy|roverwhat about upgrade for train18:10
weshayrlandy|rover, sshnaidm|ruck context.. all the train upgrade / update jobs should work.. check and gate at this time18:10
weshayrlandy|rover, /me just going through the train jobs..18:10
rlandy|roverweshay: two work - one not yet18:10
weshaytwo work, two not yet18:10
rlandy|rovermatbu made some updates18:10
weshayaye18:11
rlandy|rovertwo don't work?18:11
rlandy|rovershoudl be one18:11
weshayya.. undercloud-container upgrade18:11
weshayjust opened18:11
rlandy|rovercorrect18:11
weshayworks in rocky18:11
rlandy|roverthat is what the bug is about18:11
weshaystandalone-upgrade and undercloud-upgrade are failing18:11
rlandy|rovermatbu sent an update on status yesterday18:11
weshaythat's two different issues no?18:11
weshayright.. he was hitting an issue w/ ovn18:12
rlandy|roversshnaidm|ruck: ^^ same as master?18:12
rlandy|roverchecking18:12
weshayrlandy|rover, master doesn't matter18:12
sshnaidm|ruckrlandy|rover, weshay I think it was fixed in weekend, but maybe broke updates18:12
weshaymaster upgrades can fail all they want18:12
weshayI think we want to try to keep master updates passing... that's all18:13
sshnaidm|ruckovsdb-tool: I/O error: /var/lib/openvswitch/ovnsb.db: failed to lock lockfile (Resource temporarily unavailable)18:13
rlandy|roverperiodic-tripleo-ci-centos-7-standalone-upgrade-train look slike the only failure to me18:13
weshaysshnaidm|ruck, rlandy|rover you guys almost done :))18:13
weshayrlandy|rover,  http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containerized-undercloud-upgrades&branch=stable%2Ftrain18:14
rlandy|roverhttps://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released18:14
rlandy|rovercheck job18:14
rlandy|roverah18:14
rlandy|rovernot sure of this is the same - looking18:15
weshayah.. this is passing in periodic periodic-tripleo-ci-centos-7-singlenode-featureset050-upgrades-trainopenstack/tripleo-cimasteropenstack-periodic-latest-releasedmaster32402019-11-19T10:14:14SUCCESS18:15
rlandy|roverwas only focused on periodic18:15
rlandy|roverweshay: pls don't add it to criteria again18:16
weshayrlandy|rover, aye..18:16
rlandy|roverit's there18:16
rlandy|roverand that is what broke promotions last time :)18:16
weshayrlandy|rover, ruh roh http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-singlenode-featureset050-upgrades-train/75cfae8/logs/undercloud/home/zuul/18:16
weshayrlandy|rover, upgrades are not running :(18:17
rlandy|roverha18:17
weshaywe need to fix these names18:17
weshaydefault back to upstream18:17
rlandy|roverare they on other releases?18:17
weshaythey are in check18:17
rlandy|roversec - getting tea18:17
* weshay looked at a rocky job18:17
rlandy|roverchecking stein18:18
rlandy|roverhttp://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-singlenode-featureset050-upgrades-stein/02ae479/logs/undercloud/home/zuul/18:20
rlandy|roverweshay: ^^ to compare18:20
weshayooof18:20
rlandy|roverlol18:20
weshayrlandy|rover, this smells like emit releases18:20
rlandy|roverhold on18:20
weshaybut first getting the names right would top my personal agenda18:21
rlandy|roverI'm not done checking yet18:21
*** dsneddon_ has joined #oooq18:22
rlandy|roverweshay: ugh ... logs are gone ... http://logs.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-singlenode-featureset050-upgrades-rocky/f90d502/18:22
rlandy|roverwhat's wrong with the names?18:24
rlandy|roverby any other name ....18:24
rlandy|roverwould fail the same way18:25
weshayrlandy|rover, totally.. but upstream never adopted featureset naming convention18:25
weshayso mapping fs050 to undercloud-upgrades is confusing18:25
weshayI'll raise it to the team in planning for the ruck/rover to fix tech debt on18:26
weshayand some other items18:26
weshayrlandy|rover, fyi.. looks like rel-del approves queens rocky at 14 days https://code.engineering.redhat.com/gerrit/#/c/185675/18:26
weshayI'll udpate the board in a bit18:26
*** dsneddon_ has quit IRC18:26
rlandy|roverweshay: nice18:26
rlandy|roverI'll look into why upgrades is not running18:26
weshaythank you!!18:26
* weshay is going to drive home at lunch time... out for 1.5 hours18:27
*** jpena is now known as jpena|off18:32
rlandy|roverhmmm ... no emit_releases_file19:03
weshaymay have a docker.io issue 2019-11-19 18:41:48 | HTTPError: 500 Server Error: Internal Server Error for url: https://auth.docker.io/token?scope=repository%3Atripleomaster%2Fcentos-binary-swift-proxy-server%3Apull&service=registry.docker.io19:04
weshayor that could be bogdan's patch19:06
weshayrfolco, are we celebrating? http://logs.rdoproject.org/49/23749/3/check/periodic-tripleo-ci-centos-7-standalone-master/4d15e33/logs/undercloud/etc/yum.repos.d/component.repo.txt.gz19:09
rfolcoweshay, just saw this, checking if the rpms are there19:10
rfolcoupdated I mean19:10
*** TrevorV has joined #oooq19:11
weshay2019-11-19 18:01:57,046 16918 INFO tripleo_common.image.image_uploader [  ] Playbook:19:11
weshay- gather_facts: 'no'19:11
weshay  hosts: localhost19:11
weshay  tasks:19:11
weshay  - import_role:19:11
weshay      name: tripleo-modify-image19:11
weshay    name: Import role tripleo-modify-image19:11
weshay    vars:19:11
weshay      container_build_tool: buildah19:11
weshay      modified_append_tag: -updated-2019111917503619:11
weshay      source_image: 192.168.24.1:8787/tripleomaster/centos-binary-neutron-metadata-agent-ovn:feff260509b82feb934f5c91c90456d8fa8d7d71_7bfb84d119:11
weshay      target_image: 192.168.24.1:8787/tripleomaster/centos-binary-neutron-metadata-agent-ovn:feff260509b82feb934f5c91c90456d8fa8d7d71_7bfb84d119:11
weshay      tasks_from: yum_update.yml19:12
weshay      update_repo: component,delorean-current19:12
weshay      yum_repos_dir_path: /etc/yum.repos.d19:12
weshayI see that..19:12
rfolcoalso seen this yes19:12
rfolcobut the part that the container installs the rpms19:12
rfolcotrying to remember where is it19:12
weshayrfolco, don't see updates19:13
weshay2019-11-19 18:02:20,033 17878 INFO tripleo_common.image.image_uploader [  ]   stdout: No packages were found for update...19:13
weshayrfolco, suggestion.. put those two repos on a centos node.. manually19:16
weshayand check to see if yum priorities works out19:16
rfolcoaaah my bad19:18
rfolcoweshay, prio=99 low19:18
rfolcodammn stupid me19:18
*** dsneddon_ has joined #oooq19:19
rfolcoweshay, current-tripleo has19:21
rfolcoopenstack-nova-common-20.1.0-0.20191116083422.c956a88.el7.noarch.rpm19:21
rfolcoagainst19:21
rfolcoopenstack-nova-common-20.1.0-0.20191119165014.0644f03.el7.noarch.rpm19:21
weshay - --net19:21
weshay2019-11-19 18:06:04,638 17881 INFO tripleo_common.image.image_uploader [  ]   - host19:21
weshay2019-11-19 18:06:04,638 17881 INFO tripleo_common.image.image_uploader [  ]   - centos-binary-nova-api-working-container19:21
weshay2019-11-19 18:06:04,638 17881 INFO tripleo_common.image.image_uploader [  ]   - /tmp/yum_update.sh19:21
weshay2019-11-19 18:06:04,638 17881 INFO tripleo_common.image.image_uploader [  ]   - component,delorean-current19:21
weshay2019-11-19 18:06:04,638 17881 INFO tripleo_common.image.image_uploader [  ]   delta: '0:00:06.754728'19:21
rfolcofrom nova component19:21
weshay2019-11-19 18:06:04,639 17881 INFO tripleo_common.image.image_uploader [  ]   end: '2019-11-19 18:06:01.204898'19:21
weshay2019-11-19 18:06:04,639 17881 INFO tripleo_common.image.image_uploader [  ]   rc: 019:21
weshay2019-11-19 18:06:04,639 17881 INFO tripleo_common.image.image_uploader [  ]   start: '2019-11-19 18:05:54.450170'19:21
weshay2019-11-19 18:06:04,639 17881 INFO tripleo_common.image.image_uploader [  ]   stderr: ''19:21
weshay2019-11-19 18:06:04,639 17881 INFO tripleo_common.image.image_uploader [  ]   stderr_lines: <omitted>19:22
weshay2019-11-19 18:06:04,639 17881 INFO tripleo_common.image.image_uploader [  ]   stdout: |-19:22
weshay2019-11-19 18:06:04,639 17881 INFO tripleo_common.image.image_uploader [  ]     Loaded plugins: fastestmirror, ovl, priorities19:22
weshay2019-11-19 18:06:04,639 17881 INFO tripleo_common.image.image_uploader [  ]     Determining fastest mirrors19:22
weshay2019-11-19 18:06:04,639 17881 INFO tripleo_common.image.image_uploader [  ]     28 packages excluded due to repository priority protections19:22
weshay2019-11-19 18:06:04,640 17881 INFO tripleo_common.image.image_uploader [  ]     No packages marked for update19:22
weshay2019-11-19 18:06:04,640 17881 INFO tripleo_common.image.image_uploader [  ]   stdout_lines: <omitted>19:22
weshayno updates found for nova-api19:22
rfolcosame for nova-common, pretty much the same packages19:22
weshayrfolco, you could force the issue w/ excludes in the nova* in the current-tripleo repo config19:22
weshaybut one would think that yum would find that update19:23
weshaybut it's not19:23
rfolcoI think I'll just set prio=1 and retrigger while I think on a better approach19:24
weshayrfolco, think you did it backwards :P19:24
weshayA priority of 1 is the highest setting, and 99 is the lowest.19:24
rfolcoyes19:24
rfolco<rfolco> weshay, prio=99 low19:24
rfolco:)19:24
weshayrfolco, you want the component to always have priority.. so set it to 119:24
weshay:)19:25
weshayshould work next run :)19:25
rfolco\o/19:25
weshayok.. /me jumps in my car...19:25
weshaywill be afk for 1.5 hours19:25
rfolcoglad to have something at 45' of second half19:25
rfolcok19:25
rlandy|roverweshay: pls take a look at https://review.rdoproject.org/r/#/c/23756/19:27
rlandy|roverand see if these are the jobs you expect19:27
rlandy|roverto replace fs00219:27
weshayrlandy|rover, the job names and base look good19:29
weshayif they are periodic I assume that will include the push to the image server19:30
rlandy|roverweshay: they should19:30
rlandy|roverI want to merge those and try the train jobs19:30
rlandy|roverif I mess that images push - no check jobs will fail19:31
*** d0ugal has quit IRC19:32
weshayrlandy|rover, will do a little more digging when I get home19:32
weshaythanks rlandy|rover !19:32
rlandy|roversure19:36
rfolcorlandy|rover, new cix categories added to line 85 - https://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw?both19:41
rlandy|roverthank you19:41
rfolconp19:41
rlandy|roverlol - emit_release file is nowhere19:44
*** holser has joined #oooq19:48
*** dsneddon_ has quit IRC19:56
*** jfrancoa has joined #oooq19:58
*** dsneddon_ has joined #oooq19:58
rlandy|roverhttp://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-train/56f292c/logs/undercloud/home/zuul/19:59
rlandy|rover^^ that updates19:59
rlandy|roverso 37 works19:59
*** jfrancoa has quit IRC20:02
rlandy|rover2019-11-19 10:19:47.287205 | primary | /home/zuul/src/opendev.org/openstack/tripleo-ci/scripts/oooq_common_functions.sh: line 68: [: =: unary operator expected20:02
*** dsneddon_ has quit IRC20:03
*** dsneddon_ has joined #oooq20:06
*** dsneddon_ has quit IRC20:11
*** dsneddon_ has joined #oooq20:13
*** fmount has quit IRC20:22
rlandy|roverweshay: I think we're missing a tag20:22
*** tosky has quit IRC20:27
*** fmount has joined #oooq20:28
rlandy|rovergot it20:29
rlandy|roverlol - overcloud or undrecloud update20:33
*** d0ugal has joined #oooq20:49
rlandy|roverweshay: https://review.rdoproject.org/r/#/c/23763/20:49
*** d0ugal has quit IRC20:53
*** d0ugal has joined #oooq20:55
*** d0ugal has quit IRC21:03
*** holser has quit IRC21:04
*** holser has joined #oooq21:09
*** holser has quit IRC21:11
weshayrfolco, WHAHOOO :)21:12
weshayhttp://logs.rdoproject.org/49/23749/3/check/periodic-tripleo-ci-centos-7-standalone-master/43b8b00/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz21:12
weshay   Updating:21:12
weshay2019-11-19 19:51:20,861 17929 INFO tripleo_common.image.image_uploader [  ]      openstack-nova-common21:12
weshay2019-11-19 19:51:20,862 17929 INFO tripleo_common.image.image_uploader [  ]                        noarch 1:20.1.0-0.20191119165014.0644f03.el7 component 299 k21:12
weshay2019-11-19 19:51:20,862 17929 INFO tripleo_common.image.image_uploader [  ]      openstack-nova-compute21:12
weshay2019-11-19 19:51:20,862 17929 INFO tripleo_common.image.image_uploader [  ]                        noarch 1:20.1.0-0.20191119165014.0644f03.el7 component 5.3 k21:12
weshay2019-11-19 19:51:20,862 17929 INFO tripleo_common.image.image_uploader [  ]      openstack-nova-migration21:12
weshay2019-11-19 19:51:20,862 17929 INFO tripleo_common.image.image_uploader [  ]                        noarch 1:20.1.0-0.20191119165014.0644f03.el7 component 5.9 k21:13
weshay2019-11-19 19:51:20,862 17929 INFO tripleo_common.image.image_uploader [  ]      python2-nova      noarch 1:20.1.0-0.20191119165014.0644f03.el7 component 3.6 M21:13
weshay2019-11-19 19:51:20,862 17929 INFO tripleo_common.image.image_uploader [  ]21:13
weshayrlandy|rover, /me checks21:13
weshayrfolco++21:13
*** holser has joined #oooq21:14
weshayrlandy|rover, oh nice and easy21:14
weshaywf21:14
rlandy|rovertesting21:23
rlandy|roverhttps://review.rdoproject.org/zuul/stream/bd5535bd3664405e8c74689c65d0fbd4?logfile=console.log21:24
rlandy|roveroh good - train is promoting21:31
rlandy|roverlooks like master tried as wel21:31
rlandy|roverwell21:31
rlandy|roverhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-train is passing again21:37
rlandy|roversshnaidm|ruck: ^^ think I am going to put it back in criteria21:38
rlandy|roverweshay: ^^21:39
weshayoooh21:39
weshayif it's passing again.. then yes please21:39
* weshay looks21:39
rlandy|roverhttps://trello.com/c/3HNYEfKg/1223-cixlp1852770tripleociproa-no-more-ip-addresses-available-on-network-neutrontempestpluginscenariotestsecuritygroupsnetworksecgrou21:40
rlandy|roverweshay: I want to close that out21:40
rlandy|roverno outage atm21:40
weshayk21:40
rlandy|roverwe still have the review to adjust testing to tickle tls stuff21:41
weshayI think ade is getting close21:41
rlandy|roverwrong card - moved the other one21:47
rlandy|roverremoving above from skip list and testing21:47
rlandy|roverweshay: can we move this card somewhere that is not going to get questioned every CIX meeting? https://trello.com/c/gkvG3M6Q/1165-cixlp1846504tripleociproa-rdo-mirror-proxy-not-working-correctly21:50
weshayrlandy|rover, lolz... so I suppose we can nuke it.. it's still an issue21:51
weshayrlandy|rover, not sure if I shared the idea w/ you21:51
rlandy|roverweshay: yeah but CIX is not going to solve it21:51
weshaybut I was thinking we could include the proxy in the promotions....21:51
rlandy|rovermaybe our team board taiga21:51
rlandy|rovercool - add it to sprint ruck/rover someone who might actually action it21:52
weshayso when we promote a branch.. we trigger some action that would kill the ttl on the proxy21:52
* rlandy|rover fumbles and mumbles about it each meeting21:52
weshayrlandy|rover, have I shared that thought w/ you?21:52
rlandy|roverno - there is a note to chandan on the card though21:52
weshayrlandy|rover, think about.. ping me tomorrow w/ your opinion21:52
* weshay adds the thought there21:53
rlandy|roverweshay: let's just try it21:53
weshayrlandy|rover, ok.. I'll get jpena and crew on that card21:53
rlandy|roverjust trying to clean up before we had over to next ruck/rover21:54
rlandy|rovertesting revert ...21:56
rlandy|roverhttps://review.opendev.org/#/c/69509221:56
weshayrlandy|rover, ok.. comments here https://trello.com/c/gkvG3M6Q/1165-cixlp1846504tripleociproa-rdo-mirror-proxy-not-working-correctly21:56
rlandy|roverweshay: perfect - thanks - idea for retro tomorrow21:57
weshayrlandy|rover, sshnaidm|ruck you guys can skip the program call tomorrow22:01
weshayrlandy|rover, sshnaidm|ruck thank you both for keeping it GREEN22:01
rlandy|roverweshay: well on the upside fs050 does run undercloud upgrades now - on the downside, it fails .... http://logs.rdoproject.org/58/23558/8/check/periodic-tripleo-ci-centos-7-singlenode-featureset050-upgrades-train/bd5535b/job-output.txt22:01
weshayaye.. figured it would22:02
rlandy|roversure - thanks for heads up22:02
sshnaidm|ruckweshay, ack22:02
weshayrlandy|rover,  I have no idea how to check the code that checks the code22:02
rlandy|roverwas too good to be true anyways22:02
rlandy|roverweshay: very meta - what are we talking about now?22:02
rlandy|roversec - adding test job for revert22:03
weshayrlandy|rover, that the periodic upgrade job was not running upgrades22:03
rlandy|roveroh22:03
rlandy|roversshnaidm|ruck: just fyi - both master and train are promoting but both failed container push on this round - we may want to check that tomorrow morning22:04
rlandy|roverweshay: I think we should take fs050 out if criteria for train/master22:04
rlandy|roversince we merged the undercloud_upgrade fix22:04
weshayrlandy|rover, ya.. it can be removed from criteria for now, but we'll have to monitor the cix cards and put it back when it starts to pass22:06
weshayhttps://trello.com/c/XpfnGIpE/1230-cixlp1853183tripleociproa-undercloud-upgrades-on-train-is-failing-in-step-122:06
*** jtomasek has quit IRC22:07
rlandy|roverok22:08
* rlandy|rover edit criteria22:08
*** jtomasek has joined #oooq22:10
rlandy|roverok - removed from stein, train, master22:12
rlandy|roverleft in rocky and queens22:12
rlandy|roverwill remove if we have to22:12
rlandy|rovermay need to edit rocky/queens as well22:16
rlandy|roverwe shall see22:16
weshayit's failing in stein too?22:17
weshaythat's bad22:17
weshaybad upgrades team... bad22:17
rlandy|roverthey had no idea22:18
rlandy|roverour fault for false positive22:18
rlandy|roverit may pass - we don;t know22:19
rlandy|roveradding note sto ruck/rover etherpad to readd criteria if the job passes22:19
*** holser has quit IRC22:24
weshayperfect22:25
*** holser has joined #oooq22:29
*** TrevorV has quit IRC22:29
weshayrlandy|rover, oy22:38
weshayrlandy|rover, do you have two min?22:38
weshayrlandy|rover, /me wonders if you can join the retro an hour earlier?22:40
weshayrfolco, can you move the sprint retro one hour earlier?22:44
weshayand give me access to edit if possible22:44
*** jaosorior has quit IRC22:45
*** jaosorior has joined #oooq22:49
*** jaosorior has quit IRC22:52
*** holser has quit IRC23:09
rlandy|roverweshay: sorry - back23:13
rlandy|roverwhat's up?23:13
rlandy|roverweshay: yes - I can join earlier23:13
*** holser has joined #oooq23:15
*** tosky has joined #oooq23:24
*** tosky has quit IRC23:26
*** ysandeep has joined #oooq23:27

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!