Tuesday, 2020-01-21

*** chem has quit IRC00:51
*** rlandy is now known as rlandy|bbl00:55
*** rfolco has quit IRC01:41
*** dsneddon has quit IRC01:48
*** dsneddon has joined #oooq02:24
*** dsneddon has quit IRC02:49
*** apetrich has quit IRC03:10
*** dsneddon has joined #oooq03:16
*** dsneddon has quit IRC03:22
*** rlandy|bbl has quit IRC03:39
*** dsneddon has joined #oooq03:50
*** dsneddon has quit IRC03:59
*** epoojad1 has joined #oooq04:26
*** dsneddon has joined #oooq04:26
*** dsneddon has quit IRC04:31
*** udesale has joined #oooq04:44
*** udesale has quit IRC04:44
*** udesale has joined #oooq04:44
*** saneax has quit IRC04:55
*** dsneddon has joined #oooq05:04
*** dsneddon has quit IRC05:08
*** soniya29 has joined #oooq05:11
*** ykarel|away is now known as ykarel05:23
*** skramaja has joined #oooq05:25
*** udesale_ has joined #oooq05:34
*** udesale has quit IRC05:37
*** epoojad1 has quit IRC05:40
*** dsneddon has joined #oooq05:41
*** raukadah is now known as chandankumar05:45
*** dsneddon has quit IRC05:46
*** saneax has joined #oooq06:01
*** surpatil has joined #oooq06:03
*** SurajPatil has joined #oooq06:04
*** yolanda has quit IRC06:04
*** saneax has quit IRC06:06
*** epoojad1 has joined #oooq06:08
*** pojadhav has joined #oooq06:13
*** pojadhav has left #oooq06:17
*** epoojad1 has quit IRC06:18
*** dsneddon has joined #oooq06:20
*** dsneddon has quit IRC06:25
*** pojadhav has joined #oooq06:28
*** jfrancoa has joined #oooq06:50
*** saneax has joined #oooq06:53
*** dsneddon has joined #oooq06:57
*** dsneddon has quit IRC07:02
*** ksambor has joined #oooq07:02
*** soniya29 has quit IRC07:12
*** soniya29 has joined #oooq07:17
*** yolanda has joined #oooq07:25
*** jtomasek has joined #oooq07:25
*** marios has joined #oooq07:27
*** dsneddon has joined #oooq07:33
*** yolanda has quit IRC07:33
*** yolanda has joined #oooq07:34
*** ykarel is now known as ykarel|lunch07:36
*** dsneddon has quit IRC07:38
*** yolanda has quit IRC07:48
*** dsneddon has joined #oooq08:13
*** dsneddon has quit IRC08:19
*** amoralej|off is now known as amoralej08:20
*** tesseract has joined #oooq08:20
*** yolanda has joined #oooq08:21
*** ykarel|lunch is now known as ykarel08:38
*** jpena|off is now known as jpena08:52
*** holser has joined #oooq08:54
*** dtantsur|afk is now known as dtantsur08:57
*** soniya29 has quit IRC09:04
*** tosky has joined #oooq09:07
*** bogdando has joined #oooq09:07
*** apetrich has joined #oooq09:18
*** jaosorior has joined #oooq09:19
*** dsneddon has joined #oooq09:25
*** yolanda has quit IRC09:27
*** udesale_ has quit IRC09:28
*** SurajPatil has quit IRC09:28
*** udesale_ has joined #oooq09:28
*** derekh has joined #oooq09:28
*** SurajPatil has joined #oooq09:28
*** SurajPatil has quit IRC09:29
*** surpatil has quit IRC09:29
*** surpatil has joined #oooq09:30
*** dsneddon has quit IRC09:30
*** yolanda has joined #oooq09:33
*** soniya29 has joined #oooq09:36
*** yolanda has quit IRC09:49
*** jaosorior has quit IRC09:54
*** chem has joined #oooq10:02
*** dsneddon has joined #oooq10:04
arxcruzzbr|drover: chandankumar  please review and if possible merge https://review.opendev.org/#/c/703570/10:06
*** dsneddon has quit IRC10:09
*** ykarel is now known as ykarel|afk10:11
*** dsneddon has joined #oooq10:42
*** ykarel|afk is now known as ykarel10:43
*** dsneddon has quit IRC10:47
pojadhavHi all, While running overcloud deployment command for tripleo quickstart and I am facing error : AnsibleUndefinedVariable: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'inventory_ip'10:50
*** udesale_ has quit IRC10:51
pojadhavcould anyone help me to resolve this issue.10:53
sshnaidmpojadhav, look for case you use 'inventory_ip' somewhere, I'm not familiar with this variable tbh11:06
sshnaidmpojadhav, grep -r 'inventory_ip' tripleo-*11:06
arxcruzis inventory ip that one that needs to be 127.0.0.2 in quickstart ?11:19
*** surpatil has quit IRC11:26
chandankumarmarios, Hello11:27
chandankumarmarios, https://6c9c71359db665d0f3a4-5067ad0e5cf1f9e6b3e5798c03f452f6.ssl.cf1.rackcdn.com/703587/1/check/openstack-tox-docs/9906bac/docs/ci/third_party_dependencies_ci.html here hyperlinks are not working please have a look11:28
chandankumararxcruz, thanks for the review, I will update that soon11:29
marioschandankumar: o/11:31
marioschandankumar: sure will hav a look (was confusd until i realised you are asking for review @ https://review.opendev.org/#/c/703587 )11:32
zbr|droversshnaidm: does https://review.opendev.org/#/c/703586/ look small enough?11:33
marioschandankumar: commented .. not ::11:36
*** soniya29 has quit IRC11:38
*** soniya29 has joined #oooq11:39
sshnaidmzbr|drover, "warn: false" because of using shell?11:42
zbr|droveryeah, and that is not a linting error, is an ansible runtime warning.11:43
sshnaidmzbr|drover, ok11:45
sshnaidmzbr|drover, commented11:47
weshay|ruckmarios, had to revert that get_hash path update11:55
mariosweshay|ruck: yeah saw posted https://review.rdoproject.org/r/2461711:55
mariosweshay|ruck: i think it was the missing required-projects but we probably need it in all the base jobs there11:56
mariosweshay|ruck: thanks for handling the revert11:56
weshay|ruckmarios, k.. I posted a log path in the revert11:56
weshay|rucknp11:56
weshay|ruckwe can try again today11:56
mariosweshay|ruck: see https://review.rdoproject.org/r/#/c/24617/2/zuul.d/tripleo-rdo-base.yaml comments11:57
*** dsneddon has joined #oooq12:00
*** rfolco has joined #oooq12:00
sshnaidmweshay|ruck, mtg?12:01
weshay|rucksshnaidm, will catch you later today12:01
*** dtantsur is now known as dtantsur|bbl12:01
sshnaidmweshay|ruck, ack12:01
*** dsneddon has quit IRC12:05
*** yolanda has joined #oooq12:10
*** yolanda has quit IRC12:10
*** yolanda has joined #oooq12:11
*** amoralej is now known as amoralej|lunch12:15
*** jpena is now known as jpena|lunch12:21
zbr|droversshnaidm: i removed the non-esentials from patch, to make it easy to review/merge, see https://review.opendev.org/#/c/703597/1/defaults/main.yml12:28
zbr|droveri did the same with the gz one https://review.opendev.org/#/c/702862/10 -- which apparently worked fine (rdo failure unrelated, and collection did run)12:30
sshnaidmzbr|drover, what's the problem with HOME?12:30
zbr|droverthe only thing making harder to review is the sove -> default scenario move.12:30
zbr|drovertox hides HOME by default. I could passenv it, but there are other cases where HOME may be lost12:31
zbr|droveris better to have a valid fallback value instead of empty string, which makes it attempt to write to /.quickstart12:31
sshnaidmzbr|drover, how is tox related here??12:31
zbr|drovermolecule runs from tox12:32
sshnaidmzbr|drover, runs where?12:32
zbr|droveropenstack-tox-molecule12:32
sshnaidmzbr|drover, I mean on which platform12:32
sshnaidmzbr|drover, do you run log role in molecule on windows?12:33
zbr|drovernope, is does not matter which platform.12:34
zbr|droverit seems that you are not aware of the need to add passenv = HOME to make HOME available.12:34
sshnaidmzbr|drover, I think there is always default HOME in platform, no matter tox or not tox12:35
zbr|droversshnaidm:  NO, read https://github.com/tox-dev/tox/blob/e639a54eab5641c8dde353a29a516e4bf5487e37/src/tox/config/__init__.py#L670-L68012:36
zbr|droverusually we add it to passenv in tox.ini, but is even better to fix the role.12:37
sshnaidmzbr|drover, do you want to say tox unset HOME var and if you do "echo $HOME" inside a molecule you'll get undefined?12:37
zbr|droveryeah12:38
sshnaidmzbr|drover, show me this please12:38
mariosanyone know what we're using that file for? https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-ci-periodic-base/post-role.yaml don't see much history on it12:41
weshay|ruckrfolco, you get em all built?12:42
rfolcoweshay|ruck, 12012:45
rfolcoweshay|ruck, the epel workaround is that if you don't have epel repo the build will fail. It tried to disable at some point in the build12:46
rfolcoweshay|ruck, maybe its under the rhel8 condition, we may need to change that for centos812:47
*** ykarel is now known as ykarel|afk12:47
sshnaidmmarios, seems like for ovb only, to report to dlrn12:47
sshnaidmhosts: primary:tripleo-ovb-centos-712:48
zbr|droversshnaidm: check https://github.com/ssbarnea/please-sagi/blob/master/tox.ini and run tox12:49
mariossshnaidm: hmm thanks. might need to update any job definition that uses it so it can consume the changes https://review.rdoproject.org/r/#/c/24617/3/roles/dlrn-report/tasks/dlrn-vars-setup.yml12:50
chandankumarweshay|ruck, zbr|drover you might be hitting this in some jobs https://github.com/pypa/pip/issues/721712:50
chandankumarpip==20.0.1 will fix it12:51
mariossshnaidm: i couldnt find any job definition under config repo which uses that post-role.yaml or the pre-role.yaml in same directory12:51
weshay|ruckchandankumar, https://bugs.launchpad.net/tripleo/+bug/186043012:51
openstackLaunchpad bug 1860430 in tripleo "cannot import name SourceDistribution" [Critical,Triaged]12:51
zbr|droverchandankumar: good luck with that, only seeing the URL gave me headaches, there no other project I have more than pip12:52
chandankumarzbr|drover, http://logs.rdoproject.org/70/703570/1/openstack-check/tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset001/3e7d4c5/job-output.txt12:53
chandankumarseen here12:53
zbr|droveryeah pip v20 released today.... https://pypi.org/project/pip/#history12:53
zbr|drovertime to take the day off ;)12:54
zbr|droveror to watch talks on infra, probably they will pin it down12:54
sshnaidmzbr|drover, show me please when it's broken with tox molecule job, I'd like to see lookup("HOME") not working in molecule12:56
sshnaidmzbr|drover, that would justify this change12:56
sshnaidmmarios, possibly it's not used12:58
*** udesale has joined #oooq12:59
mariossshnaidm: ack thx for checking yeah that's what i was asking earlier... i can't see it used was wondering if someone remembers why/where ... git blame says rlandy will ask when around thanks13:01
*** dsneddon has joined #oooq13:07
*** rlandy has joined #oooq13:07
*** jbadiapa has joined #oooq13:09
*** dsneddon has quit IRC13:17
weshay|ruckmarios, rfolco13:18
weshay|ruck[zuul@wes workspace]$ cat containers-successfully-built.log | wc -l13:18
weshay|ruck11213:18
weshay|ruck[zuul@wes workspace]$ cat containers-failed-to-build.log | wc -l13:18
weshay|ruck213:18
weshay|ruckis my latest status13:18
weshay|ruckyou both have access to this box now13:18
*** zbr|drover has quit IRC13:18
*** zbr has joined #oooq13:19
weshay|ruck[zuul@wes workspace]$ cat containers-failed-to-build.log13:19
weshay|ruckqdrouterd13:19
weshay|ruckrabbitmq13:19
rfolcoweshay|ruck, epel!!!!13:19
weshay|ruckrfolco, can't use epel :)13:19
rfolcoweshay|ruck, you did not get it yet13:19
weshay|ruckI guess13:19
rfolcoweshay|ruck, create a fake epel.repo13:19
rfolcoit tries to disable a repo called epel13:20
rfolcoit does not use it13:20
weshay|ruckoh13:20
rfolcoand rabbit13:20
rfolcois the kolla patch pinning it13:20
weshay|ruckI have epel repos.. they are just disabled13:20
rfolcogo to docker/rabbitmq/Dockerfile.j2 and remove 3.7.* pinning13:20
weshay|ruckrfolco, let's get it in a review and pull the review in the role13:21
*** pojadhav is now known as pojadhav|away13:21
rfolcoweshay|ruck, it is in Mark's patch, its pinning 3.7.* I don't know why13:22
rfolcohttps://review.opendev.org/#/c/699414/9/docker/rabbitmq/Dockerfile.j213:22
*** jpena|lunch is now known as jpena13:23
weshay|ruckrfolco, let's take this review...  resubmit w/ a new commit-id and pull it in our ci13:23
rfolcook13:24
rfolcocat containers-expected-to-build.log | wc -l13:24
rfolco12213:24
rfolcoI've got 12013:24
rfolcoexcluded skydive, trying to build it again w/ skydive13:24
weshay|ruck99% sure we don't need those.. but we'll ask in #tripleo mtg13:25
weshay|ruckrfolco, what about line 2613:25
weshay|ruck   {% set rabbitmq_packages = rabbitmq_packages + [13:25
weshay|ruck            'rabbitmq-server-3.7.10'13:25
rfolcoweshay|ruck, thats what I said, I don't understand why the pinning, it says remove the pinning when the deps are resolved. And I tried without the pinning, it works13:26
weshay|ruckrfolco, it's pinned on two lines though13:26
rfolcoweshay|ruck, it falls on dnf case btw13:26
* weshay|ruck asking about the second13:26
rfolcoweshay|ruck, the second will fall on centos7 I believe13:26
chandankumarsshnaidm, weshay|ruck https://review.opendev.org/#/c/703555/13:27
weshay|rucksoniya29, can you test https://review.rdoproject.org/r/#/c/24616/13:27
chandankumaros_tempest dir linls13:27
soniya29weshay|ruck, sure13:28
rfolcomjturek, did you add your topic to the hackmd agenda ? https://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw?both13:29
weshay|ruckrfolco, ping me w/ your dup patch of https://review.opendev.org/#/c/699414/13:29
rfolcocommunity ci call starts in 1 min at https://meet.google.com/oiv-geho-mai13:29
rfolcoagenda https://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw?both13:29
weshay|ruckpanda, please make sure you attend ^13:30
rfolcomarios, weshay|ruck arxcruz zbr sshnaidm rlandy panda mjturek baha ^13:30
*** marios is now known as marios|call13:30
rfolcosorry13:30
rfolcohttps://meet.google.com/bqx-xwht-wky13:30
rfolcocorrect gmeet ^13:30
arxcruzrfolco: the calendar is pointing to a different code13:30
mjturekjust did rfolco sorry13:31
*** skramaja has quit IRC13:31
arxcruzah13:31
arxcruzok13:31
*** saneax has quit IRC13:44
*** dsneddon has joined #oooq13:48
*** dsneddon has quit IRC13:53
*** amoralej|lunch is now known as amoralej13:57
*** dtantsur|bbl is now known as dtantsur14:01
zbrgz dropping: https://review.opendev.org/#/c/702862/ and its parent, look for previous runs, or https://e6792871b9906e88b5d6-ba5467d2de73878d3dd4296de3d69c75.ssl.cf1.rackcdn.com/702862/10/check/openstack-tox-molecule/142504e/14:07
*** marios|call is now known as marios14:09
*** bhagyashris_ has joined #oooq14:09
*** bhagyashris_ has quit IRC14:10
weshay|rucksshnaidm, available now if you want14:23
sshnaidmweshay|ruck, k14:23
weshay|ruckhttps://meet.google.com/opj-yscd-ktw?authuser=114:23
weshay|ruckrlandy, please ping me later re: the osp container issue14:24
rlandyweshay|ruck: k - working on it - retesting now14:25
*** dsneddon has joined #oooq14:26
*** dsneddon has quit IRC14:31
rlandymarios: where is the question in your review? may be used downstream - will check14:31
mariosrlandy: ack will comment there sec but 14:41 < marios> anyone know what we're using that file for? https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-ci-periodic-base/post-role.yaml don't see much history on it14:35
rlandymarios: so it's like this ...14:42
rlandyhttps://github.com/rdo-infra/review.rdoproject.org-config/tree/master/playbooks/tripleo-ci-periodic-base14:42
rlandyhas playbooks that do dlrn reporting14:42
rlandydownstream we can't import playbooks14:42
rlandybut we can reuse roles14:43
rlandyhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/dlrn-report/tasks/main.yml14:43
rlandyis what gets used downstream14:43
rlandyhttp://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-config.git/tree/playbooks/dlrn/post-dlrn.yaml14:44
rlandyso now we will have to fix internal/downstream as well for your get-hash change14:45
rlandyit will break those jobs14:45
*** soniya29 has quit IRC14:45
rlandymarios: iirc, we added those playbooks to consume the same roles upstream14:47
rlandymarios: I can put in associated patches so that the get-hash move will not break downstream14:48
rlandyit will atm14:48
rlandyweshay|ruck: ^^ fyi14:48
mariosrlandy: ack thx, makes sense. yeah we need to change that since it consumes the file i'm changing. we now have a requirement for rdo-infra/ci-config wherever we are using get-hash or anythign that calls get-hash14:49
rlandymarios: ok - give me a few - I'll find the equivalent downstream requirements14:50
mariosrlandy: ok no rush thanks14:50
mariosfor clarity it's the pre-yaml we're most interested in14:53
mariosrlandy: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-ci-periodic-base/pre-role.yaml#L7 since that calls https://review.rdoproject.org/r/#/c/24617/3/roles/dlrn-report/tasks/dlrn-vars-setup.yml14:53
mariosrlandy: but i couldn't see where it was used in a job definition to update required-projects... hound didn't help me14:54
rlandyhound won;t search downstream14:54
rlandythe problem is inheritance14:54
rlandyand reuse14:54
rlandysec - commenting on the patch14:54
mariosrlandy: mine does (hav bash alias that does all 3 hound)14:54
rlandymarios: we never used the playbook I gues14:55
rlandythe role is the important one14:55
mariosrlandy: like that (hound) fwiw http://paste.openstack.org/raw/788643/14:55
mariosrlandy: so i couldn't find it in the d/stream hound either14:55
rlandyyou'll find       include_role:14:56
rlandyname: dlrn-report14:56
mariosrlandy: yeah i saw that bit14:56
mariosrlandy: but i was looking for a job pre that has that pre-role.yaml14:56
rlandymarios: k - responded on patch14:59
rlandymarios: actually, this would be a good time to make use of the dlrn_report role15:00
weshay|ruckrfolco, patch?15:00
rlandymarios: because then we would have a single place to deal with this15:01
rfolcoweshay|ruck, ok I was looking at the diff15:01
rlandyside point :)15:01
rlandymarios: sum total, please don't delete that playbook15:01
rfolcoweshay|ruck, these are built for centos7, not built for centos8 (and doesn't fail/complain) :15:01
rfolcoec2-api15:01
rfolconova-serialproxy15:01
rfolcopanko-base15:01
rfolcopanko-api15:01
rfolcosensu-base15:01
rfolcoweshay|ruck, will create the patch now15:02
mariosrlandy: no i am not proposing to delete it. my question was, is it being consumed somewhere, because if so we need to ensure the job definition that uses it include rdo-infra/ci-config in required-projects:15:02
mariosrlandy: sounds like 'not yet' so thanks for the sanity check in that case i don't need to update something more or block https://review.rdoproject.org/r/#/c/24617/3/playbooks/tripleo-ci-periodic-base/pre.yaml15:02
rlandymarios: no - juts put a temp -1 there while I ma putting up equivalent downstream patch15:03
rlandywill be up for review shortly15:03
mariosrlandy: ack please go ahead and do that ^^15:03
mariosrlandy: and you can remove it when you're ready15:03
mariosrlandy: i will revisit tomoorw15:03
mariostomorrow15:03
rlandyack15:03
mariosrlandy: thank you15:03
weshay|ruckzbr, comments on https://review.opendev.org/#/c/702862/1015:04
weshay|ruckzbr, for size.. fyi http://logs.rdoproject.org/62/702862/10/openstack-check/tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset001/428f9e6/logs/quickstart_files/log-size.txt15:05
zbrweshay|ruck: clark is not avl, but got confirmation from fungi that is ok. they do the compression.15:07
weshay|rucknice15:07
zbrthere are some low concerns reg temp disk on worker, but we are not asked to do anything. if they find an issue, besure I will be pinged first.15:07
zbr0.5G is not an issue.15:08
chandankumarweshay|ruck, zbr sorry you mean we are going to upload 0.5 g temp dir on log server?15:12
weshay|ruckper job15:12
* weshay|ruck checks what we have now w/ compression15:12
chandankumarweshay|ruck, I think we need to discuss this with jpena and team15:12
zbrchandankumar: not log server, zuul-worker.15:13
weshay|ruckhttp://logs.rdoproject.org/12/558412/14/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein-branch/72bf1ed/logs/quickstart_files/log-size.txt15:13
weshay|ruck147M15:13
chandankumarzbr, does zuul-worker will not go OOM?15:14
chandankumardue to 0.5 g15:14
*** TrevorV has joined #oooq15:14
weshay|ruckzbr, [08:16:39] <pabelanger> weshay|ruck: set zuul_log_compress: true in your base-minimal jobs and will fix15:17
weshay|ruckhttps://review.opendev.org/#/c/567696/315:17
jpenaweshay|ruck: ^^ at least that change will only compress console output, not every single file15:20
rfolcoweshay|ruck,  https://review.opendev.org/703640 Remove rabbitmq pinning for CentOS815:22
weshay|ruckjpena, :(15:25
weshay|ruckrfolco, k15:25
migiweshay|ruck: owalsh thinks this patch broke the featureset021 https://review.opendev.org/#/c/699502/15:32
weshay|ruckmigi, hrm.. k.. I will test that15:32
weshay|ruckmigi++15:33
weshay|ruckthanks for looking into it man :)15:33
migiweshay|ruck: it's owalsh who found it :)15:33
migiweshay|ruck: but it's just a guess so don't take it as resolution15:35
*** dsneddon has joined #oooq15:37
zbrweshay|ruck: so what to do?15:38
zbrenable the change now and look to improve the compression logic on upload-logs after? https://opendev.org/zuul/zuul-jobs/src/commit/7755ef1b8d4d7803c9ad330a1bd9599715e06c60/roles/upload-logs/tasks/main.yaml15:40
weshay|ruckmigi, I can test w/ a test revert15:40
migiweshay|ruck: look at this file, all IPs are same https://sf.hosted.upshift.rdu2.redhat.com/logs/36/189436/1/check/periodic-tripleo-ci-centos-7-bm_envD-1ctlr_2comp-featureset021-master/9150aee/logs/overcloud-controller-0/etc/ssh/ssh_known_hosts.gz15:40
zbrit will take at least two hours to merge, so enough time to make it upload-logs big more agressive15:40
migiweshay|ruck: same for other overcloud-*15:40
weshay|ruckzbr, we need logs to work.. so continue.. as we work out compression15:41
zbri would write a bash script that archives any .txt/.log file above 10mb15:41
weshay|ruckzbr, we probably will need an option to compress for rdo15:41
weshay|ruckjpena, ^15:41
zbrjpena: i was considering making this task more generic: https://opendev.org/zuul/zuul-jobs/src/commit/7755ef1b8d4d7803c9ad330a1bd9599715e06c60/roles/upload-logs/tasks/main.yaml#L35-L4315:42
*** dsneddon has quit IRC15:42
*** owalsh has joined #oooq15:42
zbrobviously that there is no existing scenario for upload-logs15:43
weshay|ruckrfolco, I think you have a conflict http://logs.rdoproject.org/40/703640/1/openstack-check/tripleo-build-containers-rhel-8-master/805846d/job-output.txt15:43
zbrin fact i cannot update this role as is upsteam, they don't care about that archival.15:44
weshay|ruckrfolco, although that applied I think for me15:45
weshay|ruckmarios, ping me if you have a sec15:45
zbrdo we have an existing role that could be hacked to also do the compresion or I just add a new one?15:46
chandankumarsee ya15:48
*** chandankumar is now known as raukadah15:48
mariosweshay|ruck: sure in 10 mins ok?15:50
weshay|ruckmarios, np15:50
weshay|ruckkopecmartin, still cool for tomorrow's demo / show and tell?15:50
*** bogdando has quit IRC15:52
kopecmartinweshay|ruck: it depends on what exactly is the expected outcome of the dome15:53
*** jtomasek has quit IRC15:58
*** udesale has quit IRC15:59
*** Trevor_V has joined #oooq16:00
mariosweshay|ruck: o/16:03
weshay|ruckkopecmartin, meh.. just a reintro to the topic, patches.. nothing new16:03
weshay|ruckbecause we haven't moved on it yet16:03
weshay|ruckmarios,  k16:03
*** TrevorV has quit IRC16:04
weshay|ruckmarios, rfolco you too if you have time https://meet.google.com/zzt-usnv-brc?authuser=116:04
rfolcoweshay|ruck, @tc call16:04
migiweshay|ruck: owalsh prepared review https://review.opendev.org/#/c/703649/216:06
weshay|ruckmigi, thanks16:09
weshay|ruckrfolco, down to16:20
weshay|ruck[zuul@wes workspace]$ cat containers-failed-to-build.log16:20
weshay|ruckqdrouterd16:20
weshay|ruckdid you get that to build?16:20
*** ykarel|afk is now known as ykarel16:20
rfolcoyes weshay|ruck16:21
rfolcotrunk.registry.rdoproject.org/tripleomaster/centos-binary-qdrouterd                       65570a311e1303d8084d12ee82868e6d0f411e8a_41250740   5ea8db777fb2   20 hours ago   539 MB16:21
jpenazbr: sorry, I was on a call. About the upload-logs role, it might be possible to iterate through the "{{ zuul.executor.log_root }}/" directory and compress all text files (or something similar). It would also be possible to honor the zuul_log_compress variable, so it works for all logs and not just for the console16:21
jpenahowever, it looks like an expensive operation to do on the executor16:21
zbri already started writing that code16:22
rfolcoweshay|ruck, If I remember right, that was the one that failed first time because epel didn't exist. It tried to disable it with yum-manager-config --disable epel16:22
weshay|ruckjpena, well.. we were doing something like that.. so now we just need to do it for rdo16:22
weshay|ruckbut that will be a second step16:22
weshay|ruckrfolco, oh.. epel on the container itself.. in disabled16:22
* weshay|ruck looks at log16:22
*** jtomasek has joined #oooq16:25
rfolcoweshay|ruck, or change this16:27
rfolcodocker/qdrouterd/Dockerfile.j2:RUN {% if distro_package_manager == 'dnf' %}dnf config-manager{% else %}yum-config-manager{% endif %} --disable epel centos-release-opstools extras16:27
rfolcowhich is more appropriate I guess16:27
rfolcolet me add this to the kolla patch16:28
rfolcoso we can ignore epel hacks16:28
weshay|ruckrfolco++16:28
weshay|ruckya.. let's add to the patch so it's repeatable16:29
weshay|ruckthank you16:29
*** holser__ has joined #oooq16:31
rfolcoweshay|ruck, what about opstools ? should we assume there will be a opstools repo ?16:31
*** holser has quit IRC16:31
weshay|ruckzbr, how do you want to handle .sh files? https://review.opendev.org/#/c/702862/16:31
weshay|rucklet's get that fixed.. .then merge this :)16:31
weshay|ruckand this.. https://review.opendev.org/#/c/703586/ when molecule passes16:31
*** tosky has quit IRC16:33
zbrweshay|ruck: how sh files ar handles is not my fault.16:34
zbri am not doing anything about them.16:34
zbrnot sure if is a bug or a feature16:34
zbrinteresting is that .bash is loaded as text16:35
*** pojadhav|away is now known as pojadhav16:36
sshnaidmzbr, maybe we can rename their extensions16:36
sshnaidmzbr, like we did with txt.gz16:36
sshnaidmzbr, if .bash is loaded as text, then from script.sh to script.sh.bash16:37
zbrsshnaidm: file bug first, announce upstream.16:37
zbri don't want more workarounds unless we have no other option16:37
sshnaidmzbr, bug?16:37
weshay|ruckzbr, yes.. I know.. re: it's not your fault16:37
weshay|ruckbut let's rename it to something that will render for now16:38
sshnaidmzbr, it's not workarounds, we already do it in log role16:38
zbrif sh is served as binary and .bash is not, is clearly an upload bug.16:38
sshnaidmzbr, lemme show you..16:38
zbrsshnaidm: i know how to rename files, no need to tell me.16:38
zbri said, that is a log server bug16:39
weshay|ruckzbr, aye..16:39
weshay|ruckzbr, and it's also inconsistent16:39
weshay|ruckbecause they rendered yesterday16:39
sshnaidmzbr, https://github.com/openstack/ansible-role-collect-logs/blob/7277ae921f66874458303061b49625aff2b0fc6c/tasks/publish.yml#L12316:39
weshay|ruckso it could be only on some cdn's16:40
sshnaidmzbr, I show you where we do it in the role, not how to rename :)16:40
weshay|ruckI'm fine w/ .sh -> .sh.log  or .sh.bash16:40
weshay|ruckwhat ever16:40
sshnaidmzbr, as you see we have a few extensions there, not only sh16:41
zbrone does not write workarounds without mentioning bugs for which these are needed, I will -2 it.16:41
weshay|rucksshnaidm, ya.. but those other extensions I think are working w/ his patch16:41
sshnaidmweshay|ruck, do all extensions work?16:42
zbrasked upstream, not desired, and fixable.16:42
weshay|rucksshnaidm, everything except .sh16:42
weshay|ruckafaict16:42
sshnaidmweshay|ruck, that's great16:43
weshay|ruckzbr, ok. so if we're fixing in infra.. let's merge ur patch16:44
sshnaidmzbr, browser considers .sh as application/x-sh format16:44
sshnaidmseems like anything except .sh would work16:44
zbrprobably hackable at client side but for convenience, better to fix upstream.16:45
weshay|ruckzbr, sshnaidm thinking I'm going to hit workflow https://review.opendev.org/#/c/702862/1016:46
weshay|ruck5.16:46
weshay|ruck4.16:46
weshay|ruck3.16:46
weshay|ruck2.16:46
weshay|ruckobjections?16:46
*** jpena is now known as jpena|brb16:46
sshnaidmweshay|ruck, it requires 2 +2s16:46
weshay|rucksshnaidm, is volunteering to ruck/rove w/o browsable logs in16:46
weshay|ruck5.16:46
weshay|ruck4.16:46
weshay|ruck3.16:47
weshay|ruck2.16:47
weshay|ruck:)16:47
sshnaidmweshay|ruck, you have 2 +2s16:47
weshay|ruck:)16:47
weshay|ruckzbr++16:47
weshay|ruckzbr, so next up other than molecule is to figure out compression for rdo16:48
sshnaidmzbr, but still wonder why do you add || true to rm16:48
zbralready did the role locally, but I am going to propose it upstream16:48
zbrsshnaidm: because rm fails when with glob.16:48
zbralternative was to use failed_when: false16:49
*** marios is now known as marios|out16:49
zbrbut better not to delay merging it with another set of 2-3h.16:49
zbrit also has tests, something missing in before16:50
sshnaidmzbr, $ rm -rf /tmp/nonexisting/* ; echo $?16:50
sshnaidm016:50
sshnaidmzbr, what do I do wrong?16:50
zbryou can try to run with quotes, or as non-root, anyway even rm can fail.16:52
sshnaidm Cannot Merge - need to rebase, "Fix set -o pipefail" was just merged16:53
sshnaidmzbr, no, it can't16:53
*** dsneddon has joined #oooq16:53
sshnaidmat least with --force16:54
sshnaidm-f, --force16:54
sshnaidm              ignore nonexistent files and arguments, never prompt16:54
weshay|ruckrlandy, fyi.. no action.. rekicking env D w/ a patch16:55
rlandyweshay|ruck: what do mean by no action?16:55
rlandypatch pls16:55
weshay|ruckhttps://code.engineering.redhat.com/gerrit/#/c/189436/16:56
rlandyperiodic-tripleo-ci-centos-7-bm_envD-1ctlr_2comp-featureset021-master queued16:56
*** marios|out has quit IRC16:56
rlandypatience16:56
rlandyweshay|ruck: wrt osp failures - we have a change in podman versions16:57
weshay|ruckah16:58
*** dsneddon has quit IRC16:58
rlandyweshay|ruck: hmm we have a node issue16:58
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/periodic-tripleo-ci-centos-7-bm_envC-1ctlr_1comp-featureset001-train/57a3303/job-output.txt16:58
rlandywhich is why your job won't start16:58
sshnaidmzbr, anyways you need to rebase16:59
rlandy2020-01-21 12:07:11.882705 | TASK [fetch-zuul-cloner-fork : Install zuul-cloner shim dependencies]16:59
raukadahweshay|ruck, rlandy zbr https://review.rdoproject.org/r/#/c/24617/ going to merge it16:59
weshay|ruckrlandy, we probably need to update required projects16:59
weshay|ruckrlandy, to include ci-scripts?16:59
weshay|ruckraukadah, ok.. dokie.. will watch for fall out17:00
zbrsshnaidm: if you rebasse it an not merge https://review.opendev.org/#/c/703597/ it will fail.17:00
zbrmolecule will not run without fixing the HOME part.17:00
rlandyyesterday we ran pretty clean17:00
rlandytoday something kicked17:00
raukadahrlandy, thank you for checking on downstream stuff17:00
rlandyweshay|ruck: wel your job started17:01
sshnaidmzbr, what is the problem with HOME part?17:01
rlandyraukadah: lol - you're thanking me??17:01
weshay|ruckrlandy, job failed on 2020-01-21 12:07:11.882705 | TASK [fetch-zuul-cloner-fork : Install zuul-cloner shim dependencies]17:01
rlandyyeah17:01
weshay|ruckbut all also failed to report17:01
rlandythat's new17:01
raukadahrlandy, yes, as marios is sleeping so17:01
weshay|ruckrlandy, but ya.. the changes that marios made to upstream17:01
weshay|ruckprobably also need to made to internal17:01
weshay|ruckw/ regards to required projects17:01
*** tesseract has quit IRC17:01
zbrpip was fixed, it was broken for ~3h17:01
rlandyit won't get anywhere if it fails that early17:01
raukadahzbr, please update the commit message on ~ home patch and other case when user home is root17:02
rlandyweshay|ruck: ha - your job passed that17:02
rlandy2020-01-21 16:59:40.356870 | TASK [fetch-zuul-cloner-fork : Install zuul-cloner shim]17:03
rlandy2020-01-21 16:59:40.893934 | primary | changed17:03
rlandypossible temp change17:03
zbri am going to include the HOME patch in gz, one to avoid the chain.17:03
raukadahzbr, also please also use the same branch topic for gz related patches so that everyone has eyes on others patches what is going on17:04
rlandyraukadah: how familar are you with latest podman changes?17:04
rlandypodman-1.4.2-6.module+el8.1.0+4830+f49150d7.x86_64 vs podman-1.6.4-2.module+el8.1.1+5363+bf8ff1af.x86_6417:05
raukadahrlandy, need to take a look based on the issues17:05
raukadahtoo much diff17:05
rlandythe issue I spoke about at the meeting17:05
rlandyrror initializing image from source docker://192.168.24.1:8787/rh-osbs/rhosp16-openstack-cinder-volume:20200110.1-updated-20200120192136: unsupported docker v2s2 media type: \"\""17:06
raukadahnot seen that one17:06
raukadahrlandy, may be zbr can help17:06
raukadahrlandy, in which job it is coming?17:06
sshnaidmzbr, what the problem are you trying to solve with HOME patch?17:07
zbrsshnaidm: you know what, please download the patch, remove the HOME part from it and run theat molecule scenario.17:07
zbrusing "tox -e molecule"17:08
raukadahrlandy, from random google search got this https://github.com/containers/libpod/issues/201317:08
rlandyyeah I read that ... opened this issue on Dec 14, 201817:08
sshnaidmzbr, no, I don't know, please show me where it fails in job17:08
rlandyversion is three days old17:08
rlandyI need to try lock the podman version17:09
raukadahrlandy, which job it is?17:09
zbrsorry but I already wasted too much effort today explaining the issue, evening giving links to tox source code, and also creating repository for you at https://github.com/ssbarnea/please-sagi/blob/master/tox.ini17:09
rlandytripleo-ci-rhel-8-standalone-rhos-1617:09
raukadahrlandy, ah17:09
rlandy2020-01-20 19:31:16 |  podman                                  x86_641.6.4-2.module+el8.1.1+5363+bf8ff1af            rhosp-rhel-8.1-appstream 12 M17:09
rlandy2020-01-20 19:31:16 |  podman-manpages                         noarch1.6.4-2.module+el8.1.1+5363+bf8ff1af            rhosp-rhel-8.1-appstream 176 k17:09
zbrrun tox there and look at what it prints17:09
rlandy^^ getting podman version17:09
sshnaidmzbr, I'd like to see where it fails in job, I'm not a tox tester17:10
weshay|ruckrlandy, interesting17:10
sshnaidmzbr, if you want to test tox, it's your right, but it's not related to CI and this role17:10
raukadahrlandy, rlandy not encountering in rhel8 podman 1.7 standlone jonb17:10
rlandydiff registry17:10
raukadahweshay|ruck, rlandy https://review.rdoproject.org/r/#/c/24605/ anfd https://review.rdoproject.org/r/#/c/24609/17:10
sshnaidmzbr, I don't ask for explanations, if you see job failure please provide logs, that's all I'm asking17:11
raukadahmay be this misses something in ^^ above review17:11
rlandyraukadah: nah - would have to set the same job up in downstream17:11
rlandywe work with the internal registry17:11
raukadahrlandy, will get the patches up tomorrow17:11
raukadahfor downstream job17:11
rlandyraukadah: is there a way to lock the podman version?17:12
rlandybasic test17:12
raukadahrlandy, may be undercloud_rpm_dependencies: >- podman == <version > might work, need to check17:14
raukadahin release file?17:14
*** jtomasek has quit IRC17:14
rlandyhmmm17:15
rlandylet me try that17:15
rlandyidk17:16
rlandycoming from rhosp-rhel-8.1-appstream17:16
raukadahlet's see what zuul says17:16
*** rfolco is now known as rfolco|brb17:16
rlandyrpm install17:16
raukadahit is going to be a simple yum install comand i think17:16
rlandyraukadah: no worries - late for you17:16
rlandywill try work it out17:16
raukadahrlandy, https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/undercloud-setup/templates/install_packages.sh.j2#L1117:17
raukadahrlandy, It will work17:17
raukadahunless inject repo part updates that17:18
raukadahi mean from repo setup scripts17:18
rlandyidk17:24
rlandyneed to finish ipa install anyways17:24
rlandywill try this later17:24
*** chem has quit IRC17:24
*** jpena|brb is now known as jpena17:26
*** chem has joined #oooq17:26
*** jtomasek has joined #oooq17:26
*** amoralej is now known as amoralej|off17:46
*** ykarel is now known as ykarel|away17:47
*** yolanda has quit IRC17:52
*** yolanda has joined #oooq17:54
weshay|ruckrlandy, https://review.rdoproject.org/r/#/c/24623/17:59
weshay|rucksshnaidm, https://review.rdoproject.org/r/#/c/24623/18:00
rlandyweshay|ruck: fallout from https://review.rdoproject.org/r/#/c/24617/?18:01
weshay|ruckrlandy, aye18:03
rlandycharming18:03
weshay|ruckrfolco|brb, fyi https://review.opendev.org/#/c/700877 https://review.opendev.org/#/c/694799 based on comments in https://review.opendev.org/#/c/703640/218:05
*** dsneddon has joined #oooq18:09
*** dtantsur is now known as dtantsur|afk18:11
*** jtomasek has quit IRC18:13
*** dsneddon has quit IRC18:14
*** jpena is now known as jpena|off18:34
sshnaidmweshay|ruck, zbr OVB logs w/o gzip take 512 MB, with gzip 158 MB18:38
sshnaidmit's most space consuming job :/18:38
zbrsshnaidm: tomorrow I will have a CR to improve upload role to archive big files, i already have the code locally18:40
zbryou only set the min size to compress, and will compress on those, easy to tune.18:40
sshnaidmzbr, that's if problem is with big files, and not with a lot of small ones18:41
sshnaidm var/log is very expensive18:42
zbrtruncate?18:42
sshnaidm301 M and 55 M18:42
*** dsneddon has joined #oooq18:45
*** dsneddon has quit IRC18:50
weshay|rucksshnaidm, zbr right.. so we need an option for 3rd party to compress18:51
weshay|ruckthat is take 218:51
zbrweshay|ruck: can we detect from inside role if we are on 3rd party?18:52
weshay|ruckzbr, it's a setting son, and each env will need a different one :)18:52
weshay|ruckzbr, let me show you18:52
weshay|ruckzbr, ensure compression is off here https://opendev.org/openstack/tripleo-ci/src/branch/master/toci-quickstart/config/collect-logs.yml18:53
weshay|ruckzbr, ensure compression is set to true https://opendev.org/openstack/tripleo-ci/src/branch/master/toci-quickstart/config/testenv/multinode-rdocloud.yml  https://opendev.org/openstack/tripleo-ci/src/branch/master/toci-quickstart/config/testenv/ovb-rdocloud.yml18:54
weshay|ruckzbr, make sense?18:54
zbrsure, put back the option that I now I ignore (artcl_gzip_only)18:54
zbrnow I want you and sagi, to test https://review.opendev.org/#/c/702862/ locally via "tox -e molecule", so you get a better hands-on experience.18:55
zbrfor exampl I was forced to include fix for bug missed by arx, https://review.opendev.org/#/c/702862/13/tasks/collect/container.yml18:56
zbrbecause just running the code uncovers bugs.18:56
weshay|ruckzbr, point me at what I need to +2 :)18:56
* weshay|ruck looks18:56
weshay|ruckzbr, eek.. please update the commit msg when ur ready18:57
weshay|ruckhttps://review.opendev.org/#/c/702862/18:57
zbri am waiting for zuul to reply to remove the skipped jobs and correct the msg18:57
weshay|ruckk18:58
weshay|ruckah.. tis slow aint it18:58
zbrrunning molecule on zuul is not really identical with local, surprises do appear from time to time18:58
weshay|ruckjust kicked18:58
zbr~8min locally, could be less than 4 if we sort the pyyaml18:58
zbrin fact i am confident that I can bring this role under 2 mins18:59
zbrwe have lots of shell scripts we do run sync, when they could be async.19:00
zbralso we could store info in a yaml file instead of so many very small files.19:00
weshay|ruck[zuul@wes workspace]$ cat containers-successfully-built.log | wc -l19:08
weshay|ruck11419:08
weshay|ruck[zuul@wes workspace]$ cat containers-build-errors.log | wc -l19:08
weshay|ruck019:08
weshay|ruckrfolco|brb, ^19:08
zbri wonder how to reply to https://bugzilla.redhat.com/show_bug.cgi?id=179017419:10
openstackbugzilla.redhat.com bug 1790174 in python-virtualenv "virtualenv activate chokes when run from bash scripts in strict mode due to missing PS1" [Medium,New] - Assigned to python-maint19:10
zbrmainly I asked to backport a bug that was fixed more than an year ago in virtualenv....19:10
*** holser__ has quit IRC19:12
zbri wonder what they will say when I mention outdated pip, or tox which is fully missing.19:13
weshay|ruckzbr, so from a ssh connection you source a virt_env and connection drops?19:13
zbrthat is a side effect because "source activate" fails19:14
weshay|ruckah this bug19:14
zbrjust because PS1 is not defined19:14
zbrthe old undefined variable issue19:14
zbrweshay|ruck: interesting pattern. another issue related to undefined variables ;)19:15
*** dsneddon has joined #oooq19:15
weshay|ruckzbr, ask me when.. I say now!!!19:16
weshay|ruckzbr, :)19:16
zbrmore of a "now?".... regex19:16
weshay|ruckzbr, fix your commit msg19:16
weshay|ruck:)19:17
weshay|ruckur going to forget19:17
zbrnope, still running again locally, found another bug.19:17
zbrwithout func tests this fix would have taken one print to get right.19:18
*** tosky has joined #oooq19:25
zbrweshay|ruck: i decided to leave sova scenario alone as it was keep causing problems, for example is not safe to run locally.19:31
zbras it would require root, which I am not going to give.19:31
weshay|ruckzbr, you have to protect the children19:33
zbrhttps://88ecc26cc7b7b4427f70-0dd3b493df614cc7e41a46754664ea83.ssl.cf2.rackcdn.com/702862/16/check/openstack-tox-molecule/a8589bb/tox/reports.html20:06
zbrthat line makes me kinda happy:         "msg": "Collected 230 files"20:07
zbri also activated the timings, maybe this will motivate others to pay more attention on where time is wasted20:08
*** jtomasek has joined #oooq20:10
weshay|ruckzbr++20:12
zbri still think that https://review.opendev.org/#/c/703597/ worths being done.20:13
zbrwe just avoided it, and I do not see any risks in it.20:14
weshay|ruckzbr, we'd have to change a lot of other defaults to match that.. in the end .. it's a tox thang20:14
weshay|ruckI have spoken20:15
weshay|ruck:)20:15
weshay|ruckzbr, fwiw https://review.opendev.org/#/c/703533/20:15
*** holser has joined #oooq20:16
zbrweshay|ruck: that is the magic index?20:16
weshay|ruckaye20:16
zbrwhy not including it in collect logs?20:16
weshay|ruckit is20:16
zbris see it part of tripleo-ci docs20:17
zbri mean part of the role20:17
weshay|ruckbecause it's part of tripleo-ci .. not a collect-logs role :)20:17
weshay|rucksay if openstack-ansible wanted to drop in their own doc20:17
weshay|ruckor infrared wanted to20:17
weshay|ruckand it was originally the footer20:17
weshay|ruckuntil infra killed that20:18
weshay|ruckI digress20:18
zbrpersonally I would have being scripted it, not hardcoded like here.20:18
zbri think that what we do not fix is the missing index on rdo, because upstream we do have it20:20
zbrand we can add stuff to it using zuul_return module20:21
zbrweshay|ruck: fyi, look at https://review.opendev.org/#/c/692356/ -- only the DATE when it originated.20:24
weshay|ruckzbr, I'm familiar20:24
weshay|ruckzbr, as core.. you need to boost reviews ;)20:29
weshay|ruckrlandy, https://review.opendev.org/#/c/703533/ as you have time.. rfolco|brb as well20:29
*** yolanda has quit IRC20:29
zbras in finding ways to bribe my mates :D20:29
weshay|ruckzbr, pretty much20:30
*** yolanda has joined #oooq20:30
rlandyin abit20:31
*** yolanda has quit IRC20:46
*** yolanda has joined #oooq20:50
weshay|ruckzbr, test report looks nice.. thanks20:52
weshay|ruckrfolco|brb, we need a planning mtg21:15
weshay|ruckbefore ths21:15
weshay|ruckthrs21:15
*** Trevor_V has quit IRC21:33
*** jtomasek has quit IRC21:38
*** jfrancoa has quit IRC22:38
*** holser has quit IRC22:54
*** holser has joined #oooq23:36
*** dsneddon has quit IRC23:39
*** fmount has quit IRC23:39
*** rfolco|brb is now known as rfolco23:41
rfolcoweshay|ruck, me you and raukadah ?23:41
rfolcopre-planning you mean23:42
*** tosky has quit IRC23:47
*** holser has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!