Wednesday, 2020-06-10

« Tuesday, 2020-06-09 Index Thursday, 2020-06-11 »
*** jmasud has joined #oooq00:06
*** Goneri has quit IRC00:08
*** rlandy has quit IRC00:12
*** Goneri has joined #oooq00:49
*** jmasud has quit IRC00:55
*** Goneri has quit IRC01:03
*** saneax_AFK has joined #oooq02:14
*** rfolco|rover has joined #oooq02:41
*** saneax_AFK has quit IRC02:49
*** jmasud has joined #oooq02:49
*** jmasud has quit IRC03:08
*** jmasud has joined #oooq03:25
*** ykarel|away is now known as ykarel04:11
*** jmasud has quit IRC04:36
*** jmasud has joined #oooq04:38
*** jtomasek has joined #oooq04:48
*** ratailor has joined #oooq04:52
*** udesale has joined #oooq04:56
*** jmasud has quit IRC05:03
*** marios has joined #oooq05:14
*** Tengu has quit IRC06:07
*** Tengu has joined #oooq06:09
*** whoami-rajat has quit IRC06:17
*** whoami-rajat has joined #oooq06:18
*** saneax_AFK has joined #oooq06:22
chandankumarysandeep, Hello06:44
chandankumarysandeep, it is defined here https://opendev.org/openstack/tripleo-image-elements/src/branch/master/elements/interface-names/install.d/70-clear-net-ifnames#L906:45
chandankumarI need to check downstream repo what is there06:46
chandankumarhere it is already sedded06:46
ysandeepchandankumar, thank you! yes we need to get rid of it in downstream06:48
ysandeepatleast for overcloud-full.qcow06:48
chandankumarysandeep, can you check the respective tripleo-image-elements repo downstream06:48
ysandeepchandankumar, yes looking currently06:49
chandankumarysandeep, https://code.engineering.redhat.com/gerrit/gitweb?p=openstack-tripleo-image-elements.git;a=blob;f=elements/interface-names/install.d/70-clear-net-ifnames;h=d276decd1b01301006bbbb984871cd1ffc246dad;hb=refs/heads/rhos-17.0-trunk-patches06:52
*** ratailor has quit IRC06:53
*** jmasud has joined #oooq06:54
*** ratailor has joined #oooq06:56
*** saneax_AFK is now known as saneax_07:00
ysandeepchandankumar, thank you07:07
*** skramaja has joined #oooq07:13
*** ccamacho has joined #oooq07:24
*** tosky has joined #oooq07:29
*** amoralej|off is now known as amoralej07:51
*** chem has quit IRC07:56
*** jpena|off is now known as jpena07:56
*** chem has joined #oooq07:58
*** jmasud has quit IRC07:58
*** dtantsur|afk is now known as dtantsur08:32
chandankumarcgoncalves, Hello08:43
chandankumarcgoncalves, I need some help on fixing this issue https://logserver.rdoproject.org/36/27636/9/check/periodic-tripleo-ci-centos-8-scenario011-standalone-master/671c94f/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz08:44
chandankumarcgoncalves, tempest.lib.exceptions.SSHTimeout: Connection to the 192.168.24.137 via SSH timed out.08:44
*** jmasud has joined #oooq08:48
*** jbadiapa has joined #oooq09:10
*** ykarel is now known as ykarel|lunch09:13
cgoncalveschandankumar, looking. something basic failed on the network side as tempest could not connect to a cirros VM via FIP09:14
*** pojadhav|ruck is now known as pojadhav|lunch09:26
*** ysandeep is now known as ysandeep|lunch09:48
*** ykarel|lunch is now known as ykarel10:04
*** pojadhav|lunch is now known as pojadhav|ruck10:14
*** sshnaidm|afk is now known as sshnaidm10:14
*** jfrancoa has joined #oooq10:15
*** jmasud has quit IRC10:17
*** jmasud has joined #oooq10:18
*** ysandeep|lunch is now known as ysandeep10:20
ysandeepmarios, o/ Hey , Could you please add https://code.engineering.redhat.com/gerrit/#/c/201168/ to your review list.. just interface name change according to new OS - will be quick .10:23
ysandeeppatch is in "Ready to Submit" state not sure what that mean.10:23
mariosysandeep: for d/stream we need to also click 'submit' for it to merge but i don't have permissions to do that so you'll have to ask ronelle later10:24
mariosysandeep: she must have just forgotten10:25
ysandeepmarios, ack o/ and thank you for sharing what "Ready to Submit" means.. I was wondering about this since morning.10:26
mariosysandeep: :) np10:26
*** derekh has joined #oooq10:49
*** jmasud has quit IRC11:01
chandankumarcgoncalves, still we are seeing same issue after removing octavia kvm env file https://logserver.rdoproject.org/36/27636/10/check/periodic-tripleo-ci-centos-8-scenario011-standalone-master/bc5d28e/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz11:39
*** jpena is now known as jpena|lunch11:40
chandankumardoes this tests triggered some where? so that we can see what is missing in our config11:40
cgoncalveschandankumar, mot sure what's causing that. I compared the tripleo vs rdo job definition and they look alike. I removed the octavia-kvm.yaml env just to make sure it was not causing problems11:51
cgoncalveschandankumar, is there any job def delta between tripleo and rdo jobs that I am missing?11:51
chandankumarcgoncalves, we have this job https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario010-ovn-provider-standalone where we run the same tests, but there it is also broken11:55
chandankumarcgoncalves, the job definitions are same there aslo https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_694/719695/5/check/tripleo-ci-centos-8-scenario010-ovn-provider-standalone/694af49/logs/undercloud/var/log/tempest/tempest_run.log11:56
cgoncalveschandankumar, ignore ovn-provider.11:56
weshay|ruckmarios, hey.. help me think of which jobs we can start removing from tripleo-ci check/gate.. SO MANY JOBS :) https://review.opendev.org/#/c/730763/11:57
weshay|ruckthat's a large wall to climb each review11:57
mariosweshay|ruck: o/ yeah that's part of the problem, maybe we can consider that if current run fails?12:01
mariosweshay|ruck: but i mean, maybe we could remove the multinodes12:01
mariosweshay|ruck: but the buildimage/containers ones we need cos that's what the change is touching...12:02
weshay|ruckmarios, ya.. not related your patch specifically.. but have been thinking about cutting down the # of jobs against tripleo-ci12:02
mariosweshay|ruck: and the scenarios too can go i think12:02
*** amoralej is now known as amoralej|lunch12:03
mariosweshay|ruck: cos of that i guess (I am touching layout.yaml) https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/standalone-jobs.yaml#L14512:03
weshay|ruckagree.. scenarios can probably go unless the featureset file changes12:04
*** derekh has quit IRC12:06
*** derekh has joined #oooq12:06
cgoncalvesbeagles, re: https://logserver.rdoproject.org/36/27636/10/check/periodic-tripleo-ci-centos-8-scenario011-standalone-master/bc5d28e/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz, tempest failed to connecto to a cirros VM via FIP. anything that stands out that we may have tweaked in the scn010 job and need also in the RDO job side?12:10
rfolco|roverarxcruz, chandankumar, sc10 keeps failing load balancer tempest test.. can we replace its test or fix it somehow ? https://bugs.launchpad.net/tripleo/+bug/188158412:11
openstackLaunchpad bug 1881584 in tripleo "SC10 train periodic job failing on LoadBalancerScenarioTest consistently" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco)12:11
cgoncalvesbeagles, https://review.rdoproject.org/r/2763612:11
beaglescgoncalves: I'll look ...12:11
cgoncalvesthank you12:11
rfolco|roverarxcruz, chandankumar last failure https://review.rdoproject.org/zuul/build/8970cb6d944e465bb35af8ff0c10716012:11
*** derekh has quit IRC12:12
*** derekh has joined #oooq12:12
rfolco|roverarxcruz, chandankumar ah wes is adding one more test here https://review.opendev.org/#/c/733038/12:13
chandankumarrfolco|rover, I am doing some experiment here https://review.rdoproject.org/r/#/c/27912/12:14
rfolco|roverchandankumar, ok let me know how it goes12:14
rfolco|roverchandankumar, let me know if you found anything or have any progress on images12:14
chandankumarweshay|ruck, rfolco|rover please remove +w12:16
chandankumarhttps://review.opendev.org/#/c/733038/112:16
weshay|ruckchandankumar, problem to have both basic-ops and whitelist octavia tests?12:17
*** rlandy has joined #oooq12:18
chandankumarweshay|ruck, it is meant for octavia testing, there is no problem but better to have octavia tests there12:18
chandankumarweshay|ruck, https://logserver.rdoproject.org/12/27912/1/check/periodic-tripleo-ci-centos-8-scenario010-standalone-master/9643be2/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz12:18
chandankumarfrom last experiment12:18
chandankumarI am finding out more tests to add there12:18
weshay|ruckchandankumar, k.. just patch on top12:19
weshay|ruckonce you have more tests... you can remove basic-ops12:19
chandankumarok12:19
beaglescgoncalves: nothing stands out ... I'll think about it though.. its' a bit weird ... the rdo job pretty closely mirrors everything12:22
chandankumarrfolco|rover, I built the image locally successfully12:24
*** derekh has quit IRC12:24
chandankumarand in the image directories are there12:24
cgoncalvesbeagles, agreed. I also compared and they are relatively the same :/12:24
*** derekh has joined #oooq12:24
rfolco|roverchandankumar, that's why I suspect its an I/O issue.... tried cache=writethrough without luck12:24
beaglesjust wondering if the standalone node is configured in some way that affects the fip behavior12:24
beagleslike at the quickstart level12:25
chandankumarsshnaidm, ^^12:25
beaglesmaybe instead of running the octavia tempest test, try just running a basic network fip test12:26
beaglescgoncalves: does this test check access to the amphora and that's what is failing or is it through access to a user VM?12:27
cgoncalvesdoes neutron have such test where the client is *outside* the cloud (i.e. not a Nova VM)?12:27
beagleshrm12:27
cgoncalvesbeagles, tempest -> cirros VM SSH access12:27
*** ratailor has quit IRC12:28
beaglescgoncalves: ack12:28
arxcruzrfolco|rover: chandankumar checking the logs, it's a connection issue, and we are using cirros 4, i believe those ssh issues are more stable on cirros 5 image12:28
beaglescgoncalves: so if we just tweaked the whitelist to run a simple neutron tempest test we might learn something12:29
chandankumarbeagles, cgoncalves I will switch to neutron tests and let's see12:29
beagleschandankumar++12:29
beaglesthanks!12:29
cgoncalvesbeagles, maybe, yes12:29
chandankumarany specific neutron tests we want to run or test network basic ops?12:29
rfolco|roverarxcruz, ok will re-test w/ new cirros... arxcruz chandankumar pls review this one https://review.opendev.org/#/c/733676/12:30
cgoncalvesbasic ops sounds enough but I don't know if it tests tempest->cirros via FIP12:30
chandankumarit exercies floating ips tests12:31
cgoncalveschandankumar, maybe just VM -> VM via FIP. I'm thinking the problem we're seeing is specifically to tempest->Nova VM12:31
*** jpena|lunch is now known as jpena12:38
chandankumarcgoncalves, ok12:40
chandankumarcgoncalves, I have added network basics ops and neutorn scenario tests let's see how it goes12:40
cgoncalvesack12:41
ysandeeprlandy, o/ hello , seem https://code.engineering.redhat.com/gerrit/#/c/201168/ will need push to submit button as well12:45
rlandydone12:46
*** Goneri has joined #oooq12:48
ysandeeprlandy, thank you :)12:50
rfolco|roverpojadhav|ruck, chandankumar weshay|ruck: I am testing ykarel's workaround - https://review.rdoproject.org/r/#/c/27998/5/playbooks/tmp.yaml --> here: https://review.rdoproject.org/r/#/c/27986/12:53
chandankumaraye12:54
*** udesale_ has joined #oooq12:54
*** rlandy_ has joined #oooq12:54
*** udesale has quit IRC12:57
*** rlandy has quit IRC12:58
*** rlandy_ is now known as rlandy12:59
*** jschlueter has quit IRC13:01
rlandyysandeep: weshay|ruck: I reworked the patches for baremetal so that they should be mergeable... pls see reviews in card https://tree.taiga.io/project/tripleo-ci-board/task/176213:02
weshay|ruckk13:02
weshay|ruckthanks13:02
rlandyI left out the adding the work for the actual virthost setup in every job13:02
rlandytakes too long13:02
*** amoralej|lunch is now known as amoralej13:02
rlandyand we do that only on reprovision13:02
rlandythe notes are there on how to do that13:02
rlandyysandeep: weshay|ruck: we are now failing on ssh to the undercloud13:03
rlandywhich I think is cloud related13:03
rlandysince we go through the zuul jumbox13:03
rlandyjumpbox13:03
ysandeeprlandy, yes.. i spend many hours today - troubleshooting undercloud unreachability for Baremetal job (even though undercloud locally was running accessible) - https://sf.hosted.upshift.rdu2.redhat.com/logs/30/201630/6/check/periodic-tripleo-ci-rhel-8-bm_envA-3ctlr_1comp-featureset001-baremetal-rhos-17/873ac5a/job-output.txt but could not get what's wrong there.. apart from n/w range change public to  private in zuul-info/host-info.primary.y13:03
ysandeepaml13:03
rlandyand it shows up consistently since we changed the network13:04
ysandeeplater read chat on rhos-ops about - PSI private networks with FIPs are not functional atm13:04
rlandywill decide what to do here after the prod chain discussion13:04
*** holser has quit IRC13:05
ysandeeprlandy, ack, I was trying to rework prework on BM - https://review.opendev.org/#/c/734654/ but later realized you posted similiar patch with a different approach.13:05
ysandeepI will abandon mine after few tests13:06
*** jschlueter has joined #oooq13:07
weshay|ruckrlandy, and now for your moment of mashugina13:07
weshay|ruckopenstack-tox-lintersFAILURE in 7m 11s13:07
* weshay|ruck runs locally13:07
*** holser has joined #oooq13:07
rlandyweshay|ruck: OMG lol - let me fix that13:08
rlandyysandeep: that's fine - good to consider all approaches13:09
rlandyBoth of them - ugh13:09
rlandyweshay|ruck: the linter is correct - fixing13:10
weshay|ruckrlandy, aye.. easy ones :)13:10
weshay|ruckrlandy, reminder, prod-chain mtg at the bottom of the hour13:12
rlandyweshay|ruck: wouldn't miss this fight for the world13:12
*** jschlueter has quit IRC13:15
weshay|ruckzuul++13:16
weshay|ruckrlandy, did you see that zuul points out the linter errors inline13:16
rlandyweshay|ruck: yes - specially for linter idiots like me13:17
weshay|ruckso cool13:17
weshay|ruckzbr cool stuff :)13:17
rlandyon my grave stone, you can write ... she put up a brave fight against linters13:17
chandankumarrlandy, does image build worked downstream?13:23
*** skramaja has quit IRC13:25
weshay|ruckrfolco|rover, how did you get this data?13:27
weshay|ruck- total of 97 hits since 06-03-202013:27
weshay|ruck- 75% in master, 25% in ussuri13:27
weshay|ruck- 70% on vexxhost, 30% on rdocloud13:27
rlandychandankumar: hey - in what context?13:28
rlandychandankumar: in the job that uploads images or on BM?13:28
zbrweshay|ruck: not sure what you refer too, i personally had only ugly discoveries today13:28
zbrweshay|ruck: but if you can confirm something to me, it would be great13:28
zbrare we allowed to switch to py36 everywhere with ci/deployment code? (even on maintenance branches).13:28
zbrbecause now we have py36 on both c7/c8, so we no longer have excuses13:28
rfolco|roverweshay|ruck, kibana13:28
rlandywrt BM, we don;t know - the job fails on ssh now :(13:28
chandankumarrlandy, on the BM13:28
rlandythe other image build seems fine13:28
weshay|ruckzbr, just highlighting improvements in zuul that are cool.. no action required13:28
rlandychandankumar: lots of issues on downstream cloud13:28
chandankumarok13:28
weshay|ruckrfolco|rover, k.. great13:28
rlandyunrelated to image build13:28
rfolco|roverweshay|ruck, https://review.rdoproject.org/analytics/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-30d,mode:quick,to:now))&_a=(columns:!(_source),index:AXJvxHvGHHNS04O3aJ0_,interval:auto,query:(query_string:(query:'%22free%20physical%22')),sort:!('@timestamp',desc))13:29
rfolco|roverweshay|ruck, then just click on fields like branch on the left side and you'll see the % charts, so cool13:30
*** ratailor has joined #oooq13:30
*** ykarel is now known as ykarel|afk13:31
zbrweshay|ruck: ahh, yeah, basically small changes that I find useful in daily tripleo work.13:32
zbri was surprised that nobody bothered to fix the "Failed to install some of the specified packages" one.13:33
zbrwhat "some" means was more of a trade secret13:33
*** ratailor has quit IRC13:42
*** TrevorV has joined #oooq14:01
*** ykarel|afk is now known as ykarel14:02
*** rlandy is now known as rlandy|mtg14:08
*** udesale_ has quit IRC14:10
*** ccamacho has quit IRC14:11
zbrwho knows what leftover goodies we still have under ci-config/jenkins folder?14:15
zbri am asking because I seen these some usage of the virtualenv and i wanted to know if we should migrate these to py3/venv14:16
chandankumarzbr, you can check with ykarel14:19
*** sshnaidm is now known as sshnaidm|bbl14:23
chandankumarcgoncalves, beagles Here is the results from neutron and network basic ops tests https://logserver.rdoproject.org/36/27636/11/check/periodic-tripleo-ci-centos-8-scenario011-standalone-master/ea111b6/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz14:25
chandankumar{0} neutron_tempest_plugin.scenario.test_floatingip.FloatingIpSeparateNetwork.test_east_west(SRC with FIP,DEST with FIP) [55.310350s] ... ok14:25
*** jschlueter has joined #oooq14:26
chandankumarbeagles, cgoncalves please review this one https://review.opendev.org/#/c/729518/ also when free14:27
rlandy|mtgsshnaidm|bbl: hey .. when you get back .. you said you edited the DNS on the router  - can you point me to where you made that change14:28
rlandy|mtgDNS Name Servers14:30
rlandy|mtg    10.11.5.19 10.5.30.4514:30
rlandy|mtgmaybe there?14:30
*** rlandy|mtg is now known as rlandy14:31
chandankumarrfolco|rover, rlandy weshay|ruck https://review.rdoproject.org/r/#/c/28042/14:35
chandankumarrlandy, marios can we merge this https://code.engineering.redhat.com/gerrit/#/c/201152/14:40
chandankumarbuild test packages downstream working fine14:40
rlandychandankumar: so we need candidate and trunk deps together?14:41
rlandyiirc, we were discussing the diff14:41
chandankumarproof : https://sf.hosted.upshift.rdu2.redhat.com/logs/84/200084/30/check/tripleo-ci-rhel-8-standalone-rhos-17/77605b6/logs/undercloud/var/log/extra/package-list-installed.txt14:42
mariosrlandy: yeah that was my question last time too... otherwise no objection here chandankumar14:42
chandankumarrlandy, we need both14:42
rlandyI meant to try that with regular standalone14:42
rlandythere was some clash with openvswitch versions14:42
rlandywhen we used used14:42
marioschandankumar: (I don't have +2 on tripleo-environments)14:42
chandankumarrlandy, marios I have answered in commit message14:42
rlandyok - let me read through14:42
chandankumarand comments on the release file also14:42
chandankumarmarios, thanks!14:43
rlandyjust in the middle of working on the network problem14:43
chandankumarrlandy, no problem take your time14:43
rlandychandankumar: I'm fine with it if it passes standalone14:43
rlandybut I want to test project for my sanity14:43
chandankumarrlandy, tested here https://code.engineering.redhat.com/gerrit/#/c/200084/14:44
rlandyon afternoon TODO list14:44
chandankumarrlandy, aye no problem14:44
rlandychandankumar: ah - I see disabled by defualt14:46
rlandyok then14:46
rlandynp14:46
rlandychandankumar: mind if I rebase that?14:47
chandankumarrlandy, nope14:47
chandankumarrlandy, go ahead14:47
rlandychandankumar: looks ok - will merge after we fix the network issue14:50
rlandydon't want to change multiple things concurrently14:50
mariosrlandy: Attempt 1 of 20 to get DLRN hash failed to get a response.14:51
chandankumarrlandy, weshay|ruck longer patch https://review.opendev.org/#/c/727200/ when free14:51
mariosrlandy: is that the error per chance? (network error)14:51
rlandymarios: probably14:51
mariosrlandy: i am geting it there https://sf.hosted.upshift.rdu2.redhat.com/logs/43/201543/15/check/periodic-tripleo-ci-rhel-8-standalone-upgrade-rhos-17/dfb944e/logs/emit_releases_file.log14:51
rlandygive me a bit14:51
mariosrlandy: k thanks14:51
chandankumarrlandy, do we have any jobs running with rhos-16?14:51
chandankumarI will test build test packages on that also14:52
rlandyI can out in a workaround to fix the problem but I would prefer to get the source fixed - hence chatting with admins14:52
rlandychandankumar: ack we do - not yet though14:52
rlandywe need to get dlrn on 1614:52
rlandylet's talk about that at tomorrow's sync14:52
chandankumaryup sure14:52
rlandychandankumar: ^^ pls bring that up14:52
*** ccamacho has joined #oooq14:54
*** ykarel is now known as ykarel|away15:01
*** ysandeep is now known as ysandeep|away15:30
weshay|ruckrfolco|rover, pojadhav|ruck thinking about promoting https://trunk.rdoproject.org/api-centos8-ussuri/api/civotes_agg_detail.html?ref_hash=b4caf97568e312eeeaa44f69efea640015:40
weshay|ruckfs35 passed, 20 failed w/ two tempest errors, fs001 failed on network... 39 passed15:40
weshay|ruckfull tempest passed in standalone jobs15:40
*** jaosorior has joined #oooq15:40
weshay|ruckthink we have a decent enough build15:41
rfolco|rovercool, thanks for covering promotions weshay|ruck15:41
rlandyweshay|ruck: https://code.engineering.redhat.com/gerrit/#/c/202943/ - just fyi ... if we can't get the nameserver sorted on a cloud level15:44
chandankumarweshay|ruck, rfolco|rover can we merge this one https://review.rdoproject.org/r/#/c/28042/15:45
*** marios is now known as marios|out15:49
weshay|ruckrfolco|rover, pojadhav|ruck ok.. ussuri is promoting15:59
weshay|ruckchandankumar++16:01
weshay|ruckchandankumar, +2 on dlrn internal.. but I guarantee16:05
weshay|ruck  pip:16:05
weshay|ruck    name: rdopkg16:05
weshay|ruck    virtualenv: "{{ build_repo_dir }}/dlrn-venv"16:05
weshay|ruck    state: latest16:05
weshay|ruck  when: not dlrn_pre_installed|bool16:05
weshay|ruckusing latest will bite us16:05
chandankumarweshay|ruck, not it will not unless we get a new release on pypi16:14
weshay|ruckright.. which we will, and we'll eventually hit something16:14
chandankumarweshay|ruck, I will check with jpena to gate DLRN and rdopkg before releasing new builds16:15
jpenachandankumar: what kind of gating would you need?16:16
weshay|ruckjpena, standalone job that automatically tries to build a package16:16
chandankumarjpena, once a new tag of dlrn or rdoinfo it should npot break build-test-packages16:16
weshay|ruckso has a depends-on injected into it16:16
chandankumar*rdopkg16:16
*** jmasud has joined #oooq16:17
weshay|ruckchandankumar, we could write a playbook that just calls build-test-packages16:17
weshay|ruckand injects a change16:17
jpenahm, we do have a job for that on every commit, see https://softwarefactory-project.io/r/#/c/18473/ (dlrn-rpmbuild-tripleo-ci-oooq and dlrn-rpmbuild-tripleo-ci-oooq-rhel8)16:17
weshay|ruckthat would be much quicker16:17
jpenamaybe we just need to adapt those16:17
weshay|ruckhot dam.. you have it16:18
weshay|ruckjpena++16:18
weshay|ruckcan we make it vote?16:19
jpenawe could if needed. In general, we never merge anything if one of those jobs fails, even if they are non-voting16:20
*** amoralej is now known as amoralej|off16:30
weshay|ruckrfolco|rover, upstream gate returning to a more stable version of itself16:37
rfolco|roverweshay|ruck, cool, I checked twice today, rate pass is over 96%16:37
weshay|ruckPASS16:40
weshay|ruck96.7%16:40
weshay|ruckFAIL16:40
weshay|ruck2.9%16:40
weshay|ruckTIMED_OUT16:40
weshay|ruck0.5%16:40
weshay|ruckUssuri Branch, Upstream Gate16:40
weshay|ruck Last 24 hours16:40
weshay|ruckPASS16:40
weshay|ruck98.2%16:40
weshay|ruckFAIL16:40
weshay|ruck1.8%16:40
weshay|ruckTrain Branch, Upstream Gate16:40
weshay|ruck Last 24 hours16:40
weshay|ruckPASS16:40
weshay|ruck98.7%16:40
weshay|ruckFAIL16:40
weshay|ruck1.3%16:40
weshay|ruckStein Branch, Upstream Gate16:40
weshay|ruck Last 24 hours16:40
weshay|ruckPASS16:41
weshay|ruck100.0%16:41
weshay|ruckRocky Branch, Upstream Gate16:41
weshay|ruck Last 3 days16:41
weshay|ruckpretty good :)16:41
rfolco|rovernice16:41
rfolco|roverweshay|ruck, look at this16:41
rfolco|roverhttps://review.rdoproject.org/r/#/c/27986/16:41
rfolco|roverykarel|away you rock16:41
weshay|ruckchandankumar, are you familiar w/ yatins patch ^16:43
rfolco|roverweshay|ruck, no missing files ... maybe we need a better config for /usr/lib/tmpfiles.d/ but this is a good workaround for now16:43
rfolco|roverchandankumar is aware weshay|ruck16:43
chandankumarweshay|ruck, yes16:44
chandankumarweshay|ruck, patch is merged now16:44
chandankumarthe job will be passing in periodic pipeline16:44
weshay|ruckrfolco|rover, chandankumar so can we cover how this fixes the issue in the next comm call?16:44
rfolco|roverweshay|ruck, https://review.rdoproject.org/r/#/c/2804116:44
chandankumarweshay|ruck, https://bugs.launchpad.net/tripleo/+bug/1882664/comments/6 explained here16:45
openstackLaunchpad bug 1882664 in tripleo "error: unpacking of archive failed on file /usr/share/ansible/plugins/modules/pacemaker_cluster.py;5eded785: cpio: open failed - Inappropriate ioctl for device" [High,Triaged]16:45
chandankumarthere were two issues16:45
chandankumarone missing files and corrupted rpms16:45
chandankumarit will fix both via workaround16:45
chandankumarneed to find a proper solution16:45
rfolco|roverweshay|ruck, I am still looking on a better fix for py3 on c7 and a less ugly hack on systemd tmpfiles.d16:45
weshay|ruckbut how does stopping the cleanup in /tmp16:45
weshay|ruckfix that?16:45
rfolco|roverthe cleanup is not in tmp weshay|ruck16:46
rfolco|rovertmp.conf could be any name16:47
rfolco|roverit is not cleaning up files older than 10d or 30d16:47
weshay|ruckwhere is the src dib to 30d16:48
chandankumarweshay|ruck, ykarel|away removed the temp cleaning condition16:48
chandankumarfrom temp.conf16:48
chandankumar /usr/lib/tmpfiles.d/tmp.conf is available in any local system generated by systemd16:49
rfolco|roverI think some config is added during the image build/ package install, coz my rdo cloud vm does not have any 20d or 30d configured on tmp.conf16:50
weshay|ruckchandankumar, OH WAIT16:51
weshay|ruckare we saying that systemd is cleaning files WHILE we're building images?16:51
rfolco|roverpossibly16:51
weshay|ruckand that's what is killing us16:51
rfolco|rover# Clear tmp directories separately, to make them easier to override16:51
rfolco|roverq /tmp 1777 root root 10d16:51
rfolco|roverq /var/tmp 1777 root root 30d16:51
weshay|ruckso the host system.. nodepool.. runs log cleanup16:51
rfolco|roverthis is what we see in a fresh vm16:51
*** marios|out has quit IRC16:52
rfolco|rovershoudn't clean /etc/pki right ?16:52
rfolco|roverso some config is being added there... I think ykarel|away knows more since he held a vm and got this conclusion16:52
weshay|ruckwell.. the image is mounted while the build is going on16:52
chandankumardib creates these dir in /tmp as a chroot16:52
weshay|ruckya16:52
weshay|ruckchandankumar, but systemd log cleanup in zuul jobs seems like something that is not needed at all16:53
weshay|rucksince the dam job only runs for 3hours max16:53
weshay|ruckzbr, ^16:53
weshay|ruckthis is nuts....16:53
rfolco|roverprivate is usualy an empty dir16:53
weshay|ruckwe knew the root cause would be nuts16:53
weshay|ruckbut DAM16:53
rfolco|roverso it might think the file is not needed16:53
rfolco|roverunused files16:54
weshay|ruckstill16:54
weshay|ruckfuck cleaning /tmp for a 3 hour lifespan16:54
weshay|ruckshould be turned off in nodepool conf16:54
weshay|ruckimho16:54
weshay|ruckrfolco|rover, chandankumar how certain are we this IS the fix16:55
weshay|ruckguess we'll see in time16:55
weshay|ruckBUT DAAAAMMMMMM16:55
rfolco|roverweshay|ruck, I'll respin my testproject, its working on both periodic and check16:56
rfolco|roverand...16:56
weshay|ruckrfolco|rover, bah.. let's just monitor the periodic jobs16:56
rfolco|roverykarel|away fixes periodic , we need similar fix for check16:56
weshay|ruckrfolco|rover, no need to keep spinning16:56
rfolco|roverok ok16:56
rfolco|roverI am paranoic now16:56
rfolco|roverweshay|ruck, just need a similar fix for check16:56
weshay|ruckykarel|away, rfolco|rover I would think we would want the cleaning of tmp files OFF in the nodepool image16:56
rfolco|roverweshay|ruck, ok, let me search for this how to completely disable systemd delete service16:57
*** derekh has quit IRC17:00
*** jpena is now known as jpena|off17:01
chandankumarhttps://questions.wizardzines.com/ is very nice to learn new things17:09
*** dtantsur is now known as dtantsur|afk17:10
*** TrevorV has quit IRC17:25
*** TrevorV has joined #oooq17:29
*** jbadiapa has quit IRC17:30
*** jmasud has quit IRC17:36
*** sshnaidm|bbl is now known as sshnaidm17:40
sshnaidmrlandy, it was dns in private-subnet settings17:40
rlandygit it - thanks17:40
rlandygot17:40
sshnaidmrlandy, didn't help much though17:40
rlandyk - we got it sorted17:41
rlandyweshay|ruck: https://bugzilla.redhat.com/show_bug.cgi?id=154585017:51
openstackbugzilla.redhat.com bug 1545850 in rhel-guest-image "rhel-guest-image contains a resolv.conf with an address" [High,Closed: duplicate] - Assigned to nobody17:51
rlandyhttps://bugzilla.redhat.com/show_bug.cgi?id=154584217:52
openstackbugzilla.redhat.com bug 1545842 in rhosp-director-images "resolv.conf contains nameserver 192.168.122.1 on rhel7.5" [High,Closed: errata] - Assigned to aschultz17:52
*** jmasud has joined #oooq17:54
*** jbadiapa has joined #oooq18:06
*** jmasud has quit IRC18:16
*** rlandy is now known as rlandy|mtg19:05
*** sshnaidm is now known as sshnaidm|afk19:05
*** jmasud has joined #oooq19:54
*** jmasud has quit IRC20:04
weshay|ruckrlandy|mtg, when you have a sec https://review.rdoproject.org/r/#/c/28044/20:11
rlandy|mtglooking20:23
*** rlandy|mtg is now known as rlandy20:23
rlandyweshay|ruck: sshnaidm|afk: Alex is right ...20:24
rlandyperiodic-tripleo-ci-rhel-8-standalone-swift-rhos-17openstack/tripleo-ci-internal-jobsmastercheck202956,31 hr 38 mins 14 secs2020-06-10 18:45:19SUCCESS20:24
weshay|rucknice20:24
rlandythat took the job time back down again ...20:25
rlandyperiodic-tripleo-ci-rhel-8-standalone-swift-rhos-17openstack/tripleo-ci-internal-jobsmastercheck202956,31 hr 38 mins 14 secs2020-06-10 18:45:19SUCCESS20:25
rlandyperiodic-tripleo-ci-rhel-8-standalone-swift-rhos-17openstack/tripleo-cimasteropenstack-component-swiftmaster2 hrs 1 min 21 secs2020-06-10 17:34:09SUCCESS20:25
rlandyweshay|ruck: hmm ... how does this work? https://review.rdoproject.org/r/#/c/28044/1/zuul.d/project-templates-components.yaml20:36
rlandywhere the name of the pipeline is repeated?20:36
rlandyzuul joins them all together?20:36
* rlandy checks rdocloud20:37
weshay|ruckthe name of the pipeline or component template?20:37
weshay|ruckthat's just the project template20:37
weshay|rucknot the zuul queue20:37
rlandywhatever it is, it works :)20:38
weshay|ruckhttps://review.rdoproject.org/r/gitweb?p=config.git;a=blob;f=zuul.d/tripleo.yaml#l1820:41
*** rfolco|rover has quit IRC21:23
*** jfrancoa has quit IRC22:07
*** jmasud has joined #oooq22:21
*** TrevorV has quit IRC22:30
*** tosky has quit IRC23:15
*** rlandy has quit IRC23:39
*** jmasud has quit IRC23:44
*** jmasud has joined #oooq23:59
« Tuesday, 2020-06-09 Index Thursday, 2020-06-11 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!