Thursday, 2020-02-27

« Wednesday, 2020-02-26 Index Friday, 2020-02-28 »
*** tosky has quit IRC00:21
*** rlandy|bbl is now known as rlandy02:56
*** d0ugal has quit IRC03:00
*** d0ugal has joined #oooq03:16
*** rlandy has quit IRC03:44
*** skramaja has joined #oooq03:50
*** jmasud has quit IRC04:13
*** jmasud has joined #oooq04:14
*** udesale has joined #oooq04:35
*** ykarel|away is now known as ykarel05:08
*** saneax has joined #oooq05:56
*** ratailor has joined #oooq06:01
*** marios|ruck has joined #oooq06:17
*** jmasud has quit IRC06:25
*** jmasud has joined #oooq06:28
ykarelraukadah, weshay|ruck i got the issue with scenario004 centos8 https://bugs.launchpad.net/tripleo/+bug/186496206:38
openstackLaunchpad bug 1864962 in tripleo "[Ussuri][CentOS8][Scenario004][rabbitmq-server-3.8] do not work with rabbitmqctl status | grep -F "{rabbit,"" [High,Triaged]06:38
ykareli have some idea, but will check with bandini about this06:39
ykarellocally i can install 004 fine with some changes06:41
*** ccamacho has quit IRC06:49
*** jfrancoa has joined #oooq06:51
*** jbadiapa has joined #oooq07:19
*** ykarel is now known as ykarel|lunch07:35
*** tesseract has joined #oooq07:53
*** jmasud has quit IRC08:12
*** raukadah is now known as chandankumar08:13
chandankumarykarel|lunch, cool08:14
*** jmasud has joined #oooq08:14
*** tosky has joined #oooq08:17
*** ccamacho has joined #oooq08:18
*** udesale has quit IRC08:31
*** udesale has joined #oooq08:32
*** amoralej|off is now known as amoralej08:43
*** jpena|off is now known as jpena08:51
chandankumarykarel|lunch, fixing release file08:56
chandankumarzbr, https://review.opendev.org/#/c/710203/09:01
*** apetrich has joined #oooq09:21
*** holser has joined #oooq09:22
pandazbr: what's the name of libselinux-python in centos-8 ?09:22
pandazbr: mmhh maybe this was already fixed and I've overwritten the code in my patch09:22
pandayep09:25
pandazbr: nevermind thanks09:25
*** ykarel|lunch is now known as ykarel09:30
ykarelchandankumar, ack09:30
*** dmellado has quit IRC09:59
*** sshnaidm|afk has joined #oooq10:17
zbro/10:20
zbrtoday and tomorrow i will be a conference, but i will do my best to do reviews10:21
pandazbr: how many peple are there in you ?10:21
pandazbr: how does it feel to project yourself ?10:22
zbrlots, but they are usually in conflict ;)10:22
sshnaidm|afkpanda, hi, can you review please https://review.rdoproject.org/r/#/c/25308/ it's blocking me in vexx migration10:29
*** sshnaidm|afk is now known as sshnaidm10:29
zbrsshnaidm: marios|ruck : this should be wf, alredy passed by i had to update the commit message.10:43
zbrhttps://review.opendev.org/#/c/710111/10:43
sshnaidmzbr, only for master?10:45
zbryou are welcomed to backport, i need to go now.10:45
zbrprobably it would work fine with cherry pick, is minimal.10:46
marios|ruckzbr: ack will check in bit10:46
* marios|ruck fooooood11:08
ykarelchandankumar, ook fulltempest centos8 working https://logserver.rdoproject.org/17/25417/1/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-master/91ffba4/logs/undercloud/var/log/tempest/stestr_results.html.gz11:39
ykarelthe failure is known and is same as centos7 one11:39
sshnaidmpanda, is the new promoter code that you write automatically updated on promoter-server?11:39
ykareltotal time taken by tempest is approx 2.5 hr11:40
ykarelneed to check how much it takes in c711:41
ykarelthere also similar, so we should be good11:42
chandankumarykarel, we can put depends on here in stnadalone full also to make sure11:43
pandasshnaidm: only the parts that are merged11:47
sshnaidmpanda, I see, can you please hold on with merges today? I'd like to reduce possible changes11:48
pandasshnaidm: It doesn't depend on me, but the only reviews that are going to be merged are the ones with the refector of legacy code taht don't include any new features. The reviews are small, fairly tested, and all tests are passing, and until now ruck and rover did not have to revert any change.11:50
sshnaidmpanda, I'm asking about today only11:51
pandasshnaidm: like I said, it doesn't depend on me, mostly on rucck/rover, the plan with wes is to merge all the refactor patches before end of sprint, so merge at least one per day, and yesterday we already missed one.12:00
*** amoralej is now known as amoralej|lunch12:08
*** jmasud has quit IRC12:12
* chandankumar headed hoem12:12
weshay|ruckykarel++12:14
*** jmasud has joined #oooq12:15
weshay|ruckchandankumar, thank you for https://review.rdoproject.org/r/#/c/25414/12:16
weshay|ruckmarios|ruck, I was pinging zbr about python-six yesterday12:17
weshay|ruckit's showing up in the upstream as well12:17
weshay|ruckhttp://dashboard-ci.tripleo.org/d/cockpit/cockpit?orgId=1&fullscreen&panelId=42212:18
zbrweshay|ruck: where? i need review, and not one that was build many days ago.12:18
marios|ruckweshay|ruck: having a s****y moment. spent a lot of today doing that https://bugs.launchpad.net/tripleo/+bug/1864646/comments/10 but https://bugs.launchpad.net/tripleo/+bug/1864646/comments/11 just now :/12:18
openstackLaunchpad bug 1864646 in tripleo "periodic standalone full-tempest-master failing neutron_tempest_plugin.scenario.test_security_groups" [Critical,Triaged]12:18
zbrthere are 3 ways to avoid the misisng six issue.12:18
* marios|ruck trashes office12:19
*** udesale_ has joined #oooq12:19
* marios|ruck brb12:19
zbrweshay|ruck: create a bug for six issue, and I will post on it the 3 approaches12:19
*** udesale_ has quit IRC12:21
chandankumarweshay|ruck, regarding container_build_id patch I am not sure that needs to handled in emit release scriot12:21
*** udesale_ has joined #oooq12:21
weshay|ruckzbr, don't see it much today upstream.. but will find examples12:21
*** udesale has quit IRC12:22
zbrweshay|ruck: i looked this morning and was not able to find more examples.12:22
weshay|ruckchandankumar, hrm.. will look12:22
marios|ruckweshay|ruck: i can file a bug in a sec12:22
bhagyashrisweshay|ruck, 1-112:22
marios|ruckweshay|ruck: zbr: i only saw the one in rdo jobs hadn't noticed it upstream12:22
weshay|ruckpanda, fyi.. did you see https://projects.engineering.redhat.com/browse/OSPPRODCH-10212:22
weshay|ruckpanda, jpena will start grooming this work soon12:23
zbreasiest fix is to install python3 and use it instead.12:23
weshay|ruckzbr++12:23
weshay|ruckmarios|ruck,12:24
weshay|ruck[11:33:02] <weshay|ruck> https://e8dabbe3d00824f512de-e25d1f8bae8601aed7ec89510f7333bc.ssl.cf2.rackcdn.com/709837/6/gate/openstack-tox-linters/ab853de/job-output.txt12:24
weshay|ruck[11:33:08] <weshay|ruck> https://ef542942f55810504313-258e10c2898eead8733af072f71435b9.ssl.cf5.rackcdn.com/702931/6/check/openstack-tox-linters/b0af1eb/job-output.txt12:24
weshay|ruck[11:33:14] * amoralej is now known as amoralej|off12:24
weshay|ruck[11:33:20] <weshay|ruck> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c38/709972/3/check/openstack-tox-linters/c38daab/job-output.txt12:24
marios|ruckweshay|ruck: did you see that         * https://bugs.launchpad.net/tripleo/+bug/1864646/comments/1112:25
openstackLaunchpad bug 1864646 in tripleo "periodic standalone full-tempest-master failing neutron_tempest_plugin.scenario.test_security_groups" [Critical,Triaged]12:25
marios|ruckweshay|ruck: zbr: filing bug for the six thing12:28
pandaweshay|ruck: nice.12:28
chandankumarweshay|ruck, how can we make container build push job permanent in periodic pipeline?12:28
weshay|ruckchandankumar, I thought it was triggered w/ a zuul var and a post playbook?12:32
weshay|ruckmarios|ruck, can tell you12:32
marios|ruckweshay|ruck: zbr: https://bugs.launchpad.net/tripleo/+bug/186500012:34
openstackLaunchpad bug 1865000 in tripleo "openstack-tox-linters failing with ModuleNotFoundError: No module named 'six'" [Critical,Triaged]12:34
zbrweshay|ruck: look at https://hackmd.io/mJtmKdsKSqG4H2IweaCfPg -- should help fixing the issue12:34
*** jpena is now known as jpena|lunch12:35
marios|ruckzbr: add relevant info to https://bugs.launchpad.net/tripleo/+bug/186500012:35
weshay|ruckzbr, ah k12:35
weshay|ruckcool12:35
*** marios|ruck has quit IRC12:36
zbrnice, before me logging in, marios posted the link, we are effective.12:36
*** pojadhav is now known as pojadhav|away12:36
*** marios|ruck has joined #oooq12:37
*** ratailor has quit IRC12:50
*** Goneri has joined #oooq12:50
ykarelweshay|ruck, marios|ruck wrt CIX discussion on ci.centos issue12:56
*** rlandy has joined #oooq12:56
ykarelisn't revert of venv fixes not helping?12:56
ykarelhttps://review.opendev.org/#/c/708264/12:56
weshay|ruckykarel, yes.. for most branches, just not queens12:57
marios|ruckykarel: it breaks multinode queens job12:57
marios|ruckykarel: but fixes the issue with the sudo12:57
marios|ruckykarel: as summarized there https://bugs.launchpad.net/tripleo/+bug/1863632/comments/712:57
openstackLaunchpad bug 1863632 in tripleo "lastest virtenv workarounds broke native tripleo-quickstart install" [Critical,Triaged]12:57
ykarelweshay|ruck, marios|ruck, ack me looks12:59
rlandyscrum @ meet.google.com/oiv-geho-ma13:00
marios|ruckthanks rlandy i was having so much fun i forgot about that13:01
rlandyweshay|ruck: arxcruz: zbr: chandankumar panda ^^13:01
rlandysshnaidm ^^13:01
*** marios|ruck is now known as marios|ruck|call13:02
chandankumarykarel, did we removed ansible hack from release file on standalone sceranrio testing13:02
ykarelchandankumar, u mean ansible-2.8.8 one?13:02
weshay|ruckarxcruz, zbr panda NU?13:03
chandankumarykarel, yes13:03
ykarelchandankumar, if yes amoralej|lunch has a WIP patch for that13:03
rlandywaiting for you all on scrum13:03
ykarelto test new deps13:03
weshay|ruckzbr, nvrmind13:03
weshay|ruckzbr, is at a conference13:03
weshay|rucksshnaidm, ?13:04
rlandypanda???13:04
sshnaidmweshay|ruck, network..13:04
*** dmellado has joined #oooq13:05
pandaI'm terrible at keeping up with meetings these days :(13:05
*** sshnaidm_ has joined #oooq13:05
*** udesale_ has quit IRC13:05
*** udesale_ has joined #oooq13:05
*** matbu has quit IRC13:06
chandankumarweshay|ruck, https://review.opendev.org/71017713:08
*** sshnaidm has quit IRC13:08
marios|ruck|callhttps://releases.openstack.org/ rlandy weshay|ruck13:11
marios|ruck|callit is extended train13:11
*** matbu has joined #oooq13:13
*** amoralej|lunch is now known as amoralej13:18
*** matbu has quit IRC13:21
chandankumarrlandy, https://review.opendev.org/#/c/710230/13:22
*** matbu has joined #oooq13:23
weshay|ruckhttps://opendev.org/openstack/tripleo-ci/src/branch/master/scripts/emit_releases_file/emit_releases_file.py#L84-L12713:24
ykarelchandankumar, ansible downgrade in release file do not work with build test packages13:24
ykarellast check in test patch failing13:25
ykareldnf --enablerepo="gating-repo" update -y13:26
ykarelbecause of ^^13:26
weshay|ruckykarel, chandankumar /me wonders if we can just fix the image in the dib?13:27
ykarelweshay|ruck, what fix in dib?13:27
*** rfolco has joined #oooq13:28
*** rfolco has quit IRC13:29
weshay|ruckthe ansible install13:31
*** jpena|lunch is now known as jpena13:34
*** matbu has quit IRC13:40
*** matbu has joined #oooq13:42
ykarelweshay|ruck, issue is we have ansible2.9 in RDO, so even if we install ansible-2.8 in image13:42
ykarelwhen we deploy ansible will get updated13:43
ykarelsoon we will remove ansible 2.9 from RDO repo13:43
ykareland with that issue will get fixed13:43
ykarelmarios|ruck|call, i commented for ci.centos issue, can we try that https://review.opendev.org/#/c/708791/413:43
ykareli tried to reproduce that issue locally with queens13:44
zbrweshay|ruck: on tripleo-upgrade there is an already existing change on linters https://review.opendev.org/#/c/710111/13:45
chandankumarykarel, https://logserver.rdoproject.org/14/25414/7/check/tripleo-ci-centos-8-multinode-1ctlr-featureset010/21313c1/logs/undercloud/home/zuul/undercloud_install.log.txt.gz13:49
chandankumarykarel, timedout13:49
chandankumarykarel, need to dig what is wrong there13:49
chandankumarwe fixed the container hash issue there13:49
ykarelchandankumar, iirc yesterday similar issue was with fs027?13:50
ykareltimeout thing13:50
chandankumarykarel, yes, but that time out at container image preapre trhis time might be different story13:50
ykarelchandankumar, okk13:50
chandankumarrlandy, https://review.opendev.org/#/c/710203/ merge it13:54
chandankumarrlandy, weshay|ruck needs reviews on c8 standalone https://review.opendev.org/#/c/710177/13:55
marios|ruck|callykarel: k, i was digging during call just now i noticed something strange about installation of virtualenv13:55
marios|ruck|callykarel: trying that now https://review.opendev.org/#/c/708791/5/roles/build-test-packages/tasks/main.yml13:55
*** marios|ruck|call is now known as marios|ruck13:55
marios|ruckykarel: can try that next 15:43 < ykarel> marios|ruck|call, i commented for ci.centos issue, can we try that https://review.opendev.org/#/c/708791/413:56
rlandylooking13:57
ykarelmarios|ruck, don't think PS5 will help13:59
ykarelas virtualenv is already installed, so the task will skip now instead of returning ok for queens13:59
marios|ruckykarel: but it doesn't skip13:59
marios|ruckykarel: that is the problem13:59
ykarelmarios|ruck, it's say [ok]13:59
marios|ruckykarel: trying to understand why13:59
marios|ruckykarel: yea so in train/stein etc it skips that cos it is already installed13:59
marios|ruckykarel: but for queens it re-installs it14:00
marios|ruckykarel: trying to understand why and if by disabling it it solves the issue14:00
weshay|ruckpanda, let's chat for a hot sec14:00
ykarelmarios|ruck, ohhk /me rechecks14:00
weshay|ruckpanda, https://meet.google.com/yhc-himt-cvq?authuser=114:00
marios|ruckykarel: commented there wrt the packages https://review.opendev.org/#/c/708791/4/roles/build-test-packages/tasks/main.yml when i checked they looked the same14:00
ykarelmarios|ruck, i meant setuptools rpm14:02
ykarelnot the pip things14:03
ykarelpip things same for all realease14:03
marios|ruckykarel: k14:03
ykareli guess the skip and ok thing can be due to setuptools behavior14:03
weshay|ruckpanda, https://opendev.org/openstack/tripleo-ci/src/branch/master/scripts/emit_releases_file/emit_releases_file.py#L84-L12714:03
marios|ruckykarel: weshay|ruck: added links there .. it re-installs virtualenv for queens so explicitly disabling it (skipped on other branches)14:04
marios|ruckweshay|ruck: ykarel: there https://review.opendev.org/#/c/708791/5/roles/build-test-packages/tasks/main.yml14:04
ykarelmarios|ruck, ack to try that, if it works will14:04
ykarelwell14:04
rlandychandankumar: weshay|ruck: promote-to-promote-components is fixed now ... https://logserver.rdoproject.org/11/25411/4/check/periodic-tripleo-centos-8-master-component-baremetal-promote-to-promoted-components/2cb1c35/job-output.txt14:04
rlandyofficially moving on from component pipeline14:04
marios|ruckykarel: yeah if it works we can try to understand more about why it happens but lets see14:05
chandankumarrlandy, sweet14:05
ykarelmarios|ruck, ack14:06
*** ykarel is now known as ykarel|away14:08
rlandyreviewing standalone now14:09
*** fmount has quit IRC14:11
rlandychandankumar: why does https://review.opendev.org/#/c/710177/ still kick tripleo-ci-centos-7-standalone rather than the centos-8 one?14:11
ykarel|awaymarios|ruck, btw issue with jenkins job is known?14:11
ykarel|awayERROR! 'static' is not a valid attribute for a IncludeRole one?14:12
chandankumarrlandy, no idea need to look into that14:12
marios|ruckykarel|away: not seen that have link?14:12
ykarel|awaymarios|ruck, in the cico patch itself https://ci.centos.org/job/tripleo-quickstart-gate-master-delorean-quick-basic/60/consoleText14:12
rlandychandankumar: k - also still kicks 7 scenarios 1-314:12
ykarel|awaymarios|ruck, looks like too old tripleo-quickstart-extras code is used there somehow14:12
chandankumarrlandy, weshay|ruck all patches added to hackmed including ykarel|away experiments also14:13
rlandychandankumar: that's great - thanks14:14
chandankumarrlandy, feel free to take over any patches14:14
rlandychandankumar: ok - ping me when are you EOD14:14
rlandywhere are we with OVB?14:14
chandankumarrlandy, no progress, there today14:15
rlandychandankumar: ok -going to add centos-8 standalone and 1-3 to master pipeline14:15
chandankumarrlandy, we just need config patch14:15
chandankumarrlandy, https://review.rdoproject.org/r/2541514:16
rlandychandankumar: merging ^^, ok?14:16
chandankumarrlandy, aye14:17
rlandychandankumar: one more quick patch coming up ... hold on14:17
chandankumaraye14:18
marios|ruckykarel|away: k one thing at at time14:18
marios|ruckykarel|away: thanks for pointing14:18
rlandyhttps://review.rdoproject.org/r/25428 Fix typo in manila component job14:21
chandankumardone14:21
rlandythanks14:21
chandankumarrlandy, do we need to fix force_periodic var also? if we want to run component job then?14:22
chandankumarthat time we need to use promote source each time in job?14:22
rlandychandankumar: no ... we run those jobs on current-tripleo14:22
rlandypromote_source was added14:23
rlandyI added it yesterday14:23
chandankumarok14:23
chandankumarcool14:23
rlandychandankumar: https://review.rdoproject.org/r/#/c/25412/14:23
rlandysee results14:23
chandankumarzbr, sshnaidm_ can we also move our all linters job to c8?14:23
chandankumarrlandy, ah thanks!14:24
rlandychandankumar: which one is the multinode review?14:26
rlandyweshay|ruck: starting a review to change the master component pipeline14:27
weshay|ruckk14:28
chandankumarrlandy, WIP, add centos-8 containers-multinode - https://review.rdoproject.org/r/2541414:28
rlandyadded centos-8 standalone jobs we just merged14:28
zbrchandankumar: not really all, i am aware about one issue caused by lack of having epel-release enabled by default14:28
rlandycool - will check that out14:28
zbrsome linters need re2-devel14:28
chandankumarrlandy, add to the etherpad14:28
zbrwhich cames from epel-release14:28
chandankumarweshay|ruck, please add all your experiments to this hackmd https://hackmd.io/HrQd03c9SxOMtFPFrq50tg?view14:28
zbrso we cannot use bindep to install it14:28
weshay|ruckchandankumar, k14:29
zbrbut if we sort this aspect we should be fine14:29
chandankumarrlandy, we got new section from amoralej also https://hackmd.io/HrQd03c9SxOMtFPFrq50tg#Moving-CentOS-8-deps-repo-to-CBS-builds-test-only14:29
rlandyI see - great - changing hackmd name14:29
chandankumaraye14:30
chandankumaramoralej, thanks!14:30
rlandyst least title14:30
weshay|ruckso this is moving from component to centos-8 ci status?14:30
chandankumarweshay|ruck, kind of14:31
rlandystriking out what we merged - on the hachmd14:31
marios|ruckweshay|ruck: master should promote lets see https://review.rdoproject.org/zuul/buildset/44b4cf00217f486ba8ee9b93fd2da8b2 (promoter hasn't picked it up yet don't see it promoting) and train is good had one today14:31
marios|ruckjust to have at least some positivity14:32
marios|ruckweshay|ruck: maybe wait for that before ripping out the pipeline? ^14:32
rlandychandankumar: can https://review.opendev.org/#/c/708606/ merge?14:33
chandankumarrlandy, yes14:33
chandankumarmarios|ruck, ^^ good to merge14:33
rlandythanks :)14:33
weshay|ruckmarios|ruck, master is staying put in rdo14:33
weshay|ruckfor now14:33
marios|ruckweshay|ruck: ah. i though you said you'd start ripping it out today14:34
marios|ruckweshay|ruck: or you mean just check/gate14:34
weshay|ruckmarios|ruck, I'm going to start removing jobs14:34
weshay|ruckin master14:34
weshay|ruckcriteria will change with and already has14:34
marios|ruckweshay|ruck: k so we'll have pipeline just reduced ack14:35
weshay|ruckaye14:35
chandankumarykarel|away, https://review.opendev.org/#/c/70995914:35
chandankumarykarel|away, please say something since we have ansible 2.8 now14:35
marios|ruckchandankumar: k permission to land14:35
marios|ruckchandankumar: (+A 16:33 < chandankumar> marios|ruck, ^^ good to merge14:36
chandankumarzbr, may be we can coordinate with delivery team to get it fixed14:36
* marios|ruck afk couple mins14:36
rlandychandankumar: can the centos-8 full-tempest job also be added to the pipeline?14:43
rlandypassing??14:44
chandankumarrlandy, not yet, till marios|ruck fix the job14:44
rlandyok14:44
marios|ruckchandankumar: which job/bug?14:44
marios|ruckchandankumar: full-tempest like master?14:44
chandankumarmarios|ruck, yes14:45
chandankumarfull temoest like master14:45
marios|ruckchandankumar: that ? https://bugs.launchpad.net/tripleo/+bug/186464614:45
openstackLaunchpad bug 1864646 in tripleo "periodic standalone full-tempest-master failing neutron_tempest_plugin.scenario.test_security_groups" [Critical,Triaged]14:45
marios|ruckchandankumar: this is first i'm hearign of it... do we have a bug for centos8 or is it the same as that ^^ neutron sec groups?14:46
chandankumarmarios|ruck, getting logs14:46
rlandyhttps://review.rdoproject.org/r/25429 Replace centos-7 jobs with centos-8 jobs in the master pipeline14:47
rlandyweshay|ruck: chandankumar: ^^14:47
rlandylet's start this thing14:47
chandankumarykarel|away, what was the review link for full tempest?14:48
chandankumarI missed that14:48
rlandyI'll take over the multinode review ... https://review.rdoproject.org/r/#/c/25414/14:50
rlandychandankumar: ^^14:50
chandankumarrlandy, aye sure14:50
rlandychandankumar: did the new centos-8 job devel go ok or was using the new parenting stuff a pain? just surveying14:51
chandankumarrlandy, good question, never tried that, but it will work14:51
chandankumarrlandy, we can fix that14:51
chandankumar(sorry)14:52
chandankumarrlandy, but for multinode it is working14:52
rlandychandankumar: :) .. wasn't a question to fix anything14:52
rlandyok14:52
chandankumarrlandy, so standalone will also work14:52
rlandyjust making sure I didn't miss any base jobs14:53
chandankumarrlandy, upstream check job was also working14:53
rlandyif you are just adding the jobs you need, we're ok14:53
chandankumargo for it14:53
rlandymarios|ruck: fyi .. https://review.rdoproject.org/r/#/c/25429/14:54
rlandycoming your way14:54
*** udesale_ has quit IRC14:57
marios|ruckrlandy: ack14:57
* marios|ruck runs for the hills14:57
marios|ruckrlandy: should be fine14:57
rlandymarios|ruck: run while you can14:57
rlandyok ... let's go for it14:57
rlandyweshay|ruck: merging https://review.rdoproject.org/r/#/c/25429/.. ok?14:58
chandankumarweshay|ruck, rlandy https://logserver.rdoproject.org/66/25366/2/check/periodic-tripleo-ci-centos-8-singlenode-featureset027-master/146db11/logs/undercloud/home/zuul/undercloud_install.log.txt.gz15:02
chandankumarweshay|ruck, rlandy if we clear undercloud install it will clear container multinode also15:02
chandankumarweshay|ruck, rlandy both got timeout at same place15:03
rlandychandankumar: will look15:03
chandankumarrlandy, we can hold the node and debug?15:03
rlandychandankumar: found what centos-7 standalone is still kicking https://review.opendev.org/#/c/71017715:03
rlandysee comment15:03
chandankumarfixing it15:04
rlandytwo places :)15:04
rlandythanks15:04
chandankumarrlandy, done15:05
chandankumarcheck gate mate15:05
marios|ruckweshay|ruck: upstream check/gate queue seems to be growing...15:05
rlandyk- watching what kicks now :)15:05
weshay|ruckmarios|ruck, k.. last I looked was fine15:06
weshay|ruckmarios|ruck, nothing red there15:06
weshay|ruckoh wait15:06
marios|ruckweshay|ruck: gate @ 12hr not terrible but just from looking at http://zuul.openstack.org/status15:06
weshay|ruckretry_limits15:06
marios|ruckweshay|ruck: and seems check is pretty long 5/6 hours?15:06
weshay|rucktripleo-ci-centos-7-undercloud-containers (3. attempt)retry_limit15:07
weshay|rucktripleo-ci-centos-7-containers-undercloud-minion (3. attempt)retry_limit15:07
weshay|rucktripleo-ci-centos-7-containers-multinode (3. attempt)retry_limit15:07
weshay|rucktripleo-ci-centos-7-standalone (3. attempt)retry_limit15:07
weshay|rucktripleo-ci-centos-7-scenario000-multinode-oooq-container-updates (3. attempt)15:07
weshay|rucktripleo-ci-centos-7-scenario001-standalone (3. attempt)retry_limit15:07
weshay|rucktripleo-ci-centos-7-scenario002-standalone (3. attempt)retry_limit15:07
weshay|rucktripleo-ci-centos-7-scenario003-standalonesuccess15:07
weshay|rucktripleo-ci-centos-7-scenario004-standalonesuccess15:07
weshay|rucktripleo-ci-centos-7-scenario007-standalonesuccess15:07
weshay|ruckopenstack-tox-pep8success15:07
weshay|ruckopenstack-tox-py36success15:07
weshay|ruckopenstack-tox-py37success15:07
weshay|ruckopenstack-tox-lower-constraintssuccess15:07
rlandychandankumar: standalone 7 and 1-3 are still kicking maybe based on the depends-on15:07
rlandychecking that15:07
marios|ruckweshay|ruck: yeah i see lots of red in upstream last jobs http://tripleo-cockpit.usersys.redhat.com/d/9DmvErfZz/cockpit?orgId=1&fullscreen&panelId=63  damn15:07
rlandyoh wait15:07
rlandyno depnds-on there15:07
rlandynvm15:08
chandankumarsshnaidm_, zbr weshay|ruck https://review.rdoproject.org/r/#/c/25092/ might be interested in this review15:08
chandankumarMove rdo-centos-[7,8] and rdo-fedora-28 nodeset to Vexxhost15:08
chandankumarrlandy, ^^ you to15:08
rlandywhat??15:09
rlandy use images in Vexxhost.15:10
rlandythat's ok15:10
rlandychandankumar: it's the rdo nodesets15:10
rlandynot used much by us15:10
weshay|ruckmarios|ruck, going to #openstack-ifnra15:11
weshay|ruckinfra15:11
chandankumarrlandy, need to verify the labels there15:12
rlandywe don;t use those labels15:12
rlandyfine15:12
chandankumarrlandy, ok then15:12
marios|ruckweshay|ruck: just joined and pinged there not sure who though15:12
weshay|ruckmarios|ruck, /me reset the one patch15:14
chandankumarweshay|ruck, rlandy anything more needed, before I cutt off15:18
weshay|ruckchandankumar, do we know why fs01 is failing so early/15:18
chandankumarweshay|ruck, zuul uuid issue 100015:18
chandankumarweshay|ruck, not looked at that15:19
chandankumarweshay|ruck, may be sshnaidm_ and tristanC knows15:19
*** goneri_ has joined #oooq15:19
rlandyweshay|ruck: same issue we have had since tuesday15:20
rlandychandankumar: you're good15:20
rlandywill follow the standalone and multinode jobs15:20
chandankumarrlandy, weshay|ruck I will catch you in your late morning tomorrow15:21
rlandyok15:21
*** goneri_ has quit IRC15:21
sshnaidm_weshay|ruck, no idea yet about ovb, trying to get it on reproducer, something is weird with image maybe..15:25
rlandychandankumar: ok - found the other centos-7 standalone defs -15:29
rlandyadding to the patch15:29
chandankumarok15:29
marios|ruckweshay|ruck: 17:22 < frickler> marios|ruck: looks like something is broken, but I cannot tell yet where. doesn't seem to affect tripleo only. infra-root: this looks bad http://zuul.openstack.org/builds?result=RETRY_LIMIT15:31
weshay|ruckaye15:33
chandankumarrlandy, you need to put more patches for c8s4 also15:33
weshay|ruckmarios|ruck, notified #tripleo15:33
rlandychandankumar: more than just adding the job?15:33
chandankumarrlandy, i mean rdo jobs and config change15:34
rlandyyep - that's fine15:34
chandankumarand upstream check one15:34
chandankumarsee ya later15:41
*** chandankumar is now known as raukadah15:41
marios|ruck weshay|ruck: k master promote finally fired half hour ago ish  2020-02-27 15:12:16,424 29157 INFO     promoter Promoting the container images for dlrn hash b5ef03c9c939db551b03e9490edc6981ff582035 on master to current-tripleo15:42
marios|ruckour last centos7 full pipeline promotion15:42
marios|rucklets take a moment everyone15:42
* marios|ruck lights candle15:42
weshay|rucklolz15:42
marios|ruckweshay|ruck: going in few for my appointment as per email pcci yesterday so will pick it up tomorrow15:44
* marios|ruck cant wait15:44
weshay|ruckmarios|ruck, 0/ ;)15:45
marios|ruckhave a good one folks15:50
*** marios|ruck is now known as marios|out15:50
*** sshnaidm_ is now known as sshnaidm15:58
*** marios|out has quit IRC16:03
*** ccamacho has quit IRC16:08
*** jmasud has quit IRC16:12
*** jmasud has joined #oooq16:13
weshay|ruckrlandy, :) https://hub.docker.com/r/tripleomaster/centos-binary-horizon/tags?page=1&name=0481ffa881dbc1aa61d9810533d8522916:18
weshay|ruckrlandy, sending the rest soon16:18
rlandynice16:18
weshay|ruckfinally getting comfortable w/ this bit16:19
rlandyweshay|ruck: possible to increase the undercloud install timeout?16:19
weshay|ruckrlandy,  locally for debug yes16:19
weshay|ruckwe shouldn't need ci to install an undercloud16:19
rlandyhttps://logserver.rdoproject.org/14/25414/7/check/tripleo-ci-centos-8-multinode-1ctlr-featureset010/21313c1/job-output.txt16:19
rlandy^^ for this job16:20
rlandyno clear error16:20
weshay|ruckya.. I know it's timing out on the undercloud install16:20
rlandyhttps://logserver.rdoproject.org/14/25414/7/check/tripleo-ci-centos-8-multinode-1ctlr-featureset010/21313c1/logs/undercloud/home/zuul/undercloud_install.log.txt.gz16:20
rlandyit just doesn't finish in time16:20
rlandyhow do I up that timeout?16:20
weshay|ruckEmilienM, here ^ but not sure if we have the logs to resolve.. was going to bring one up locally16:23
EmilienMthx16:23
EmilienMhttps://logserver.rdoproject.org/14/25414/7/check/tripleo-ci-centos-8-multinode-1ctlr-featureset010/21313c1/logs/undercloud/home/zuul/containers-prepare-parameter.yaml.txt.gz16:23
weshay|ruckrlandy, EmilienM just happened to ask in tripleo by coincidence.. I think he's got super powers and is listening to me when I sleep16:24
EmilienMso it's pulling centos7 containers uh?16:24
rlandyif he has super powers, I am sure that can do better things than listen to you sleep :)16:24
weshay|ruckEmilienM, it's not jack ass16:24
weshay|ruck:)16:24
* weshay|ruck runs16:24
rlandytag: "0481ffa881dbc1aa61d9810533d85229"16:25
EmilienM0481ffa881dbc1aa61d9810533d85229 is centos8?16:25
weshay|ruckaye16:25
rlandy^^ should be centos-816:25
rlandyyeah16:25
weshay|ruckit is16:25
EmilienMah ok16:25
EmilienMcool16:25
rlandysee the 32-bit tag?16:25
weshay|ruckEmilienM, kolla doesn't delinate centos-binary by major version16:25
weshay|ruckEmilienM, /me uploading to docker.io now..16:25
rlandythose work in standalone16:25
* EmilienM surprised but happy16:26
* weshay|ruck is glad ur happy16:26
weshay|ruckhey.. can any here tell me how this BASH thing works16:26
rlandymaybe it's juts overrunning the timeout?16:26
* weshay|ruck kids16:26
* weshay|ruck runs16:27
* weshay|ruck hides16:27
weshay|ruckrlandy, no.. I was looking for the ansible log in a different run.. and it wasn't there16:27
rlandyweshay|ruck: which run?16:27
*** tesseract has quit IRC16:27
rlandywe don't have an undercloud install anywhere else yet16:27
rlandythe previous runs on the same patch pick the wrong hash16:29
rlandyok - so let's go back to increasing the timeout on undercloud install16:37
rlandywhere do I do that?16:37
*** jpena is now known as jpena|brb16:45
*** ccamacho has joined #oooq16:53
rlandyweshay|ruck: https://logserver.rdoproject.org/14/25414/7/check/tripleo-ci-centos-8-multinode-1ctlr-featureset010/21313c1/logs/undercloud/var/log/extra/errors.txt.txt.gz17:00
weshay|ruckhrm.. that may be it.. I'll get setup now for a local17:01
weshay|ruckrlandy, fyi https://gist.github.com/weshayutin/692c80f2d4179dcf2108265bd9218bc917:01
weshay|ruckrlandy, I think you can also direct copy, but didn't try that https://gitlab.cee.redhat.com/snippets/121817:01
rlandycool - happy to see docker.io17:01
rlandythis looks safer17:01
* rlandy is all about careful17:02
rlandyweshay|ruck: in the mean time ... https://review.rdoproject.org/r/#/c/25429/17:03
rlandymight as well get this party started17:03
weshay|ruckrlandy, think you can add 004 .. just mark it non-voting17:04
rlandyok17:04
weshay|ruckoh. .this is rdo17:04
weshay|rucksorry17:04
weshay|ruckadd it scenario0417:04
weshay|ruckrlandy, wait.. let's do this together17:04
rlandylol17:04
rlandywe need to add rdo-jobs first17:04
weshay|ruckhttps://meet.google.com/cib-haeu-hkx?authuser=117:04
weshay|ruckya17:04
*** skramaja has quit IRC17:13
*** jpena|brb is now known as jpena17:23
*** amoralej is now known as amoralej|off18:14
weshay|ruckrlandy, manual undercloud install on centos-8 kicked18:18
rlandyweshay|ruck: anything interesting yet?18:18
weshay|ruckrlandy, just started.. I think the container push completed18:19
rlandyoh ok18:19
weshay|ruckrlandy, so maybe let's merge the upstream jobs.. but I can't find away to verify all the images are there18:23
weshay|ruckwe must do this in the promotion.. I know we do.. looking there18:24
rlandyweshay|ruck: working on that now ... but so far the last patch hasn't even gotten to testing18:24
rlandyadd scenario00418:24
rlandyin fact .... chat?18:24
weshay|ruckk18:24
weshay|ruckhttps://meet.google.com/pvv-dymw-kso?authuser=118:25
*** holser has quit IRC18:27
*** jpena is now known as jpena|off18:33
weshay|ruckheh.. I see what I did wrong18:58
weshay|ruckit's good to go back and install by hand every once in a while18:59
rlandybrb ...19:06
*** rlandy is now known as rlandy|brb19:07
-openstackstatus- NOTICE: Memory pressure on zuul.opendev.org is causing connection timeouts resulting in POST_FAILURE and RETRY_LIMIT results for some jobs since around 06:00 UTC today; we will be restarting the scheduler shortly to relieve the problem, and will follow up with another notice once running changes are reenqueued.19:10
*** jmasud has quit IRC19:16
*** rlandy|brb is now known as rlandy19:19
rlandyugh ... another get-hash issue19:24
rlandyfixing pipeline failure19:28
-openstackstatus- NOTICE: The scheduler for zuul.opendev.org has been restarted; any changes which were in queues at the time of the restart have been reenqueued automatically, but any changes whose jobs failed with a RETRY_LIMIT, POST_FAILURE or NODE_FAILURE build result in the past 14 hours should be manually rechecked for fresh results19:43
rlandyweshay|ruck: ping , ping20:28
rlandyhttps://logserver.rdoproject.org/25/25325/6/check/periodic-tripleo-ci-centos-8-scenario001-standalone-master/2e502bc/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz20:29
rlandyoh nvm - containers are there20:30
rlandyhttps://console.registry.rdoproject.org/registry#/images/tripleomaster/centos-binary-collectd20:31
rlandyno actually no there20:31
rlandymaybe will correct when we rerun20:32
weshay|ruckrlandy, hey20:34
weshay|ruckhrm20:34
weshay|ruckrlandy, sorry was eating20:35
weshay|ruckrlandy, so.. the release file still has rdo reg I think20:35
weshay|ruckI pushed to docker.io20:36
rlandyweshay|ruck: no problem20:36
rlandyit looks like some of the containers have the new tripleo-ci-testing hash and some do not20:36
rlandymaybe the container push job was interrupted by the zuul restart20:37
rlandythe job will rerun at the next master pipeline run20:37
rlandythsi release file is tripleo-ci-testing anyways20:38
rlandythe undercloud and multinode jobs are running again now20:38
rlandywill see what happens20:38
weshay|ruckk.. ya.. and mirror crap20:38
weshay|ruckrlandy, but which reg where they using?20:39
weshay|ruckrlandy, https://review.opendev.org/71033820:39
rlandyrdo20:39
weshay|ruckk.. so.. we know they are there20:39
rlandypromotion always uses rdo20:39
weshay|ruckright20:39
rlandywell some of them were20:39
rlandylet's see what happens on next run20:39
weshay|ruckrlandy, so maybe my patch should be first20:40
weshay|ruckwe should switch the depends on20:40
rlandyI was just retesting a periodic standalone job to see that the promote_source problem was resolved20:40
weshay|ruckfor upstream check20:40
rlandydoesn't really matter20:40
rlandybecause current-tripleo hasn't moved20:40
rlandyeither way20:40
*** jfrancoa has quit IRC20:41
weshay|ruckrlandy, ah tru20:41
weshay|ruckI flipped it though20:41
rlandyok, that's fine20:44
weshay|ruckmultinode restarted20:44
rlandyI rekicked both20:45
rlandymultinode and single node20:45
rlandywill see if they get through20:45
rlandythe parenting is wrong on both but I'll fix it if they pass20:45
weshay|ruckrlandy, k.. np .. it's day120:46
rlandyweshay|ruck: feels like day 100000000020:46
rlandyI dream in centos 8 these days20:47
weshay|ruckday 1 of back to just one distro to watch over :)20:47
rlandyfor master20:47
rlandytrain is still running on two platforms20:47
* weshay|ruck will kill rhel there too20:48
rlandyeasy there20:48
weshay|ruckheh20:48
rlandyweshay|ruck: ok - so after you're done with the killing spree, going to bring the component pipeline back to once a day not just weekend20:51
rlandypretty light actaually20:51
rlandywe could probably swing it even now20:51
weshay|ruckya21:02
*** holser has joined #oooq21:02
rlandyk - will do that tomorrow21:03
sshnaidmrlandy, weshay|ruck reproduced issue with ovb and centos821:06
sshnaidmso I can look at it later21:06
weshay|rucksshnaidm++21:07
weshay|ruckget off the keyboard :)21:07
sshnaidmbtw https://review.rdoproject.org/r/#/c/25464/21:07
rlandyoh good ..21:07
sshnaidmthe image is uploading right now to openstack-nodepool tenant21:08
sshnaidmfeel free to take it from there, need to ask kieran to make it public21:08
rlandyok - so we're good to go?21:08
rlandywill do21:08
sshnaidmrlandy, with this patch and image - probably yes, I've reproducer current ovb problem with it21:09
rlandywill try catch kforde - if not will ask him tomorrow21:11
rlandyupstream-cloudinit-centos-821:16
rlandy^^ new image?21:16
*** Goneri has quit IRC21:19
*** dpawlik has quit IRC21:43
rlandyweshay|ruck: we'er to deploying overcloud on multinode21:50
rlandybut the single node job looks stuck21:50
rlandygoing to review the depends on there21:51
*** Goneri has joined #oooq21:53
rlandywoohoo - hello multinode22:00
*** holser has quit IRC22:08
weshay|ruckrlandy, :)22:14
rlandyweshay|ruck: updated review coming up as soon as we get zuul results22:15
rlandymergeable updated review22:15
weshay|ruckrlandy, it passed22:21
* weshay|ruck goes to pick up the kiddos22:21
rlandyyeah - juts waiting for results to log22:21
rlandyso we have proof22:21
weshay|ruckk ;)22:22
* weshay|ruck watching 71017722:22
rlandyI'm big into getting it in writing22:22
weshay|ruckI'll remember that22:22
* weshay|ruck goes away kid duty22:22
rlandythere are two patches in depends on we shoudl review22:22
rlandytomorrow :)22:22
rlandyonly day one :)22:22
weshay|ruckI see one22:23
rlandyhttps://review.opendev.org/#/c/710230/22:23
rlandyhttps://review.opendev.org/#/c/710203/ will merge hopefully22:23
rlandyhttps://review.opendev.org/#/c/710196/22:23
rlandy^^ those two22:24
weshay|ruckhrm k22:24
rlandygo go22:24
rlandywill ping you tomorrow22:24
weshay|ruckk22:24
*** Goneri has quit IRC22:32
rlandytripleo-ci-centos-8-multinode-1ctlr-featureset010SUCCESS in 2h 03m 27s22:36
rlandyyay!22:36
rlandyand multinode is go!!22:36
*** owalsh has quit IRC22:58
*** jmasud has joined #oooq22:59
*** jmasud has quit IRC22:59
*** owalsh has joined #oooq23:16
« Wednesday, 2020-02-26 Index Friday, 2020-02-28 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!