Tuesday, 2019-12-10

*** ysandeep has joined #oooq00:22
*** Goneri has joined #oooq00:29
*** sshnaidm is now known as sshnaidm|afk00:35
*** ysandeep has quit IRC00:46
*** rf0lc0 has joined #oooq00:51
*** rfolco has quit IRC00:51
*** Goneri has quit IRC00:55
*** ysandeep_ has joined #oooq01:17
*** ykarel|away has joined #oooq02:34
*** dmellado has quit IRC02:45
*** dmellado has joined #oooq02:48
*** bhagyashris has joined #oooq04:23
*** skramaja has joined #oooq04:27
*** udesale has joined #oooq04:41
*** ykarel|away has quit IRC04:43
*** ykarel|away has joined #oooq04:56
*** surpatil has joined #oooq05:07
*** bhagyashris has quit IRC05:20
*** raukadah is now known as chkumar|ruck05:23
*** ykarel|away is now known as ykarel05:36
*** SurajPatil has joined #oooq05:52
*** surpatil has quit IRC05:54
*** bhagyashris has joined #oooq06:00
*** saneax has joined #oooq06:00
*** surpatil has joined #oooq06:01
*** SurajPatil has quit IRC06:03
*** akahat has joined #oooq06:09
*** jtomasek has joined #oooq06:31
*** sanjayu_ has joined #oooq06:37
*** saneax has quit IRC06:40
*** marios|rover has joined #oooq06:51
*** sanjayu__ has joined #oooq06:53
*** sanjayu_ has quit IRC06:55
*** soniya29 has joined #oooq07:07
chkumar|ruckmarios|rover: Good morning07:13
chkumar|ruckmarios|rover: I have recorded all the failure which I went through till now in promotion pipeline07:13
chkumar|ruckon etherpad07:13
marios|roverchkumar|ruck: morning ack thanks will get to it in sc07:13
marios|roversec07:13
marios|roverchkumar|ruck: fs20 seems to be a common fail/problem job on all the branches... last couple weeks its the one i had to run most often (I see you noted master fail there)07:14
chkumar|ruckmarios|rover: yes we are fs020 again for master07:15
*** jfrancoa has joined #oooq07:16
*** jbadiapa has joined #oooq07:28
chkumar|ruckmarios|rover: fs020, 030, 002 passed on master, master will promote soon07:38
chkumar|ruckon centos07:38
marios|roverchkumar|ruck: nice we are really red there07:38
chkumar|ruckmarios|rover: Let's focus on train today07:39
chkumar|ruckand rhel8 bugs07:39
*** ykarel is now known as ykarel|lunch07:47
*** bhagyashris has quit IRC08:05
*** soniya29 has quit IRC08:10
*** tosky has joined #oooq08:11
marios|roverchkumar|ruck: https://bugs.launchpad.net/tripleo/+bug/1855828/comments/1 removed alert/promotion-blocker gyi08:13
openstackLaunchpad bug 1855828 in tripleo "RHEL8 overcloud build image failed with overcloud-base has no valid mapping for package openssl-perl and lvm2" [High,Triaged]08:13
marios|roverfyi08:13
*** apetrich has joined #oooq08:16
*** sanjayu__ has quit IRC08:19
*** sanjayu__ has joined #oooq08:19
*** tesseract has joined #oooq08:27
*** amoralej|off is now known as amoralej08:32
*** jpena|off is now known as jpena08:53
*** sshnaidm|afk is now known as sshnaidm08:58
*** holser has joined #oooq09:08
*** holser has quit IRC09:08
*** holser has joined #oooq09:09
*** jbadiapa has quit IRC09:34
*** jbadiapa has joined #oooq09:35
*** derekh has joined #oooq09:37
marios|roverchkumar|ruck: new one just filed https://bugs.launchpad.net/tripleo/+bug/185585609:46
openstackLaunchpad bug 1855856 in tripleo "rocky tripleo-ci-centos-7-undercloud-upgrades fail "container-update : Check if containers-prepare-parameter.yaml exists"" [Critical,Triaged]09:46
marios|roverjfrancoa: ^^ can you have a look when you get some time please ?09:46
marios|roverjfrancoa: related to comment @ https://review.opendev.org/#/c/695234/6/config/general_config/featureset050.yml and fixes for https://launchpad.net/bugs/1853812 i think09:46
openstackLaunchpad bug 1853812 in tripleo "train containerized-undercloud-upgrades failing with "error creating container the container name is already in use "" [Critical,Fix released] - Assigned to Carlos Camacho (ccamacho)09:46
*** ysandeep_ has quit IRC10:00
jfrancoamarios|rover: ack, thanks for letting me know. I'll try to have a look at it10:06
marios|roverjfrancoa: thanks10:07
*** soniya29 has joined #oooq10:07
*** dtantsur|afk is now known as dtantsur10:15
*** derekh has quit IRC10:20
chkumar|ruckmarios|rover: ack10:21
*** ykarel|lunch is now known as ykarel10:22
*** bhagyashris has joined #oooq10:22
chkumar|ruckmarios|rover: all rhel8 train standalone _ container build except fs01 passed https://review.rdoproject.org/r/#/c/24051/10:23
chkumar|ruckmarios|rover: rocky and queens fs020 still failing?10:27
marios|roverchkumar|ruck: queens passed yesterday but not promotion (has to be manual via panda still)10:31
chkumar|ruckok10:31
marios|roverchkumar|ruck: rocky https://review.rdoproject.org/r/#/c/24035/ ok too10:31
marios|roverqueens there https://review.rdoproject.org/r/#/c/24033/10:31
*** epoojad1 has joined #oooq10:40
*** bhagyashris has quit IRC10:50
pandamarios|rover: if I aver get a +2 in my patch ....10:51
*** bhagyashris has joined #oooq10:58
*** dsneddon has quit IRC11:08
*** epoojad1 has quit IRC11:10
jfrancoamarios|rover: quick question, if in tripleo-quickstart-extras I need a template from other role. what would be the best approach? so...the containers-prepare-parameters.yaml file is rendered in undercloud-deploy. in containers-update, I need to render that file in the case it does not exist (as that's the case when upgradeing from Queens to Rocky)11:12
jfrancoamarios|rover: but I don't want to simply copy it from one role to the other...as that would imply duplicating code...any suggestion?11:12
jfrancoasshnaidm: ^^^ ?11:13
marios|roverjfrancoa: only thing i can think of is adding a task list in the role you need to copy from, that renders the template as a standalone task then call it in the one you are writing?11:14
marios|roverjfrancoa: isn't the reason containers-prepare-parameters isn't rendered is because we have containerized_undercloud: false11:15
jfrancoamarios|rover: right, that would be a good option. Isolate the file rendering so we can call it from other roles11:15
marios|roverjfrancoa: wrt containerized-undercloud-upgrades vs undercloud-upgrades jobs... for rocky, i think we should only keep the containerized version? or is it starting non containerized then upgrading to containerized?11:15
jfrancoamarios|rover: but even if we would set it to true, we will need to create the file as that didn't exist in queens11:16
marios|roverjfrancoa: so kind of neither of those... or more like we just have to fix the undercloud-upgrades job to start non & upgrade to containerized11:16
marios|roverjfrancoa: 13:16 < jfrancoa> marios|rover: but even if we would set it to true, we will need to create the file as that didn't exist in queens11:16
marios|roverjfrancoa: ack ^^ ok i think that's what i meant by 'kind of neither of those'11:17
*** dsneddon has joined #oooq11:17
jfrancoamarios|rover: I guess the fix would be to create a containerized_undercloud_upgrade variable and then add the logic to create the file during the upgrade process11:17
jfrancoamarios|rover: exactly, in rocky we deploy Queens undercloud (non-containerized) and upgrade it to Rocky (containerized). That's why I say about the two variables (one to define if the undercloud we deploy is containerized or not, and another to define to what type of undercloud we upgrade to)11:19
marios|roverjfrancoa: ack this is the approach you suggested at comment @ https://review.opendev.org/#/c/695234/6/config/general_config/featureset050.yml11:19
jfrancoamarios|rover: right. So, I will try to create the variable, link it in the multinode-undercloud-upgrade playbook and then take care of the containers-prepare-parameters.yaml creation during upgrade ( I think what we said, isolate the rendering and reference it should work)11:21
marios|roverchkumar|ruck: nice master centos promoted11:21
marios|roverchkumar|ruck:++11:21
jfrancoamarios|rover: thanks a lot for the help!11:21
marios|roverjfrancoa: np man thanks for checking there let me know when you need reviews or if there is something for us to do11:21
*** dsneddon has quit IRC11:23
*** holser has quit IRC11:24
*** holser has joined #oooq11:25
*** holser has quit IRC11:29
*** epoojad1 has joined #oooq11:30
marios|roverchkumar|ruck: err... any idea why openstack-periodic-master didn't run today yet?11:39
*** rf0lc0 has quit IRC11:40
*** bhagyashris has quit IRC11:42
*** bhagyashris has joined #oooq11:42
*** ykarel is now known as ykarel|away11:43
chkumar|ruckmarios|rover: checking11:50
chkumar|ruckmarios|rover: as per this 10 0,12,18 * * *' it will start in another 50 mins i think11:52
marios|roverchkumar|ruck: right but it didn't run midnight11:53
marios|roverhttps://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master nothing from today11:53
chkumar|ruckmarios|rover: yes, it just runned midnight, so other jobs got passed11:53
chkumar|ruckbut nothing today11:53
*** dtantsur is now known as dtantsur|ikea11:54
marios|roverchkumar|ruck: but 'midnight' should be today :D11:54
chkumar|ruck:-)11:55
marios|roverchkumar|ruck: i think there might be something wrong. it will start in about 5 mins fine but it seems we are missing one run per day11:56
marios|roverchkumar|ruck: for example see https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-7-master-promote-consistent-to-tripleo-ci-testing11:56
marios|roverchkumar|ruck: 2 runs / day but it should be 3 (last few days) ... on 5th and earlier looks more normal11:57
chkumar|ruckmarios|rover: from 6th it has changed something there11:57
chkumar|ruckneed to check11:57
*** dsneddon has joined #oooq11:58
chkumar|ruckthere is no change there11:59
chkumar|ruckchekcing on sf-ops11:59
*** SurajPatil has joined #oooq12:00
*** udesale has quit IRC12:01
*** weshay_pto is now known as weshay12:03
*** surpatil has quit IRC12:03
marios|roverchkumar|ruck: o/ i just went to update that for merge conflict https://review.opendev.org/#/c/693159 (from http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-train-containers-build-push/ebff469/job-output.txt12:12
marios|roverchkumar|ruck: i saw you updated it, but why is the master job pulling the train patch? shouldn't it be https://review.opendev.org/#/c/669471 based on https://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/vars/main.yaml#L1412:12
marios|roverchkumar|ruck: or maybe i missed something there12:13
*** holser has joined #oooq12:19
*** epoojad1 is now known as epoojad1|afk12:20
weshaymarios|rover, chkumar|ruck good day sirs12:21
weshaywe need to figure out how to end your shift as ruck rovers12:21
*** amoralej is now known as amoralej|lunch12:25
marios|rovero/ weshay what do you mean? end of sprint wednesday, new ruck|rovers :D what do we need to work out ? Or are we extending the sprint?12:25
*** epoojad1|afk has quit IRC12:28
weshaymarios|rover, your shift should end wednesday.. but the sprint has an extra week12:30
chkumar|ruckweshay: welcome back12:30
chkumar|ruckmarios|rover: looking12:30
chkumar|ruckmarios|rover: nice catch12:31
pandamarios|rover: retrying queens now, thanks chkumar|ruck for +2.12:31
marios|roverchkumar|ruck: master and train running again now .. but mainly question is why is it grabbing train patch on the master container build http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-train-containers-build-push/ebff469/job-output.txt12:31
chkumar|ruckmarios|rover: few days back kolla people complained about the wron testing12:31
pandamarios|rover: you want me to add f20 to the criteria12:31
pandamarios|rover: or f00112:31
*** surpatil has joined #oooq12:32
marios|roverpanda: ah i thought you were using upstream criteria12:32
*** brault has joined #oooq12:32
weshaybhagyashris, you available?12:32
marios|roverpanda: so fs20 was the only one missing yesterday https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-wednesday-weekend%2012:32
marios|roverpanda: fs20/fs1 are both green12:32
bhagyashrisweshay, yes12:32
marios|roverpanda: (fs20 with that https://review.rdoproject.org/r/2403312:33
pandamarios|rover: ok, adding both. What are upstream criteria  I'm using what's in the repo...12:33
marios|roverpanda: yeah thats what i mean fine12:34
*** jpena is now known as jpena|lunch12:34
pandamarios|rover: ok using repo criteria and readding fs20 and fs00112:34
pandamarios|rover: fire in the hole!12:34
chkumar|ruckmarios|rover: I am confused, above log link is from train12:34
*** SurajPatil has quit IRC12:34
marios|roverpanda: ah i see fs20 is commented out in fact https://github.com/rdo-infra/ci-config/blob/b0f9de0de7a676750ead6c6a7e0055b70ac28fe3/ci-scripts/dlrnapi_promoter/config/CentOS-7/queens.ini#L2912:34
*** holser has quit IRC12:34
marios|roverpanda: but fine it should promote thanks12:34
marios|roverpanda: so is fs112:34
marios|rover(commented out)12:34
*** holser has joined #oooq12:35
chkumar|ruckmarios|rover: in train, train patches are pulled and in master http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-master-containers-build-push/699e4e5/job-output.txt12:36
pandamarios|rover: ok, now waiting for container psuh, this might take a while as we are pushing to docker and quay. I'll keep you posted.12:37
marios|roverchkumar|ruck: ok sorry maybe i mixed the logs then12:44
chkumar|ruckmarios|rover: no problem12:45
marios|roverpanda: thanks12:45
chkumar|ruckmarios|rover: I need to get rid of those patches now, may be takling tomorrow12:45
marios|roverchkumar|ruck: but how?12:45
marios|roverchkumar|ruck: merge them?12:45
chkumar|ruckmarios|rover: yes12:45
marios|roverchkumar|ruck: ack cool12:45
chkumar|rucklet's get it merged12:45
kopecmartinchkumar|ruck: I proposed patches for tempestconf update but feature sets 17 (for train) and 20 (queens and rocky) are failing .. is it worth to keep rechecking? or is there any known issue?12:46
chkumar|ruckkopecmartin: need to check12:46
chkumar|ruckkopecmartin: pass me the reviews, I will take a look12:46
kopecmartinchkumar|ruck: https://review.rdoproject.org/r/#/q/project:rdoinfo+owner:%22Martin+Kopec+%253Cxkopec42%2540gmail.com%253E%22++status:open12:47
chkumar|ruckkopecmartin: will take a look in some time12:47
kopecmartinchkumar|ruck: sure, it's not that urgent12:47
chkumar|ruckzbr: hello, need one help as infrared expert, please have a look at this patch https://review.opendev.org/#/c/682042/12:48
chkumar|ruckssbarnea: ^^12:48
zbrdone12:57
chkumar|ruckzbr: thank you :-)12:57
zbrchkumar|ruck: i am not an expert, but is easy to spot that added code is not tested on ci.12:58
*** bhagyashris has quit IRC12:58
zbrthat is big issue, ok to add plugin, as long you add a test that proves that intrared can recognize what you did.12:58
*** sanjayu__ has quit IRC13:01
*** rlandy has joined #oooq13:01
chkumar|ruckmarios|rover: regarding master periodic pipeline as the jobs defined there does not get finished so the next trigger fire got skipped13:06
marios|roverchkumar|ruck: ack thanks replied in sf-ops ... interesting... then maybe we need to move the time13:07
chkumar|ruckmarios|rover: let me fix that13:07
marios|roverchkumar|ruck: ack13:07
chkumar|ruckmarios|rover: before that let me check rhel8 pipeline13:08
marios|roverchkumar|ruck: ack the rhel8 buildimage faild13:08
chkumar|ruckthis image is now a 'thing' which always fail13:09
chkumar|ruckmarios|rover: panda sshnaidm rlandy https://review.rdoproject.org/r/#/c/24058/13:15
chkumar|ruckserious issue13:15
pandachkumar|ruck: does it run on oldest image, or it doesn't run at all ?13:16
marios|roverchkumar|ruck: is that how we had the missing overcloud-full https://bugs.launchpad.net/tripleo/+bug/1855826/comments/113:16
openstackLaunchpad bug 1855826 in tripleo "RHEL 8 master fs01 fails while getting image expected checksum after multiple retry" [Critical,Confirmed]13:16
chkumar|ruckpanda: yes13:16
chkumar|ruckmarios|rover: yes13:17
pandachkumar|ruck: yes to what ? :)13:17
marios|roverchkumar|ruck: ok but why is it different to centos though13:17
marios|roverpanda: it runs but misssing images see https://bugs.launchpad.net/tripleo/+bug/1855826/comments/113:17
chkumar|ruckpanda: the job was running, trying to get older images but rhel8 fs01 train never runned13:17
chkumar|ruckeither it get failed on images fetching13:17
*** soniya29 has quit IRC13:18
pandachkumar|ruck: marios|rover it's different, if it gets older images, it's a big bug,as it could lead to false positive. The job should look for an hash, and fail if it doesn't find it, not use whatever tripleo-ci-testing points to. If instead it fails for missing images is bad but not uper bad.13:19
rlandychkumar|ruck: let comment on https://review.rdoproject.org/r/#/c/24058/1/zuul.d/tripleo.yaml13:19
rlandyleft13:19
chkumar|ruckmarios|rover: in centos side, we donot consume the image built by fs02 job13:20
chkumar|ruckin fs0113:20
marios|roverchkumar|ruck: ah thanks13:21
chkumar|ruckrlandy: done13:21
marios|roverchkumar|ruck: add the bug too please13:22
*** bogdando has joined #oooq13:23
chkumar|ruckTripleo community meeting happening in another 5 mins add agenda here: https://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw?view13:25
*** bogdando has quit IRC13:26
chkumar|ruckrlandy: done13:28
rlandychkumar|ruck' w+13:29
weshaycommunity call starting in 1 min13:29
chkumar|ruckbaha: mjturek Hello, time for community call13:31
mjturekomw13:32
bahaThanks for the heads up13:33
*** jbadiapa has quit IRC13:33
*** jpena|lunch is now known as jpena13:37
*** bogdando has joined #oooq13:39
*** bogdando has quit IRC13:41
*** bogdando has joined #oooq13:41
*** bogdando has quit IRC13:41
*** Goneri has joined #oooq13:43
*** amoralej|lunch is now known as amoralej13:44
chkumar|rucksshnaidm: in scenario 12, are we using ovn or ovs?13:53
sshnaidmchkumar|ruck, I don't see them: https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario012-standalone.yaml#L23:L2313:55
sshnaidmchkumar|ruck, this is the new: https://review.opendev.org/#/c/682410/8/ci/environments/scenario012-standalone.yaml13:56
*** jbadiapa has joined #oooq13:59
*** dtrainor has joined #oooq14:00
*** ykarel|away has quit IRC14:04
*** dsneddon has quit IRC14:12
*** dsneddon has joined #oooq14:15
zbrwho can give me a dlrn api end-point that I can query without credentials?14:20
*** dsneddon has quit IRC14:20
*** holser has quit IRC14:21
*** epoojad1 has joined #oooq14:22
*** dsneddon has joined #oooq14:22
*** holser has joined #oooq14:23
weshaymigi, rlandy https://meet.google.com/wxs-hpcw-uoa14:25
migiweshay: need 4min14:25
weshayanyone else.. going into some details of the component pipeline14:25
chkumar|ruckmarios|rover: heading home14:26
*** chkumar|ruck is now known as raukadah14:26
raukadahmarios|rover: rlandy panda sshnaidm https://review.opendev.org/#/c/698233/ if still in mood of merging14:26
weshayhttps://docs.google.com/drawings/d/1-2nnnwgjMg6N3bkPSplzjtWgLYBwwrcdVK2Vv1qzVKk/edit14:30
marios|roverraukadah: ack14:30
*** dsneddon has quit IRC14:33
*** surpatil has quit IRC14:36
*** ykarel|away has joined #oooq14:40
*** ykarel|away is now known as ykarel14:40
*** epoojad1 has quit IRC14:40
*** dsneddon has joined #oooq15:00
*** dsneddon has quit IRC15:05
pandamarios|rover: weshay queens promoted, 2:30 to push to both registries.15:07
weshaynice15:07
weshaypanda, maybe we can chat tomorrow for 15-30 min about the promoter?15:07
marios|roverpanda: thanks15:08
pandaweshay: sure15:08
weshaycool man.. nice work on queens15:09
panda55 minutes to push to docker.io 1:15 to push to quay.io15:09
*** TrevorV has joined #oooq15:10
marios|roverpanda: i mean it usually takes just over 2 hours to do 'normal' push i mean on old/current promoter15:10
*** rascasoft has joined #oooq15:12
raukadahmarios|rover: train fs020 failed15:12
marios|roverraukadah: damn... was watching that15:13
marios|roverraukadah: its always tempest fs20 across all branches but then they pass in testproject...15:13
raukadahmarios|rover: let me check it is again ovn controller related15:14
marios|roverraukadah: but otherwise that run is very green15:14
marios|roverraukadah: so testproject for fs20 will help us15:14
marios|roverraukadah: would be nice to get another T promote for tomorrow's call (but we're green there anyway )15:14
raukadahweshay: https://bugs.launchpad.net/tripleo/+bug/1855828 need some help on to look why it is failing15:17
openstackLaunchpad bug 1855828 in tripleo "RHEL8 overcloud build image failed with overcloud-base has no valid mapping for package openssl-perl and lvm2" [High,Triaged]15:17
raukadahsshnaidm: regarding scenario 12 , i am directly porting the job to os_tempest15:17
marios|roverraukadah: that bug does not fail the container build though https://bugs.launchpad.net/tripleo/+bug/1855828 ?15:18
openstackLaunchpad bug 1855828 in tripleo "RHEL8 overcloud build image failed with overcloud-base has no valid mapping for package openssl-perl and lvm2" [High,Triaged]15:18
raukadahmarios|rover: it is overcloud image build15:19
raukadahdifferent from container build15:19
marios|roverraukadah: k same though15:19
marios|roverraukadah: i added comment 1 there earlier (i thought you saw it already)15:20
marios|roverraukadah: it is non fatal error15:20
marios|rover10:13 < marios|rover> chkumar|ruck: https://bugs.launchpad.net/tripleo/+bug/1855828/comments/1 removed alert/promotion-blocker gyi15:20
openstackLaunchpad bug 1855828 in tripleo "RHEL8 overcloud build image failed with overcloud-base has no valid mapping for package openssl-perl and lvm2" [High,Triaged]15:20
raukadahmarios|rover: ah yes forgot about that15:21
*** skramaja has quit IRC15:24
raukadahweshay: I am removing smoke test from standalone check job as it takes too much time on OSA side15:28
sshnaidmraukadah, great15:28
raukadahwe will be running smoke only in periodic or even better in full tempest standalone job15:28
raukadahweshay: what do you say?15:28
*** akahat has quit IRC15:29
weshayraukadah, which job?15:29
raukadahweshay: https://review.opendev.org/#/c/697992/15:30
raukadahweshay: https://review.opendev.org/#/q/topic:move_to_plain_standalone+(status:open+OR+status:merged)15:31
raukadahweshay: tripleo-ci-centos-7-standalone15:32
raukadahit replaces ostempest initial job15:32
raukadahsshnaidm: regarding https://review.opendev.org/#/q/topic:rhel8-containers-build-stable/train+(status:open+OR+status:merged) patches, will I create seperate patch to get rid of these patches?15:36
raukadahin last 2 weeks, we have to do rebases15:36
marios|roverraukadah: did you already tweak the openstack-periodic-master cron time (midnight) or should i ? (you said earlier you were looking at it)15:36
raukadahmarios|rover: not yet15:37
raukadahmarios|rover: please go ahead15:37
marios|roverraukadah: k going to post now15:37
raukadahrlandy: make it 8 hours15:37
sshnaidmraukadah, what exactly? which separate patch?15:37
raukadah3 times a day15:37
sshnaidmraukadah, you mean just combining them into one?15:38
raukadahsshnaidm: yes15:38
rlandyraukadah:?15:38
raukadahsshnaidm: https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/build-containers/vars/main.yaml#L14 O mean to get rid of this var15:38
sshnaidmraukadah, maybe, go for it.. I'll look also what can be done there..15:39
raukadahsshnaidm: will get the patches up tomorrow15:39
raukadahrlandy: sorry15:39
sshnaidmraukadah, ack, add me to review15:39
raukadahmarios|rover: makes it for 8 hours for periodic master pipeline15:39
raukadahrlandy: it was for marios|rover15:39
marios|roverraukadah: yeah but we will have to change the 1800 one too15:39
marios|roverraukadah: let me post and we discuss on review15:40
raukadahyes15:40
*** dsneddon has joined #oooq15:40
marios|roverraukadah: posted fs20 train there https://review.rdoproject.org/r/2406215:45
raukadahmarios|rover: cool15:47
*** ykarel is now known as ykarel|afk15:50
raukadahmarios|rover: going to look stuff, I will make sure we promote train, see ya have a nice evening ahead to all, cc panda weshay sshnaidm zbr rlandy15:56
raukadahi mean tomorrw15:56
marios|roverraukadah: o/15:57
*** dtantsur|ikea is now known as dtantsur16:02
weshaysshnaidm, which job in https://code.engineering.redhat.com/gerrit/#/c/185863/ was rdo-cloud?16:08
weshaymjturek, any way to patch us in w/ https://tmate.io/16:14
weshay?16:14
mjtureknot sure yet :-\ once we have the held node we'll play around and see16:18
*** marios|rover has quit IRC16:27
*** d0ugal has joined #oooq16:31
mjturekweshay I don't think so, cico nodes seem to require access to the jump box16:31
mjturekjpena might have access16:32
mjturekI do, and baha as well16:32
*** d0ugal has quit IRC16:36
rlandyweshay: hi - for overcloud keystone containers ... https://opendev.org/openstack/tripleo-common/src/branch/master/container-images/overcloud_containers.yaml ...16:44
rlandydo I just need docker.io/tripleomaster/centos-binary-keystone:current-tripleo16:44
rlandyor docker.io/tripleomaster/centos-binary-barbican-keystone-listener:current-tripleo as well?16:44
*** jpena is now known as jpena|brb16:45
weshayrlandy, what's the context?  are you putting up patches to build the keystone containers?16:46
weshayrlandy, that is not needed at this time16:46
rlandyweshay: ack  ...16:46
rlandyequivalent to compute job16:46
weshayit's just extra credit16:46
rlandyI think it's should be easy enough16:46
rlandyfound the files marios changed for compute16:46
weshayrlandy, there are a number of keystone containers16:47
rlandyweshay: ack - I searched dockerhub16:47
*** sanjayu__ has joined #oooq16:47
rlandy(bonus for having python script to search containers)16:47
rlandybut https://opendev.org/openstack/tripleo-common/src/branch/master/container-images/overcloud_containers.yaml#L90 only listed two16:47
rlandyfor the overcloud16:47
rlandyfollowing marios' pattern16:48
rlandyright now container build jobs is failing due to missing file16:48
weshayhrm.. actually.. it may just be the one.. container after further review16:49
sshnaidmweshay, wrt https://code.engineering.redhat.com/gerrit/#/c/185863/ - it's upshift all16:50
weshayrlandy, for standalone it looks like just centos-binary-keystone16:50
*** jaosorior has joined #oooq16:50
rlandyweshay: ack16:51
rlandyweshay: https://review.opendev.org/#/c/698295/16:52
*** ykarel|afk is now known as ykarel|away16:53
*** tesseract has quit IRC16:58
*** ykarel|away has quit IRC17:08
*** sanjayu__ has quit IRC17:18
*** jpena|brb is now known as jpena17:24
*** tosky has quit IRC17:27
rlandyweshay: ok - so help me out here ... https://sf.hosted.upshift.rdu2.redhat.com/logs/66/186766/9/check/periodic-tripleo-ci-rhel-8-standalone-rhos-17-compute-rhos-16/e0d4844/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz17:30
rlandysearch for rhosp16-openstack-nova-libvirt17:30
rlandydid this update?17:30
zbrweshay: i got a flu, so i may be slowed down a bit these this.17:31
zbrweek17:31
weshayzbr, k.. so for next week.. and the rest of the year.. are you willing to take on ruck / rover and let raukadah and marios take a break?17:32
zbri will be back only on 26, i am in Spain / offline next week :D17:33
zbrso clearly not next week.17:33
zbrbut after 26, yes.17:33
raukadahweshay: I am ok to cover next week, but not after that17:33
rlandystdout: 257decbbd25cb85b0c7b70c78026b1a160fe71e175394c267def530c046f54b417:33
zbrrlandy: is this you bitcoin wallet password?17:34
rlandyContainerNovaLibvirtImage: 192.168.24.1:8787/rh-osbs/rhosp16-openstack-nova-libvirt:20191206.1-updated-2019121010412617:34
rlandyzbr: yes - go ahead and everything in there17:34
rlandyneed to drop off lease ... brb17:36
*** rlandy is now known as rlandy|brb17:36
*** dtantsur is now known as dtantsur|afk17:38
weshayrlandy|brb, 2019-12-10 16:09:31,093 9438 INFO tripleo_common.image.image_uploader [  ]   stderr: 'Error: Unknown repo: '''''17:41
*** holser has quit IRC17:42
*** rlandy|brb is now known as rlandy17:45
rlandyyeah so I see that17:46
rlandybut it should update with the compute repo17:46
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/66/186766/9/check/periodic-tripleo-ci-rhel-8-standalone-rhos-17-compute-rhos-16/e0d4844/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz#_2019-12-10_16_12_06_17617:47
*** holser has joined #oooq17:48
*** sshnaidm is now known as sshnaidm|afk17:50
*** holser has quit IRC17:53
*** holser has joined #oooq17:54
rlandytrying with enabled repos17:58
*** jaosorior has quit IRC17:59
*** holser has quit IRC18:01
*** rf0lc0 has joined #oooq18:22
*** kopecmartin is now known as kopecmartin|off18:41
*** jpena is now known as jpena|off18:42
rlandyugh - downstream dlrn http vs. https18:52
mjturekweshay is there a person we should try to get jump box access for?19:07
*** amoralej is now known as amoralej|off19:07
mjturekI'm sure the centos folks would let us but we'd need to put in a request19:07
weshaymjturek, so.. at this time of day19:07
weshayit's just me and rlandy19:07
weshayand Folco but he's out today19:08
mjturekthat's fair, I guess I meant in general, like even if it's not today19:08
weshaymjturek, I was hoping that folco could be your perm point of contact, and he could bring in others as needed19:08
weshayyou can always ping me if all else fails19:09
mjturekweshay sounds good, do you know around when he;s returning?>19:09
weshayhe should be back tomorrow19:09
mjturekoh that's quick19:09
weshayworks east coast hours19:09
weshayrlandy, sorry I missed your pings19:09
mjturekas do I :)19:09
weshaymjturek, not much to go on w/ 2019-12-09 18:25:59 | INFO:kolla.common.utils.base:[91mcurl: (7) Failed connect to 172.17.0.1:80; No route to host19:13
rf0lc0mjturek, o/19:14
rlandyweshay: I pinged?19:15
weshayrlandy, well.. you were typing out loud19:15
rlandyweshay: sorry - just talking to myself19:15
rf0lc0mjturek, any luck on the failure above ? whats the latest?19:15
weshayrf0lc0, this look fine to you? https://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1821/logs/logs/kolla-build.conf19:16
rf0lc0looking19:16
weshayya.. identical to ours http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build-push/153fe48/conf/kolla-build.conf.txt.gz19:17
rf0lc0http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build-push/153fe48/conf/kolla-build.conf.txt.gz19:18
rf0lc0oops19:18
rf0lc0ok19:18
rf0lc0except byt19:19
rf0lc0ppc64le tag suffix19:19
weshaymjturek, nothing built successfully?19:19
weshayhttps://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1821/logs/logs/containers-successfully-built.log19:19
rf0lc0weshay, I was hoping you would remember when we saw that error and i thought it was selinux/iptables, then it was something else19:20
rf0lc0it turned out to be something else but I can't find a bug and don't recall what is that error w/ no route to host19:21
rlandyweshay: for https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/ci-scripts/tripleo-upstream/get-hash.sh, the dlrn server downstream uses http vs https upstream19:22
rlandywe have the following options:19:22
weshayparams19:22
rlandy - ask jpena|off to use https19:22
rlandy- add extra param19:22
rlandyok - second, I guess19:23
mjturekweshay rf0lc0 - heey yeah nothing builds19:23
mjturekso I'm on the node now19:23
weshaymjturek, if ur on the node.. we can all look19:23
mjturekand it seems to be that curl is the only thing fails19:23
weshayyou just have to open a tmate from your laptop19:23
weshayheh19:23
mjturekoh really??19:23
* mjturek never used tmate19:24
weshaymjturek, that's what you get for being careful and testing :)19:24
weshaylolz19:24
weshayya.. start from your laptop.. give us the ssh link.. then we can all follow you in19:24
weshayw/o asking19:24
weshayfor permission :)19:24
weshaymjturek, so yum operations work19:24
weshaybut curl does not19:24
mjtureklol will do!19:25
mjturekI believe so19:26
bahaI'm still wondering if it's a port 80 conflict of some kind but the iptables rules have been unhelpful19:26
weshayif it doesn't repeat in software factory or local.. a conflict is a good guess19:27
mjtureksorry installing is taking forever19:28
rf0lc0so it happens in the base container build trying to curl host url on port 80?19:28
mjturekcorrect rf0lc019:28
mjturekand you can ping the host, just can't curl from it19:29
mjturekcurl to it*19:29
weshaymjturek, are you sure 80 is open?19:30
weshaycan you telnet ip 8019:30
weshayor netcat?19:30
rf0lc0let me see if there is any info about ports on zuul node19:30
mjtureklet me try19:32
weshayit would have to be a firewall rule from the node to the container19:32
mjturekweshay:19:35
mjturek[root@6f37ca8cca95 /]# nc 172.17.0.119:35
mjturekNcat: No route to host.19:35
mjturek[root@6f37ca8cca95 /]# nc 172.17.0.1 8019:35
mjturek[root@6f37ca8cca95 /]#19:35
weshayhrm19:37
mjturekagain ping is fine19:37
weshayhttpd is running right?19:38
weshaync localhost 8019:38
weshaymash / hit enter19:38
weshaythen type "get head"19:38
mjturekfrom the container host right?19:38
weshayno route to host is troubling19:38
rf0lc0https://github.com/openstack/tripleo-ci/blob/master/roles/build-containers/tasks/pre.yaml#L8019:39
weshaymjturek, ya.. first from container host19:39
rf0lc0does this happen in your job ??19:39
rf0lc0mjturek, ^19:39
mjturekrf0lc0 it does19:40
mjturekone sec19:40
mjturekChain openstack-INPUT (0 references)19:40
mjturektarget     prot opt source               destination19:40
mjturekACCEPT     tcp  --  172.17.0.0/16        anywhere             tcp dpt:http19:40
mjturekfrom the iptables -L ^19:40
weshaychanged: [n18.p8h2] => {19:40
weshay    "changed": true,19:40
weshay    "cmd": [19:40
weshay        "iptables",19:40
weshay        "-I",19:40
weshay        "openstack-INPUT",19:40
weshay        "1",19:40
weshay        "-p",19:41
weshay        "tcp",19:41
weshay        "--dport",19:41
weshay        "80",19:41
weshay        "-s",19:41
weshay        "172.17.0.0/16",19:41
weshay        "-j",19:41
weshay        "ACCEPT"19:41
weshayyup19:41
weshay<172.19.3.229> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/home/jenkins/.ansible/cp/f1086b8571 172.19.3.229 '/bin/sh -c '"'"'rm -f -r /root/.ansible/tmp/ansible-tmp-1575915661.86-189905531354841/ > /dev/null19:41
weshay2>&1 && sleep 0'"'"''19:41
*** TrevorV has quit IRC19:42
mjturekweshay: within the container host, curl http://172.17.0.1:80/delorean.repo succeeds19:43
weshayand do you have a running container.. or can you start one?19:44
mjturekI do19:44
mjturekalso there seems to be a tmate server that squashes my requests when I try to open a session, sorry19:47
mjtureka local tmate server19:47
*** dsneddon has quit IRC19:58
rlandy2019-12-10 19:11:14,208 9430 INFO tripleo_common.image.image_uploader [  ]     Upgrading:20:08
rlandy2019-12-10 19:11:14,208 9430 INFO tripleo_common.image.image_uploader [  ]      openstack-nova-common      noarch1:20.1.0-0.20191203211939.1c2b7d8.el8ost compute                  345 k20:08
rlandy2019-12-10 19:11:14,208 9430 INFO tripleo_common.image.image_uploader [  ]      openstack-nova-scheduler   noarch1:20.1.0-0.20191203211939.1c2b7d8.el8ost compute                  51 k20:08
rlandy2019-12-10 19:11:14,208 9430 INFO tripleo_common.image.image_uploader [  ]      python3-nova               noarch1:20.1.0-0.20191203211939.1c2b7d8.el8ost compute                  3.6 M20:08
rlandy^^ got them now20:08
weshaymjturek, we loose you?20:09
weshayrlandy, nice20:09
mjtureknah I'm here weshay20:09
weshayrlandy, need any help w/ reviews20:09
mjturekso is baha20:09
bahaWe're just unsure of what to add20:09
weshayheh.. not to be confused w/ mwahaha20:09
mjturekcorrect :)20:09
rlandyweshay: yeah - got a few more reviews coming up ... will post out when I have the complete list20:09
* weshay wants you guys to talk on irc.. and tell jokes to each other20:09
weshayat least baha is not evil20:10
mjtureklolol]20:10
bahaNot that you know of!20:10
weshayeven better.. hidden evil is MORE evil than blatant evil20:10
*** dsneddon has joined #oooq20:14
*** dsneddon has quit IRC20:19
rlandyweshay: we can start with these two reviews ... ready to go https://code.engineering.redhat.com/gerrit/#/c/187435/ https://code.engineering.redhat.com/gerrit/#/c/186766/20:21
rlandynote that rhos-16 passes20:21
weshaygood20:21
weshaythanks20:21
weshaynice work20:21
rlandyrhos-16 component 17 fails tempest20:22
rlandyweshay: ^^20:22
weshayOH20:22
weshaywell well20:22
rlandythat's probably legit20:22
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/a7a9e323bc474bad8dbdd546426d560520:22
rlandyweshay: another one for review https://review.rdoproject.org/r/#/c/24070/20:27
rlandy^^ note upstream change20:27
rlandybecause referring to the same script in both20:27
zbrmindblowing discovery... https://github.com/ansible/ansible/issues/6273320:37
zbri should make a poster out of "We have no plans to implement this feature at this time"20:38
*** Goneri has quit IRC20:38
zbrso now you have an easy way to distribute ansible modules written in python,.... as long they do not have external dependencies.20:39
zbrthis reminds me of a dirty trick I used in the past with some of my scripts. i installed missing python modules using pip api. "try: import foo.... except ImportError: pip.install(foo)"20:40
weshayzbr, you have a sec?20:41
weshayso we only need help for one week20:41
weshaynext week20:41
zbrsure,20:41
weshayzbr, line 134 accurate? https://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw?both20:41
weshayjust need to try and help keep the lights on20:42
zbrweshay: now it is.20:44
weshayhrm.. so you can't help next week20:44
weshayk20:44
zbrweshay: approved in orange, also rec in team cal. really out / no computer.20:45
zbrbut from 26 i will be available.20:45
weshayk k.. well.. it should be very quiet20:45
zbrstarting with 23 i will have the laptop and probably i will be able to do something if needed.20:46
zbri know is very unlikely to need to do somethign, but saying in case shit happens20:47
weshayrf0lc0, may just be you and rlandy next week20:47
rf0lc0weshay, for what20:47
rf0lc0weshay, ruck?20:48
weshayto help take some pressure off marios and raukadah20:48
zbrweshay: there is a problem with 2020 summit, but we can talk about it next year.20:48
rlandyweshay: ack20:48
rlandyweshay: which days should I take20:48
rlandythe whole week?20:48
weshayI'm not sure yet..20:48
weshaylet me chat w/ marios and raukadah20:49
rf0lc0weshay, I would suggest we switch until the end of the year20:49
rlandyweshay: its fine  - marios and raukadah need a break20:49
weshayrf0lc0, I'll work w/ you on this a bit... on scheduling ruck/rover in dec https://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw?both  lines 123 - 13920:50
weshayI'm not quite sure what to do yet20:51
rf0lc0weshay, its dec 11-jan 02... its like a full sprint20:51
rf0lc0deserves a full switch20:51
rf0lc0IMHO20:51
weshayrf0lc0, we're not going to have a sprint over shutdown20:52
rf0lc0I know20:52
weshaywe'll have a do what ever20:52
rf0lc0but we switch roles20:52
zbrweshay: i will do ruck work on *24*, i did not took it off. maybe we can build a day by day calendar for that.20:52
rf0lc0just consider my suggestion :)20:52
weshayrf0lc0, thought we were going to kick the sprint out an extra week and end on 12/1820:52
weshaythat's what taiga says 21 NOV 2019-18 DEC 201920:53
rf0lc0taiga is a lier20:54
rf0lc0well maybe chandan changed it?20:54
weshayget volunteers for the additional week for ruck / rover and then call it quits until Jan20:54
rf0lc0I don't know20:54
zbrimho a sprint should not run during these times, finishing tasks from previous one and other chores sounds like a good use of these times. the only thing we should care about is ruck/rover coverage.20:54
weshayzbr, agree20:54
rf0lc0yes20:54
rf0lc0my point is20:54
rf0lc0we just switch ruck/rovers20:54
rf0lc0thats unfair for marios20:55
rf0lc0If I were him, I would like to switch tomorrow20:55
rf0lc0I agree w/ zbr point20:55
zbrrf0lc0: we do something like this, the only difference is that we will have several people, i doubt we have anyone available for the entire period.20:56
zbri will do my best to cover for the time i am not away (mainly next week)20:56
weshayrf0lc0, probably should cancel the sprint retro mtg21:00
*** jtomasek has quit IRC21:02
rf0lc0weshay, ok will move to next wed then21:04
weshaythanks21:04
*** jaosorior has joined #oooq21:05
*** jfrancoa has quit IRC21:16
rlandyweshay: rf0lc0: you can assign me to ruck/rover on whatever days. fine by me21:20
rlandyweshay: rf0lc0: question ... where should we put the downstream promote-consistent-to-tripleo-ci-testing job? in config repo (following upstream pattern) or in jobs ( departure from upstream pattern but allows for speculative testing)21:21
*** d0ugal has joined #oooq21:29
*** dsneddon has joined #oooq21:32
zbrweshay: i finally made my first collection, and it works! this is a life-changer!21:37
zbrhttps://softwarefactory-project.io/r/#/c/17186/21:37
weshayah.. cool /me looks21:37
zbrthat is a response to a problem folco had yesterday, he was not able to make the dlrn_api module to work, this fixed the deployment.21:38
zbrmainly we can bundle roles, playbooks and modules and publish them to galaxy.21:38
zbrthe only issue is that this works only with ansible 2.8, but i doubt we cannot upgrade to 2.8 everywhere if needed.21:40
*** jaosorior has quit IRC22:01
rlandyweshay: merging https://review.rdoproject.org/r/#/c/24070/, ok?22:25
rlandycould break promotions - but unlikey22:25
rlandyunlikely22:25
* weshay looks22:25
weshayrlandy, we're ok on promotions for tomorrow22:26
weshayso ack from me22:26
rlandyok - here goes22:26
weshayhave to go grab kiddos bbl or ttyt22:28
rlandysure22:29
*** apetrich has quit IRC22:54
*** rlandy is now known as rlandy|bbl23:13
*** Goneri has joined #oooq23:35
*** Goneri has quit IRC23:40
*** rf0lc0 has quit IRC23:47
*** rf0lc0 has joined #oooq23:49

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!