Friday, 2019-07-19

*** vinaykns has quit IRC00:26
*** ykarel|away has quit IRC01:17
*** sabedevops has joined #oooq01:42
*** bhagyashris has joined #oooq02:00
*** ysandeep has joined #oooq03:50
*** skramaja has joined #oooq03:54
*** sabedevops has quit IRC03:55
*** ykarel has joined #oooq03:59
*** udesale has joined #oooq04:04
*** apetrich has quit IRC04:20
*** jfrancoa has joined #oooq04:30
*** ratailor has joined #oooq04:39
*** ykarel has quit IRC04:54
*** marios has joined #oooq05:03
*** marios|ruck has joined #oooq05:04
*** aakarsh|2 has quit IRC05:07
*** ykarel has joined #oooq05:08
*** raukadah is now known as chandankumar05:36
*** ykarel_ has joined #oooq05:40
*** udesale has quit IRC05:42
*** ykarel has quit IRC05:42
*** udesale has joined #oooq05:42
*** jdennis has quit IRC05:48
*** jdennis has joined #oooq05:49
*** dsneddon has quit IRC05:51
*** dsneddon has joined #oooq05:55
*** marios has quit IRC06:01
*** marios has joined #oooq06:02
*** jbadiapa has quit IRC06:03
*** marios has quit IRC06:07
*** marios|ruck has quit IRC06:07
*** bogdando has joined #oooq06:08
*** marios|ruck has joined #oooq06:09
*** marios has joined #oooq06:09
*** udesale has quit IRC06:12
*** udesale has joined #oooq06:17
*** ykarel__ has joined #oooq06:17
*** ksambor has quit IRC06:17
*** ykarel_ has quit IRC06:19
*** holser has joined #oooq06:56
*** holser has quit IRC06:57
*** holser has joined #oooq06:57
*** ykarel__ is now known as ykarel07:08
*** slaweq has joined #oooq07:09
*** amoralej|off is now known as amoralej07:12
chandankumarpanda: hello07:12
chandankumarpanda: which dlrn hash is used in rhel-8 container build?07:12
*** zbr has quit IRC07:15
*** zbr has joined #oooq07:15
ykarelchandankumar, 65570a311e1303d8084d12ee82868e6d0f411e8a_4125074007:21
*** ykarel is now known as ykarel|lunch07:23
*** tosky has joined #oooq07:24
arxcruzchandankumar, are we still using validate-tempest? I though you move everything to os_tempest07:40
zbrit seems that ansible team decided to break everyone, the "-s" addition to rpm installs. just got https://github.com/ansible/molecule/issues/2173 in addition to the bug reported by wes on tripleo.07:41
marios|rucksshnaidm|off: any idea why "master failing periodic jobs" isn't showing data @ grafana i am digging a bit i think this commit from last week maybe https://github.com/rdo-infra/ci-config/commit/68f2c368bc2b89f641dffc3be8886515ed2d1f97 can you check when you're next around please07:48
*** jbadiapa has joined #oooq07:49
chandankumararxcruz: we will be moving all stuff to os_tempest07:56
chandankumararxcruz: there is no blocker now07:56
marios|ruckwtf the periodic master promotion jobs last ran on the 16th?07:56
arxcruzchandankumar, so why the patch for validate-tempest ?07:57
zbrpanda: interesting wes created the bug on tripleo but I see no bug against RHEL. in fact I am not even sure from which repository ansible is installed from.07:58
pandazbr: what bug ?07:59
zbrread https://bugs.launchpad.net/tripleo/+bug/183714307:59
openstackLaunchpad bug 1837143 in tripleo "rhel-8, ansible-playbook-3 fails to import python3-jinja2 in tripleo deployment" [Critical,Triaged]07:59
marios|ruckarxcruz: any idea if something happened on tuesday? stein/master and queens afaics last ran on the 16th (periodic jobs) http://cistatus.tripleo.org/promotion/08:00
arxcruzmarios|ruck, no idea, but if you see the cockpit there were 122 failed stacks yesterday08:01
arxcruz122 failed to delete stacks08:01
marios|ruckarxcruz: k thx (yeah but that started on tuesday like only one run very early tuesday morning then nothing else. and wondering why rocky is special it even promoted yesterday)08:03
pandazbr: you expected a bug in bugzilla ?08:03
pandazbr: the genreal idea for rdo on rhel jobs is the bugs to be visible to the public in launchpad08:04
*** ccamacho has joined #oooq08:04
zbrclearly we need a bugzilla bug for it, not sure where it would fit, ansible-engine or rhel-8 product.08:04
arxcruzmarios|ruck, so the jobs did not run or run but it wasn't promoted ?08:04
chandankumararxcruz: no idea08:04
chandankumararxcruz: that's why I left a comment there to move the stuff to os_tempest08:05
zbrpackaging for rhel8 is *not* ansible responsability. they package for other platforms. -- i am going to test if their own packaging has the same issue08:06
*** derekh has joined #oooq08:08
chandankumarpanda: sshnaidm|off https://review.opendev.org/#/c/669487/ -> https://review.rdoproject.org/r/#/c/21295/ -> https://review.rdoproject.org/r/#/c/21294/08:08
chandankumarin sequence08:08
chandankumarpanda: for rhel8 standalone jobs few containers are missing from rdo registry08:09
pandazbr: I think the bug was shown to reldel anyway08:09
marios|ruckarxcruz: jobs not run e.g. see that https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master08:10
marios|ruckarxcruz: last run on the 16th08:10
marios|ruckarxcruz: going afk few min biab08:10
marios|ruckarxcruz: might be one of the dependencies i have to dig a bit when i get back08:11
zbrpanda: link?08:14
pandazbr: read the full bug now, why is jinja2 in the system site packages dir considered user site package ?08:15
chandankumarykarel|lunch: https://review.opendev.org/#/c/669487/ -> https://review.rdoproject.org/r/#/c/21295/ -> https://review.rdoproject.org/r/#/c/21294/08:24
*** ykarel|lunch is now known as ykarel08:24
ykarelchandankumar, ack08:28
ykarelmarios|ruck, rfolco|rover is the issue with periodic pipeline already known?08:48
ykarelpipeline is stuck from 16th08:48
marios|ruckykarel: any idea why the https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-7-master-promote-consistent-to-tripleo-ci-testing  didn't run since 16th... all the others depends on it so no periodics are running since then08:48
marios|ruckykarel: hah i pinged you same time :D08:48
ykarel:D08:49
marios|ruckykarel: i noticed it this morning ... but still trying to understand08:49
ykarelmarios|ruck, the pipeline is still running08:49
ykarelfrom 68 hours08:49
ykarelsee https://review.rdoproject.org/zuul/status08:49
ykarel24hr one is running from 46 hours08:49
ykarelso we need attention from zuul guys08:49
ykarel#sf-ops08:49
*** panda is now known as panda|drappt08:50
ykarelsorry 46 hours is stein, 24hr one is running from 28 hours08:50
marios|ruckykarel: wtf.. ack thanks also going to file a bug and joining there08:50
ykarelack08:50
ykarelomg so it's not just periodic, regular pipeline also have jobs from 47 hours08:51
*** derekh has quit IRC09:09
*** derekh has joined #oooq09:19
chandankumarmarios|ruck: https://review.opendev.org/#/c/671447/4 needs +w on this09:19
marios|ruckchandankumar: ack09:22
*** yolanda has quit IRC09:30
*** bogdando has quit IRC09:37
chandankumararxcruz: https://tree.taiga.io/project/tripleo-ci-board/epic/1149 please have a look09:39
chandankumarI have added content to it, feel free to add more09:39
*** apetrich has joined #oooq09:40
*** dsneddon has quit IRC09:48
*** bhagyashris has quit IRC10:01
*** udesale has quit IRC10:05
*** udesale has joined #oooq10:06
*** ysandeep has quit IRC10:11
*** apetrich has quit IRC10:13
*** panda|drappt is now known as panda10:14
chandankumarpanda: I will be later for meeting by 10 mins10:17
pandachandankumar: does arxcruz know ?10:19
arxcruzpanda, i know now10:19
pandaarxcruz: you should have known before10:20
*** jfrancoa is now known as jfrancoa|lunch10:20
pandanext time , know the things you don't know before you think you don't know them10:21
arxcruzyes sir10:21
*** bogdando has joined #oooq10:22
*** dsneddon has joined #oooq10:23
*** brault has joined #oooq10:26
*** brault has quit IRC10:32
chandankumarpanda: arxcruz I am back, can we start the meeting10:37
pandachandankumar: I am 10 minutes late, and after this 10 minutes it's arxcruz turn to be late10:39
chandankumarpanda: he already joined10:39
*** udesale has quit IRC10:44
*** udesale has joined #oooq10:45
*** dsneddon has quit IRC10:47
*** udesale has quit IRC10:54
*** udesale has joined #oooq10:55
*** udesale has quit IRC11:02
*** jfrancoa|lunch is now known as jfrancoa11:15
*** dsneddon has joined #oooq11:15
marios|ruckbiab pickup kid11:25
*** tesseract has joined #oooq11:33
*** apetrich has joined #oooq11:38
*** panda is now known as panda|lunch11:42
chandankumarrfolco|rover: ping, let me know when around?11:49
weshayzbr can you please prep a rhel8 vm or bm machine for a standalone deployment11:54
weshayzbr you just need rhel811:54
weshaydlrn11:54
marios|ruckweshay: o/ fyi https://bugs.launchpad.net/tripleo/+bug/183719711:58
openstackLaunchpad bug 1837197 in tripleo "[rdo periodic jobs] RDO periodic pipeline is stuck blocking promotions." [Critical,Triaged]11:58
marios|ruckweshay: should ok now fbo dequeued it but it was going for almost 70 hours (noticed nothing ran since _tuesday_)11:59
weshaymarios|ruck thanks12:00
marios|ruckpanda|lunch: fyi depends merged so we can try that now? https://review.rdoproject.org/r/#/c/21357/12:01
chandankumarweshay: need one help12:15
chandankumarweshay: rename this pushed container trunk.registry.rdoproject.org/tripleomaster/rhel-binary-rsyslog-base:65570a311e1303d8084d12ee82868e6d0f411e8a_41250740 to trunk.registry.rdoproject.org/tripleomaster/rhel-binary-rsyslog:65570a311e1303d8084d12ee82868e6d0f411e8a_4125074012:16
chandankumarweshay: to unblock standalone ci work12:16
weshaychandankumar did we get to the point where containers were being downloaded?12:17
chandankumarweshay: yes12:18
chandankumarweshay: https://logs.rdoproject.org/70/21270/35/check/periodic-tripleo-ci-rhel-8-standalone-master/16f1def/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-07-19_03_44_3412:18
chandankumarweshay: check the last comment https://tree.taiga.io/project/tripleo-ci-board/task/116612:18
chandankumarweshay: https://review.opendev.org/#/c/624325/ changes the container name12:19
chandankumarI reverted it12:19
chandankumarnow tQE tacks rhel8 support for dlrn12:19
chandankumar*lacks12:19
chandankumaron which amoralej is looking into it12:19
chandankumarwe need to find a way to get mock rpm also12:19
amoralejchandankumar, mock is only one part of the problem12:20
chandankumaryes12:20
chandankumarweshay: so in order to proceed we need to rename the rsyslog container12:21
chandankumaror merge this patch12:21
*** udesale has joined #oooq12:21
*** dsneddon has quit IRC12:21
chandankumarweshay: : https://review.opendev.org/#/c/669487/ -> https://review.rdoproject.org/r/#/c/21295/ -> https://review.rdoproject.org/r/#/c/21294/12:21
weshaychandankumar show me the job12:21
weshaylog12:21
weshayhttps://logs.rdoproject.org/70/21270/35/check/periodic-tripleo-ci-rhel-8-standalone-master/16f1def/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-07-19_03_44_3412:21
weshaygot it12:21
chandankumarweshay: I made changes in release file to use particular dlrn hash12:22
chandankumarabove container pulled does not exist on registry12:22
weshaychandankumar why.. current-tripleo and tripleo-ci-testing are at the correct hash12:23
chandankumarweshay: I got your point that change is not needed12:28
chandankumarweshay: now look again to the missing container question12:28
*** amoralej is now known as amoralej|lunch12:28
*** skramaja has quit IRC12:29
weshaychandankumar ok.. rfolco|rover can you help chandankumar fix the name on the one container12:31
rfolco|roverweshay, k12:31
rfolco|roverchandankumar, whats wrong with the name ?12:34
*** panda|lunch is now known as panda12:36
*** dsneddon has joined #oooq12:36
pandachandankumar: for 669487 I see that build containers in centos still works, but the additional successfully-built/failed-to-build log is not really workign as expected12:40
pandachandankumar: I think we can merge anyway, but just a heads up, that that part is not working yet12:41
*** dsneddon has quit IRC12:41
weshaychandankumar what I don't get is how my local install got further than the ci job12:43
weshaychandankumar I don't see the heat stack created in your log12:44
chandankumarrfolco|rover: check last comment https://tree.taiga.io/project/tripleo-ci-board/task/116612:46
chandankumarrfolco|rover: https://review.opendev.org/#/c/624325/ changes the container name12:47
weshaychandankumar that's weird12:47
*** udesale has quit IRC12:49
chandankumarrfolco|rover: we need to change the same12:50
chandankumarweshay: please +w it https://review.opendev.org/#/c/669487/12:50
chandankumararxcruz: tempest meeting agena https://etherpad.openstack.org/p/tripleo-tempest-squad-meeting12:51
*** ratailor has quit IRC12:52
*** udesale has joined #oooq12:53
zbrweshay: damn, what i installed was an 8.1 instead of 8.0 which means that rhui does not install on it.12:53
chandankumarykarel: https://review.rdoproject.org/r/#/c/21294/12:56
*** apetrich has quit IRC12:58
*** dtantsur is now known as dtantsur|bbl12:59
chandankumararxcruz: weshay meeting time for tempest planning13:00
chandankumararxcruz: weshay https://bluejeans.com/157131391913:00
*** apetrich has joined #oooq13:02
*** ksambor has joined #oooq13:03
*** dsneddon has joined #oooq13:07
*** rlandy has joined #oooq13:09
*** rlandy_ has joined #oooq13:13
*** rlandy_ has quit IRC13:13
*** rlandy_ has joined #oooq13:13
*** rlandy_ has quit IRC13:15
*** rlandy_ has joined #oooq13:15
*** rlandy has quit IRC13:16
chandankumararxcruz: https://logs.rdoproject.org/17/171117/24/check/tripleo-ci-rhel-8-standalone-rhos-15/d73bd3e/logs/stestr_results.html13:16
*** rlandy_ is now known as rlandy13:17
chandankumararxcruz: https://code.engineering.redhat.com/gerrit/#/c/171117/13:17
*** dsneddon has quit IRC13:20
chandankumararxcruz: weshay https://tree.taiga.io/project/tripleo-ci-board/us/118813:20
rlandyweshay: hi - anything we need to touch base on before brno meeting?13:21
weshayrlandy not really13:21
weshayrlandy you are back from pto today right?13:22
rlandyweshay: yep - here today13:22
rlandyand monday13:22
weshayrlandy k.. sync back up w/ panda re: check and periodic jobs13:24
rlandyk13:24
rlandypanda: ^^ have time today to sync?13:25
weshaypanda I really need to see some movement on https://review.rdoproject.org/r/#/c/21215/13:26
*** holser has quit IRC13:27
*** vinaykns has joined #oooq13:29
*** amoralej|lunch is now known as amoralej13:31
chandankumarrlandy: welcome back :-)13:32
rlandychandankumar: thanks13:32
rlandyI'm back for two days13:32
rlandyand then going again to prod chain meeting13:32
chandankumarrlandy: https://logs.rdoproject.org/70/21270/38/check/periodic-tripleo-ci-rhel-8-standalone-master/a80106f/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2019-07-19_09_17_4513:34
chandankumarweshay: ^^13:34
rlandychandankumar: this is the error you are debugging atm?13:36
rlandyI remember this error13:37
pandachandankumar: do we want the jobs defined in check for 21418 ?13:38
pandarlandy: wanna chat ?13:38
rlandypanda: sure13:38
rlandychandankumar: I have definitely seen that error before - just have to remember how we fixed it :)13:40
pandarlandy: my bj ?13:42
rlandypanda: sure joining13:43
rlandypanda: named bj is not connecting - do you have a bj number?13:45
chandankumarpanda: nope13:46
pandarlandy: 349250866913:46
pandachandankumar: wnat me to remove it ?13:46
chandankumarpanda: we have added there just for testing that13:46
chandankumarpanda: yes go ahead13:46
pandachandankumar:13:46
pandaok13:46
chandankumarrlandy: I am dropping an email with the current status of container build and standalone13:47
chandankumarrfolco|rover: does the container got pushed?13:48
chandankumarsorry I mean the container rename part13:48
*** jfrancoa has quit IRC13:48
rfolco|roverchandankumar, I thought you would be doing that13:49
rfolco|roverchandankumar, is it just rsyslog ?13:50
rlandypanda: are you connected? bj is just spinning for me - not connecting13:50
rfolco|roverexplain me what you need chandankumar13:50
*** Vorrtex has joined #oooq13:51
pandarlandy: yep connected and no problem, let's try you bj13:51
rlandypanda: https://bluejeans.com/3492508669 looks like it's starting to work - mine is still spinning - will ping you when I have some connection13:53
chandankumarrfolco|rover: we need to rename this trunk.registry.rdoproject.org/tripleomaster/rhel-binary-rsyslog-base:65570a311e1303d8084d12ee82868e6d0f411e8a_41250740 to this trunk.registry.rdoproject.org/tripleomaster/rhel-binary-rsyslog:65570a311e1303d8084d12ee82868e6d0f411e8a_4125074013:53
chandankumarrfolco|rover: and push it13:53
*** dsneddon has joined #oooq13:53
chandankumarwe can use buildah,13:54
chandankumartag i think13:54
*** dsneddon has quit IRC13:58
pandachandankumar: what's with 669057 -1W ?13:59
*** aakarsh|2 has joined #oooq14:04
rlandyweshay: can you connect to bluejeans? panda can but I can't14:08
weshayrlandy yes14:09
pandarlandy: ok take a look here for now https://etherpad.openstack.org/p/rhel8-pipeline14:11
rlandypanda: ok - that I can see14:11
marios|ruckweshay: is that what you were referring to yesterday or is that a different one https://bugs.launchpad.net/tripleo/+bug/183721614:11
openstackLaunchpad bug 1837216 in tripleo "The tripleo-ci zuul reproducer fails to stop zuul services (docker_service) due to docker-py" [Medium,Triaged] - Assigned to Marios Andreou (marios-b)14:11
*** ykarel is now known as ykarel|away14:15
*** sabedevops has joined #oooq14:17
*** sbroderick has joined #oooq14:19
*** ykarel|away has quit IRC14:20
weshayrfolco|rover this is odd.. I don't see any rhel containers in tripleomaster atm?14:20
weshaywhat about you?14:20
rfolco|roverchecking14:20
rfolco|roverI remember I had this issue while pushing... sometimes it just doesn't show them...14:21
*** sbroderick has quit IRC14:21
rfolco|roverweshay, I see some loading now14:22
rfolco|roverhttps://console.registry.rdoproject.org/registry#/images/tripleomaster/rhel-binary-aodh-api14:22
*** dtantsur|bbl is now known as dtantsur14:23
rfolco|roverweshay, chandankumar done14:24
rfolco|roverhttps://console.registry.rdoproject.org/registry#/images/tripleomaster/rhel-binary-rsyslog14:24
rfolco|roverpushed14:24
weshayrfolco|rover perfect timeing :)14:24
weshaytiming14:24
rfolco|roverlet me know if any issues with that14:25
weshaychandankumar retrigger ^14:26
*** sbroderick has joined #oooq14:28
*** dsneddon has joined #oooq14:31
*** aakarsh|2 is now known as aakarsh14:38
*** rlandy has quit IRC14:43
*** rlandy has joined #oooq14:46
rlandypanda: bluejeans connected now ... on my bluejeans if you have time14:47
pandarlandy: weshay is stably on mine since 40 minutes, and he seems to have no intention to budge :)14:50
pandabut if you feel your conection may not work there let's all move14:50
rlandytrying to join14:50
rlandypanda: k - finally - joined your bj14:51
weshayzbr just noticed some interesting customizations for rhel 814:52
weshayzbr in the standalone deployment config..   PythonInterpreter: /usr/bin/python314:52
weshaychandankumar panda rlandy I think the interitems.. what ever may be fixed by looking at14:54
weshayhttps://gitlab.com/snippets/186926914:54
weshaynotes..14:55
weshay  PythonInterpreter: /usr/bin/python314:55
weshayrlandy chandankumar https://docs.google.com/document/d/1YyUiVZXVFU2JNRa6ur22Ek6s_o_-HYU02GqDF9gLRoI/edit#14:55
weshaypanda ^14:55
weshaychandankumar panda chandankumar https://docs.ansible.com/ansible/2.5/dev_guide/testing/sanity/no-dict-iteritems.html14:57
*** tosky_ has joined #oooq14:59
*** dsneddon has quit IRC14:59
rfolco|roverweshay, what makes a deployment / featureset ha or non-ha ?15:01
*** dsneddon has joined #oooq15:01
*** tosky has quit IRC15:02
weshaychandankumar rlandy panda that did not fix it :(15:04
weshayrfolco|rover the pacemaker service .. true ha you need 3 ctrls15:06
*** dsneddon has quit IRC15:06
rlandyweshay: k- let me dig my brain/code archives - we definitely solved this at one point15:06
weshayzbr your skills would be handy here15:06
rfolco|roverweshay, this is in scenario not featureset, found it. Thanks.15:07
zbrweshay: well, i am kinda kinda busy with what appears to be messed networking after running openstack tripleo deploy15:07
*** tosky_ is now known as tosky15:10
weshayzbr did you create the bridge?15:12
weshayzbr chandankumar fyi https://paste.fedoraproject.org/paste/NEU9X7L92WRI6Qnt-9-~yw15:13
weshayline 3315:13
weshayhttps://opendev.org/openstack/tripleo-common/src/branch/master/roles/tripleo-hieradata/templates/all_nodes.j215:13
*** sabedevops has quit IRC15:15
weshayzbr chandankumar this was fixed by Evilien in https://opendev.org/openstack/tripleo-common/commit/d60eb1b4be4fdf0c990601337473ce683658f40f15:15
zbryeah, DNS resolutions works but not outside routing, only internal network seems to be working.15:16
weshayzbr chandankumar I suspect we need a new hash15:16
weshayzbr chandankumar the hash we've pinned to is too old15:16
* weshay tries15:16
zbri think i found it...15:16
weshaychandankumar set tripleo-ci-testing to https://trunk.rdoproject.org/rhel8-master/69/ca/69cab4cd5356e6c5314a103ca760d531d004dc5a_802178e9/delorean.repo15:18
weshaythat is the current centos7 promoted hash for master15:18
rlandyhttps://review.opendev.org/#/c/66921915:18
rlandyhttps://bugs.launchpad.net/tripleo/+bug/183545615:19
openstackLaunchpad bug 1835456 in tripleo "AnsibleUndefinedVariable during all_nodes data rendering under Python3" [Critical,Fix released] - Assigned to Attila Fazekas (afazekas)15:19
chandankumarweshay: panda rlandy I am updating my patch15:19
weshaychandankumar note.. we don't have containers on that updated patch yet..15:20
weshaybut we can get the rpms on the host15:20
* weshay trying again15:20
chandankumarrfolco|rover: updated the contaners na15:20
*** dsneddon has joined #oooq15:21
weshaypanda rlandy let's get the periodics rolling :) so we don't hit this stuff :)15:21
marios|ruckerr did someone just kill grafana?15:22
marios|rucksshnaidm|off: ^ is that  you :D ?15:22
chandankumarnot me15:22
weshaychandankumar it would be really smart to turn --debug on the standalone deploy15:23
weshaymarios|ruck it probably died15:23
pandamarios|ruck: proably the memory probelems sshnaidm|off mentioned ?15:23
weshayout of memory15:23
weshaymarios|ruck I can help you restart it15:23
marios|ruckweshay: panda ah :/15:23
marios|ruckweshay: ok sure15:23
sshnaidm|offmarios|ruck, not me, it's killer15:23
marios|ruckweshay: or do we have it in a doc i can look at instead of takin yr time?15:23
weshaymarios|ruck k.. open a tmux or tmate15:23
marios|ruckweshay: one of those googledocs :D15:23
chandankumarpanda: sorry, I was testing something that so -w to avoid for comments15:24
chandankumarpanda: now updated the stuff15:25
marios|ruckweshay: on which server? i mean where are we hosting grafana15:25
*** dsneddon has quit IRC15:25
pandamarios|ruck: tripleo-infra tenant in rdocloud15:25
marios|ruckack thanks sshnaidm|off15:26
marios|ruckpanda: thanks looking15:26
pandamarios|ruck: it should be connected to your account15:26
rlandypanda: we have a merge order problem15:26
*** sbroderick has quit IRC15:26
marios|ruckpanda: oh i was going to fetch 'that doc'15:26
rlandypanda; https://review.rdoproject.org/r/#/c/21495/ is first on your list but depends on https://review.rdoproject.org/r/#/c/21270/15:26
pandarlandy: please be a circular dependency,15:27
* rlandy doesn't know how o be a circular dependency 15:27
weshaychandankumar panda rlandy zbr ok.. standalone is deploying now w/ the centos-7 promoted hash15:28
pandarlandy: what's the issue with that dependency15:28
rlandyk - let us know15:28
weshaychandankumar panda rlandy zbr specifically https://trunk.rdoproject.org/rhel8-master/69/ca/69cab4cd5356e6c5314a103ca760d531d004dc5a_802178e9/delorean.repo15:28
rlandypanda: look at the reviews15:28
rlandyhttps://review.rdoproject.org/r/#/c/21270/38/zuul.d/projects.yaml ca't mereg as is anyways15:28
chandankumaris it a new hash?15:28
pandarlandy: maybe the description there is not self explanatory15:28
pandarlandy: 21495 si the final goal, the other are dependency taht need to be merged to get there15:29
pandarlandy: but you have to start from the leaves15:29
chandankumarpanda: https://review.rdoproject.org/r/#/c/21495/ yes final goal15:29
rlandypanda: k - so we don;t need to merge the reviews https://etherpad.openstack.org/p/rhel8-pipeline in the order they are listed15:29
rlandygot it15:30
*** zbr is now known as zbr|out15:30
marios|ruckpanda: see pvt when u have 2 mins thanks15:30
rlandyI need to edit these jobs to merge them15:30
rlandypanda: chandankumar: ^^15:30
rlandyhttps://review.rdoproject.org/r/#/c/21270/38/zuul.d/projects.yaml15:30
rlandythey change the projects.yaml files15:30
chandankumarrlandy: go ahead and take it edit it15:30
rlandyk15:30
rfolco|roverweshay, ping 1-115:31
rfolco|roverweshay, knock knock15:32
chandankumarrfolco|rover: thanks for pushing the container15:32
marios|ruckweshay: so apparently i can't do it as i can't switch to tripleo-infra tenant in rdo15:32
marios|ruckthanks panda15:32
rfolco|roverchandankumar, np man :)15:33
*** ykarel|away has joined #oooq15:33
chandankumarykarel|away: containers pushed15:34
weshaymarios|ruck you can't ssh to the cockpit?15:34
chandankumarykarel|away: https://review.rdoproject.org/zuul/stream/f02cb0c4589a441f826791af2e670669?logfile=console.log15:34
weshayyou don't need the openstack console for that15:34
ykarel|awaychandankumar, nice15:34
weshayrfolco|rover I'm in my blue15:34
marios|ruckweshay: no i was trying to login via rdo cloud ... for ssh what are the creds couldn't see them in the team doc15:35
marios|ruckweshay: (i just asked in sf-ops to be added to tripleo-infra anyway )15:35
rlandypanda: there are some opendev review here that need to merge first15:37
marios|ruckweshay: ah i am in now  (marios@)15:37
marios|ruckweshay: looks like we added keys there...15:37
rlandyhttps://review.opendev.org/#/c/669918/15:37
pandarlandy: yes15:38
rlandyok - merging my own reviews :)15:39
pandaaaah ! blasphemy!15:39
rlandypanda: pls vote here https://review.opendev.org/#/c/669057 and I'll w+15:40
rlandywe can change it afterwards15:40
chandankumarpanda: rlandy I am not changing anything in release file15:41
marios|ruck weshay: am in there now ... not sure what to restart... you sound very busy so you could tell me but since you asked earlier tmate in pvt15:41
pandarlandy: done15:41
pandachandankumar: it's harmless, it doesn't touch the existing, so even if it's completely broken , it's not important15:42
rlandychandankumar: it's fine, we just need to merge https://review.opendev.org/#/c/669057 as is a depends-on15:42
chandankumarpanda: rlandy https://review.opendev.org/#/c/671017/ this one also15:42
rlandychandankumar: we're carrying around a ton of unmerged work15:42
chandankumarrebased on top of master release file15:43
rlandychandankumar: k - will w+ once testing completes15:43
rlandychandankumar; do you have any other reviews that are not on https://etherpad.openstack.org/p/rhel8-pipeline?15:44
chandankumarrlandy: nope15:46
rlandypanda: wrt https://review.rdoproject.org/r/#/c/21215/3/playbooks/tripleo-rdo-base/pre.yaml - what happened to adding those packages to the image rather?15:46
marios|ruckweshay: woo \o/ back thanks15:46
marios|rucksshnaidm|off: ^15:47
chandankumarrlandy: you are around till tuesday na?15:48
marios|ruckweshay: sshnaidm|off after restarting it the machine is sooo slow don't know if its the network or if grafana is killing it15:48
rlandychandankumar: no - I'll be traveling to brno15:48
rlandychandankumar: I am here on monday15:48
chandankumarrlandy: ack15:49
pandarlandy: I don't remember discussing to install those packages in the images. Isn't docker installed during undercloud install for example ? Anyway, I install them I login, I remove them, no interference with the rest, we just need the cached auth file for both, then even just the libraries (docker-py and friends) will work as they look for those cache file15:50
* marios|ruck almost out15:51
pandamarios|ruck: don't leave us15:51
pandamarios|ruck: in the moment of need15:51
*** dsneddon has joined #oooq15:53
* rlandy looks back15:53
weshaychandankumar TASK [Start containers for step 2] :)15:56
chandankumarweshay: rlandy https://logs.rdoproject.org/70/21270/38/check/periodic-tripleo-ci-rhel-8-standalone-master/f02cb0c/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz15:58
chandankumarweshay: rlandy we are back to same state where we left15:58
chandankumarwith rhel8 container15:58
chandankumarhttps://logs.rdoproject.org/70/21270/38/check/periodic-tripleo-ci-rhel-8-standalone-master/f02cb0c/logs/undercloud/var/log/httpd/image_serve_error.log.txt.gz15:59
chandankumarit is not about missing package, there is something wrong with htacees stuff16:02
rlandychandankumar: I'm a little out of the loop here16:05
rlandywhen I left we had that error16:05
rlandyso you pinned the containers?16:06
rlandynow you unpinned them and we have this error again?16:06
rlandylooking at weshay's trace of steps,16:06
chandankumarrlandy: yes 192.168.24.1:8787/tripleomaster/rhel-binary-rsyslog:65570a311e1303d8084d12ee82868e6d0f411e8a_4125074016:06
chandankumarrlandy: it is the same hash I pinned or unpinned16:07
chandankumarwe just renamed the rsyslog container16:07
rlandyhttps://gitlab.com/snippets/186926916:07
*** tosky has quit IRC16:08
rlandychandankumar: ^^ do you have line 53 and line 88?16:08
chandankumarhttps://logs.rdoproject.org/70/21270/38/check/periodic-tripleo-ci-rhel-8-standalone-master/f02cb0c/logs/undercloud/home/zuul/standalone_parameters.yaml.txt.gz16:09
chandankumarrlandy: 53 is not there16:09
chandankumarand 88 also not16:10
weshaychandankumar add   PythonInterpreter: /usr/bin/python316:10
*** dsneddon has quit IRC16:12
*** marios|ruck has quit IRC16:12
*** dsneddon has joined #oooq16:12
chandankumarweshay: rlandy why we need rhsm?16:14
*** marios has quit IRC16:15
weshaychandankumar NOOOOOO16:15
weshayWE DON"T WANT RHSM16:15
chandankumarweshay: ok updated with rhsm16:16
chandankumarsorry16:16
rlandychandankumar: no rhsm16:16
chandankumarwith pythoninterpreter16:16
rlandymy mistake16:16
rlandyonly 5316:16
chandankumarrlandy: https://review.opendev.org/#/c/671768/16:16
rlandychandankumar: ok - watching that16:17
chandankumarit's raining heavily at pune16:17
rlandychandankumar: but really that error had to do with not having the cert on the containers16:17
rlandywas there any fix to deal with that?16:18
chandankumardoes python3 interpreter takes care of cert16:18
chandankumar?16:18
rlandychandankumar: no16:18
* chandankumar needs to read the code16:18
rlandysee my comment above16:18
chandankumarok16:18
rlandychandankumar: weshay: was there any review to deal with getting the cert on the containers?16:18
chandankumarweshay: let's test it then we can update the patch16:19
*** amoralej is now known as amoralej|off16:19
rlandyimage_serve_error16:19
* chandankumar is reading currently this https://www.keycdn.com/support/htaccess-not-working16:20
weshayTASK [Start containers for step 4]16:20
weshaywooot16:20
rlandyweshay can correct me - but when we debugged the error before, it was no cert on containers16:20
rlandythe htaccess error16:21
weshayhttp://paste.openstack.org/show/754596/16:21
rlandyI thought we were going to put in a review to deal with that16:21
weshayrlandy the cert is on the base container16:21
rlandyok - so that did get fixed16:21
weshayrhel8-rhui16:22
weshayyes16:22
rlandybut now we are back with that error16:22
weshaywell.. in the container we're using.. not in the code yet16:22
weshayrlandy where?16:22
rlandyhttps://logs.rdoproject.org/70/21270/38/check/periodic-tripleo-ci-rhel-8-standalone-master/f02cb0c/logs/undercloud/var/log/httpd/image_serve_error.log.txt.gz16:22
rlandy^^ what chandankumar posted16:22
chandankumarrlandy: new run https://review.rdoproject.org/zuul/stream/e7ea6da2932b43deb790e23e6ffa245a?logfile=console.log16:22
weshaylooking16:23
chandankumarweshay: we are using the same containers in above job16:23
weshayrlandy so that's not the rhui16:25
weshaythat's the registry no?16:25
*** dtantsur is now known as dtantsur|afk16:25
weshayTASK [Run deployment StandalonePostDeployment] **********************************************************************************************************************************************************************16:26
chandankumarweshay: not rhui, it is happening while container image prepare step16:26
rlandythat is where we saw it before I left16:26
chandankumarhttps://logs.rdoproject.org/70/21270/38/check/periodic-tripleo-ci-rhel-8-standalone-master/f02cb0c/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz16:27
chandankumarweshay: ^^16:27
weshaywooot16:27
weshayhttp://paste.openstack.org/show/754597/16:27
weshayI see it..16:27
weshaychandankumar rlandy blue16:27
chandankumarweshay: bnmlink16:27
rlandyjoined16:28
weshayhttps://bluejeans.com/u/whayutin/16:30
chandankumarweshay: rlandy ack16:39
chandankumarsee ya on monday16:39
*** chandankumar is now known as raukadah16:39
raukadahweshay: rlandy: https://logs.rdoproject.org/70/21270/39/check/periodic-tripleo-ci-rhel-8-standalone-master/e7ea6da/logs/undercloud/home/zuul/standalone_parameters.yaml.txt.gz16:51
raukadahweshay: rlandy https://logs.rdoproject.org/70/21270/39/check/periodic-tripleo-ci-rhel-8-standalone-master/e7ea6da/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz16:52
raukadahsame error still16:52
raukadahwith py3 interpreter16:52
weshayraukadah ya.. that's unrelated16:53
raukadahaye sir17:05
*** derekh has quit IRC17:10
*** ykarel|away has quit IRC17:17
*** tesseract has quit IRC17:24
weshayraukadah just fyi.. the containers are pulling just fine from the rdo registry, the upload the local undercloud was failing17:35
weshayraukadah rlandy are retrying w/o the patch that removes the port addr on the local registry insecure address17:35
*** rlandy is now known as rlandy|brb17:36
*** bogdando has quit IRC17:42
sshnaidm|offrfolco|rover, what is taigacli server on tripleo-infra?17:45
sshnaidm|offrfolco|rover, do you still need it?17:46
*** udesale has quit IRC17:48
*** tosky has joined #oooq17:49
weshaysshnaidm|off let's keep that one17:49
weshaysshnaidm|off that one should be failry small17:53
sshnaidm|offweshay, rfolco|rover we can use hubbot, or tempestmail server17:53
weshayah that's tru too17:54
sshnaidm|offit doesn't make sense to maintain a separate server for each script17:54
weshayagree17:54
sshnaidm|offweshay, when moving to vexxhost we need to ask for bigger infra tenant17:55
weshaysshnaidm|off we should ask for a bigger EVERYTHING17:58
*** rlandy|brb is now known as rlandy17:59
raukadahmove to k8s and containerize it in a single server18:04
rfolco|roversshnaidm|off, we can move to tempestmail box18:11
rfolco|roverk18:11
rlandyweshay: we need to revert that patch18:12
rlandyhttp://logs.rdoproject.org/70/21270/40/check/periodic-tripleo-ci-rhel-8-standalone-master/f19776f/job-output.txt.gz18:12
rlandyor fix it18:12
weshayrlandy which patch?18:14
rlandyweshay: panda's https://review.rdoproject.org/r/#/c/21215/18:14
pandammhhh, so they don't accept failure, not even with rescue18:15
rlandypanda: it will never work this way18:15
pandaeither it passes or we have to ingore errors.18:15
rlandyneed become18:15
pandarlandy: even with taht, the failures should be ignoresd as we have a rescue18:15
rlandypanda: I'm going to revert18:16
pandarlandy: ok18:16
rlandyto give us time to rethink it18:16
rlandypanda: I am here today and monday - we can try merge it again with become + whatever changes you want to make18:16
pandaah the pakcage task, doesn't have the become18:17
rlandypanda: do you want to try a pact just with become in place of revert?18:17
pandathat one is failing18:17
rlandycorrect18:17
pandawithout a rescue18:17
pandabah, ensuring a package is *not* there should not require a become. :/18:18
rlandypanda: they are all failing but only the last one fails the job18:18
pandayep18:18
pandadammit, locally on centos7 even without become, that task did not fail18:23
pandathe package were absent, and ansible did not complain18:23
rlandypanda: k - merging revert - please put the new review on the etherpad when you have it and I'll review it18:27
rlandyweshay: ^^ will rerun after that merge18:27
*** bogdando has joined #oooq18:38
*** bogdando has quit IRC18:51
*** bogdando has joined #oooq18:52
*** bogdando has quit IRC18:52
pandarlandy: added https://review.rdoproject.org/r/21537 to the file19:01
rlandylooking19:01
rlandypanda: ok - let's try that after this current standalone run19:05
rlandyin case we just want to run that one again quickly19:05
pandarlandy: ok, I remember why I didn't put become originally. If you login as root, you can push or pull only as root, and I wasn't sure that was the user we use19:06
rlandywe got a fail - let's see19:08
rlandyweshay: invalid URL '192.168.24.1:8787': parse 192.168.24.1:8787: first path segment in URL cannot contain colon19:16
rlandyno good19:16
rlandyhttp://logs.rdoproject.org/70/21270/40/check/periodic-tripleo-ci-rhel-8-standalone-master/a9c577b/logs/undercloud/etc/containers/registries.conf.txt.gz19:16
rlandyweshay: ^^ same as yours does not work19:17
rlandypanda: k - let's try your patch again19:26
rfolco|roverdoes anybody know on top of your head how these featureset files are rendered ? https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/general_config/featureset041.yml - lookng for the python code/method in tq that parses/renders jinja into final yaml config19:28
sshnaidm|offrfolco|rover, it's given to ansible as -e @file and it renders jinja19:29
*** holser has joined #oooq19:30
rfolco|roversshnaidm|off, oh I see. Thanks19:32
rfolco|roversshnaidm|off, I am looking at python lib to do it actually... maybe jinja219:32
*** Vorrtex has quit IRC19:33
sshnaidm|offrfolco|rover, yes, it is19:34
sshnaidm|offrfolco|rover, I use it in sova, like: https://github.com/sshnaidm/sova/blob/master/tripleoci/page.py#L62-L6819:34
rfolco|roversshnaidm|off, excelent, thats what I am looking for19:35
weshayrlandy ugh.. so we're screwed w/ and w/o the port eh?19:35
weshayhrm.. ok..19:35
weshaywhat eles could it be.19:37
weshayrlandy so we put the change back and deal w/ the htaccess19:39
*** jtomasek has quit IRC19:45
rlandyweshay: yep - I think we have to20:03
weshayrlandy ya20:04
* weshay looking at tripleo-common code20:04
rlandyweshay: I merged panda's second attempt .. https://review.rdoproject.org/r/#/c/21537/20:06
weshayrlandy ok.. let's take it for a spin20:08
rlandyweshay: yep - rekicking20:08
rlandyweshay: also restoring https://review.opendev.org/#/c/669918/20:09
*** brault has joined #oooq20:34
*** holser has quit IRC20:48
*** holser has joined #oooq20:48
*** aakarsh has quit IRC20:55
*** holser has quit IRC21:14
weshay:26 <EvilienM>  it says 002221:28
weshay15:26 <EvilienM>  for me21:28
weshay15:27 <weshay>  [zuul@upstream-rhel-8-rdo-cloud-0002079204 ~]$ umask                                                                                                                                           │·····················21:28
weshay15:27 <weshay>  000221:28
weshay15:27 <weshay>  boom?21:28
weshay15:27 <EvilienM>  here you go21:28
weshay15:27 <EvilienM>  alright21:28
weshay15:27 <EvilienM>  have a good weekend21:28
weshay15:27 <weshay>  lolz thanks21:28
weshay15:27 <EvilienM>  your nodepool is broken21:28
weshay15:27 <weshay>  aye21:28
*** aakarsh has joined #oooq21:31
weshayrlandy we need to set umask to 022 in https://code.engineering.redhat.com/gerrit/gitweb?p=openstack/sf-config.git;a=blob;f=nodepool/elements/virt-customize/rhel8-cloud.yaml;h=60a02389e20f7af2e2299250206d26500a1cf6c5;hb=refs/heads/master21:37
*** holser has joined #oooq21:38
rlandyhttps://softwarefactory-project.io/cgit/config/tree/nodepool/elements/21:38
rlandyhttps://softwarefactory-project.io/cgit/config/tree/nodepool/elements/rhel-8-packages-needs21:38
*** tosky has quit IRC21:43
*** brault has quit IRC21:44
*** vinaykns has quit IRC21:46
*** holser has quit IRC22:13
*** holser has joined #oooq22:14
*** holser has quit IRC22:17
*** tcw has quit IRC22:33
rlandyweshay: https://softwarefactory-project.io/r/1595222:34
*** tcw has joined #oooq22:37
rlandyweshay: you back?22:40
*** rascasoft has quit IRC22:40
weshayrlandy aye22:40
weshaylooks like it got further22:40
weshayw/ the umask22:40
rlandyweshay: https://softwarefactory-project.io/r/#/c/15952/1/nodepool/elements/rhel-8-packages-needs/install.d/10-packages <- correct?22:40
rlandyshould take effect on next login22:41
weshayI think 02222:41
rlandyok - misread chat above - changing22:41
*** rascasoft has joined #oooq22:42
rlandyk - updated22:42
rlandyadding downstream22:42
weshayrlandy I write up a summary email22:48
rlandyweshay: ack - just submitting downstream review22:49
rlandyweshay: https://code.engineering.redhat.com/gerrit/176555 Set umask to 0022 for RHEL 822:51
weshayrlandy hrm.. so now that I'm not rushing22:55
weshay002 seems to be more open than 02222:55
weshayugh22:57
weshay# By default, we want umask to get set. This sets it for login shell22:57
weshay# Current threshold for system reserved uid/gids is 20022:57
weshay# You could check uidgid reservation validity in22:57
weshay# /usr/share/doc/setup-*/uidgid file22:57
weshayif [ $UID -gt 199 ] && [ "`id -gn`" = "`id -un`" ]; then22:57
weshay    umask 00222:57
weshayelse22:57
weshay    umask 02222:57
weshayfi22:57
weshayin /etc/profile22:58
rlandyweshay: so I change it back?22:58
weshayrlandy perhaps we should hold off on image modify.. and update the umask in a task in ci22:58
weshayso we can test it more readily22:58
rlandyI have the email ready to send22:58
weshaybecause I'm a little fuzzy22:58
weshayoh k22:58
rlandyweshay: fine - I'll mark the review wf -222:59
rlandyand we can try that on Monday22:59
rlandyweshay: k - email sent23:01
rlandyweshay: we can try that comment in the release file.23:01
weshayrlandy k.. going to send mine as well23:01
rlandyk - will read on sunday23:02
rlandyweshay: do you want to keep the node>23:02
rlandythe held node?23:02
rlandyeither I need to keep the session open or add your keys23:02
weshayrlandy not sure how much value it is at this point23:03
weshayrlandy I think some poking at changing the umask prior to kicking off standalone would be more informative at this point23:03
weshaychandan can do that23:03
rlandyk - I'll give the node back23:05
weshayrlandy ok.. thanks for the help!!!23:06
rlandysure - didn't do much23:06
rlandysee you next week23:06
*** rlandy has quit IRC23:07

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!