Wednesday, 2020-08-19

*** Goneri has quit IRC00:41
*** ysandeep|away is now known as ysandeep01:46
ysandeeprlandy|bbl, ^^ i will try to debug - Invalid database user cinder@02:00
*** weshay|ruck has quit IRC02:05
*** weshay has joined #oooq02:06
*** apetrich has quit IRC02:14
ysandeeprlandy|bbl, I think i found something,  notice - "cinder::keystone::authtoken::auth_uri": "http://%2522%2522:5000", - https://sf.hosted.upshift.rdu2.redhat.com/logs/75/208575/31/check/periodic-tripleo-ci-rhel-8-standalone-rhos-16-2/0e5c0b1/logs/undercloud/etc/puppet/hieradata/service_configs.json02:23
ysandeepvs "cinder::keystone::authtoken::auth_uri": "http://192.168.24.1:5000", in good jobs02:23
*** jmasud has quit IRC02:33
*** jmasud has joined #oooq02:36
*** rlandy|bbl is now known as rlandy02:38
rlandyysandeep: hey - yeah ... if you can look into ^^ that would be great02:38
rlandysee you in a few hours02:38
*** rlandy has quit IRC02:38
*** rfolco has quit IRC02:54
*** ykarel|away has joined #oooq03:48
*** ykarel|away is now known as ykarel04:14
*** saneax has joined #oooq04:18
*** raukadah is now known as chkumar|rover04:25
*** skramaja has joined #oooq04:37
*** yolanda has quit IRC04:41
*** ratailor has joined #oooq04:42
ykarelchkumar|rover, is issue known for centos7 pip?04:47
ykarelfailure example:- https://f05144106d047809c3b3-8b5cc018a02fed487ca2a6d59a4ee9d5.ssl.cf2.rackcdn.com/746592/1/gate/tripleo-ci-centos-7-containers-multinode-train/12933b7/job-output.txt04:47
chkumar|roverykarel: nope04:48
ykareli seen multiple failures, seems mirror related04:48
ykarelwith some it pass and some it fail04:48
chkumar|roverykarel: noted on hackmd05:02
ykarelchkumar|rover, ack it's affectoing gate job05:02
ykarelchkumar|rover, and is train standalone issue known?05:03
ykarel The task includes an option with an undefined variable. The error was: the inline if-expression on line 1 evaluated to false and no else section was defined. ?05:03
ykarelseen in rdo gate job, should also be hitting in periodic and gate05:03
chkumar|roverykarel: https://0af46641a0193018034c-60d68d0bfd7f8f0178e466da97b3a245.ssl.cf2.rackcdn.com/746160/3/gate/tripleo-ci-centos-8-standalone/053a58b/job-output.txt05:04
ykarelchkumar|rover, okk perfect05:04
ykarellikely due to control vip patch05:05
chkumar|roverykarel: https://review.opendev.org/725782 ?05:06
ykarelchkumar|rover, yes05:07
ykareli see there is test patch from ysandeep on changing this05:07
ykarelhttps://review.opendev.org/#/c/743464/3/roles/standalone/tasks/main.yml05:07
ykarelcan add else clause to fix it05:08
chkumar|roverykarel: let me open the bug, you can send the fix05:09
ysandeepchkumar|rover, i think ronelle already opened the bug05:09
ysandeepchkumar|rover, ykarel https://review.opendev.org/74677205:09
ysandeepchkumar|rover, https://bugs.launchpad.net/tripleo/+bug/189207805:11
openstackLaunchpad bug 1892078 in tripleo "Train standalone jobs are failing check/gate - unable to resolve tripleo_deploy_control_virtual_ip" [Critical,In progress] - Assigned to Ronelle Landy (rlandy)05:11
ykarelhmm ^^ is it, but it's failing need to check05:11
ysandeepmy patch was just for my downstream work till this issue is fixed, you can ignore that can we can focus on rlandy patch https://review.opendev.org/#/c/746772/05:12
chkumar|roverhttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_aeb/746772/4/check/tripleo-ci-centos-8-scenario001-standalone/aeb554e/logs/undercloud/home/zuul/standalone_deploy.log05:12
chkumar|rover '\"192.168.24.3\"/32' is not valid under any of the given schemas",05:13
ykarelextra quotes are injected05:13
chkumar|roveryes05:13
chkumar|roverremoving quotes will fix it i think05:14
chkumar|rover "{{ standalone_ha_control_virtual_ip }}"05:14
ykarelyes +105:14
chkumar|roverlet me update that05:14
ykarelack05:15
ykarelchkumar|rover, i think also need to handle tripleo_deploy_ha_control_plane not defined case05:16
ykareltripleo_deploy_ha_control_plane|default(false)|bool should handle it05:16
ykareland also standalone_ha_control_virtual_ip --> standalone_ha_control_virtual_ip|default()05:17
ykarelchkumar|rover, ^ need to handle ^^ too05:19
chkumar|roveron it05:20
chkumar|rovernow check05:22
chkumar|roverykarel: ^^05:22
*** ykarel_ has joined #oooq05:22
ykarel_chkumar|rover, okk ignore, first one also not necssary as it's always defined05:23
ykarel_just test with train to ensure nothing else is missed05:23
chkumar|rover tripleo_deploy_ha_control_plane|default(false)|bool should handle it  ?05:23
ykarel_u already send a patch, let me check05:24
chkumar|roverok05:24
*** ykarel has quit IRC05:25
ykarel_chkumar|rover, you can revert to previous patch, extra conditions are not needed, if want to keep then replace  tripleo_deploy_ha_control_plane|bool and tripleo_deploy_ha_control_plane|default(false)|bool --> tripleo_deploy_ha_control_plane|default(false)|bool05:26
ykarel_as both are duplicated05:27
chkumar|roveryes05:28
chkumar|rovernow check05:29
ykarel_chkumar|rover, now looks better, let's see how ci goes05:30
ykarel_also test in train standalone05:30
*** ykarel__ has joined #oooq05:33
chkumar|roverykarel_: sending05:35
*** ykarel_ has quit IRC05:36
*** ykarel__ is now known as ykarel05:45
*** marios has joined #oooq05:58
*** marios is now known as marios|ruck05:58
*** jfrancoa has joined #oooq06:02
*** udesale has joined #oooq06:02
*** dmellado has quit IRC06:11
marios|ruckchkumar|rover: \o/06:28
chkumar|rovermarios|ruck: \o06:28
chkumar|rovermarios|ruck: https://review.opendev.org/746772 with ref https://launchpad.net/bugs/1892078 please have a look, thanks!06:29
openstackLaunchpad bug 1892078 in tripleo "Train standalone jobs are failing check/gate - unable to resolve tripleo_deploy_control_virtual_ip" [Critical,In progress] - Assigned to chandan kumar (chkumar246)06:29
marios|ruckchkumar|rover: ack06:29
*** apetrich has joined #oooq06:29
*** dmellado has joined #oooq06:41
*** yolanda has joined #oooq06:50
chkumar|rovermarios|ruck: https://review.opendev.org/#/q/fbd575cbe88400bad1ce4d37b914d7ae8a43e15c for https://bugs.launchpad.net/tripleo/+bug/189200806:50
openstackLaunchpad bug 1892008 in tripleo "periodic OVB train centos 7 failing overcloud configuration (config download) Invalid cross-device link" [Critical,In progress] - Assigned to Oliver Walsh (owalsh)06:50
marios|ruckchkumar|rover: yeah thanks i saw that one yesterday will do ... was happy to see it was picked up so quickly06:52
marios|ruckno retrospective in calendar chkumar|rover ?06:54
chkumar|rovermarios|ruck: yes, might be rfolco missed it06:54
chkumar|roverlet me send it06:55
marios|ruckack06:55
marios|ruckysandeep: o/ forgot to share this the other day if you are trying out irssi might find it useful https://quadpoint.org/articles/irssi/#hilight-window06:57
marios|ruckakahat: ^06:57
ysandeepmarios|ruck, thank you06:57
*** jbadiapa has joined #oooq06:57
ysandeepmarios|ruck, i found it fascinating using irc from terminal :)06:58
marios|ruckysandeep: ack its cool once you have it setup and get used to it but its just a matter of preference some people don't like it07:04
*** dmellado has quit IRC07:07
marios|ruckchkumar|rover: actually not sure it will interrupt the check run07:17
marios|ruckchkumar|rover: nice we got the train promotion (as we expected) last night \o/07:18
marios|ruckchkumar|rover: so very green for prog call07:18
chkumar|roverawesome07:21
chkumar|roverykarel: c8 train standalone is now running at os_tempest07:22
chkumar|rovermarios|ruck: got the retro invite?07:28
marios|ruckchkumar|rover: checking07:28
marios|ruckgmail being a bit weird here07:28
marios|ruckchkumar|rover: ack got it07:28
chkumar|rovermarios|ruck: cool, thanks!07:29
marios|ruckchkumar|rover: hmm same time as program call07:29
chkumar|rovermarios|ruck: want me to shift a little?07:29
chkumar|rovermay be 30 mins later07:29
marios|ruckchkumar|rover: yeah OK07:29
marios|ruckchkumar|rover: i might join a little late then07:29
marios|ruckvs missing it07:29
chkumar|roverdone07:30
marios|ruckthanks07:30
marios|ruckweshay: will you cancel today's cix (clash with open forum all hands) i think last time it was cancelled07:31
marios|ruckweshay: asking you cos jbuchta is out07:31
ykarelchkumar|rover, ack07:35
akahatmarios|ruck, thank you!!07:44
marios|ruckchkumar|rover: rate limiting still a thing https://bugs.launchpad.net/tripleo/+bug/1889122/comments/4407:45
openstackLaunchpad bug 1889122 in tripleo "mirror timeouts in upstream causing undercloud and standalone failures" [Critical,Triaged] - Assigned to Alex Schultz (alex-schultz)07:45
marios|ruckakahat: :) np07:45
chkumar|rovermarios|ruck: yes07:49
Tengucan't we talk to infra and drop that rate limitation? since it's "ours"....07:51
marios|ruckchkumar|rover: fyi fix-released @ https://bugs.launchpad.net/tripleo/+bug/1891971/comments/207:51
openstackLaunchpad bug 1891971 in tripleo "ERROR: No matching distribution found for pprint (from -r /home/zuul/src/opendev.org/openstack/tripleo-ci/test-requirements.txt (line 6))" [Critical,Fix released]07:51
chkumar|rovermarios|ruck: yes07:52
chkumar|roverTengu: Does pep8 glanceclient issue fixed?07:52
Tenguchkumar|rover: seems to be fixed yep07:53
*** dmellado has joined #oooq07:53
Tenguhitting other issues, but not related to that one07:53
*** jpena|off is now known as jpena07:58
*** dtantsur|afk is now known as dtantsur08:16
zbrchkumar|rover: https://review.opendev.org/#/c/731497/ please (very easy)08:25
chkumar|roverzbr: thanks!08:26
*** tosky has joined #oooq08:28
chkumar|roverysandeep: do we have a dnm patch for rhos-16.2 https://review.opendev.org/#/c/745819/ testing?09:01
ysandeepchkumar|rover, i tested it for previous patchset, but 16.2 are hitting number of issues so i have removed for now - Let me try to grab previous logs09:04
chkumar|roverysandeep: I just wanted to make sure package is getting built and installed09:05
chkumar|roverysandeep: can you pass me the log url?09:05
ysandeepchkumar|rover, Without https://review.opendev.org/#/c/745819/ it was failing earlier on :-09:13
ysandeephttps://sf.hosted.upshift.rdu2.redhat.com/logs/75/208575/19/check/periodic-tripleo-ci-rhel-8-standalone-rhos-16-2/24f7a98/job-output.txt09:13
ysandeepWith https://review.opendev.org/#/c/745819/, it passed earlier step but later hit other issue and failed on some unit tests:-09:13
ysandeephttps://sf.hosted.upshift.rdu2.redhat.com/logs/75/208575/21/check/periodic-tripleo-ci-rhel-8-standalone-rhos-16-2/b31aec6/logs/delorean_logs/component/tripleo/0d/8d/0d8d7d375f136d7f3d6c9f0ffe68c87fc8ae84b4_dev_dev/mock.log.txt.gz09:13
ysandeepRight now, we have removed depends-on which was causing trigger of buildtestpackage role(to avoid above unit test failure issue), Once basic standalone job is green i have plans to retest it.09:13
ysandeepwe can keep https://review.opendev.org/#/c/745819/ on hold, if you want to see run again09:14
chkumar|roverysandeep: ok. let's test later, since it is not going to affect upstream09:15
chkumar|roverysandeep: thanks for the link09:15
*** apetrich has quit IRC09:15
ysandeepchkumar|rover, thanks to you for review , I might buzz you for help with buildtestpackage role if I hit more issues.09:17
chkumar|roverysandeep: https://sf.hosted.upshift.rdu2.redhat.com/logs/75/208575/21/check/periodic-tripleo-ci-rhel-8-standalone-rhos-16-2/b31aec6/logs/delorean_logs/projects.ini09:19
chkumar|roverdownstream_distro_branch=rhos-16.2-rhel-8-trunk09:19
chkumar|roverdownstream_tag=osp-16.209:19
chkumar|roverlooks ok09:19
ysandeepyes09:21
chkumar|roverysandeep: thanks for working on that :-)09:21
ysandeepchkumar++ :)09:22
*** ratailor has quit IRC09:27
*** ratailor_ has joined #oooq09:27
*** ysandeep is now known as ysandeep|lunch09:35
marios|ruckchkumar|rover: oof bad buildset for train but all known/hitting https://bugs.launchpad.net/tripleo/+bug/189207809:42
openstackLaunchpad bug 1892078 in tripleo "Train standalone jobs are failing check/gate - unable to resolve tripleo_deploy_control_virtual_ip" [Critical,In progress] - Assigned to chandan kumar (chkumar246)09:42
marios|ruckchkumar|rover: well, also some tempest09:42
chkumar|rovermarios|ruck: https://review.opendev.org/#/c/746772/ will fix it and tested it on train here https://review.opendev.org/#/c/746835/09:43
marios|ruckchkumar|rover: yeah but anyway we're good for train handoff 3 promotions this week.09:43
* marios|ruck shakes fist at ussuri09:43
chkumar|rovermarios|ruck: yes09:44
marios|ruckchkumar|rover: oof bad buildset for train but all known/hitting https://bugs.launchpad.net/tripleo/+bug/189207809:44
openstackLaunchpad bug 1892078 in tripleo "Train standalone jobs are failing check/gate - unable to resolve tripleo_deploy_control_virtual_ip" [Critical,In progress] - Assigned to chandan kumar (chkumar246)09:44
marios|rucksorry accidental duplication ^09:44
chkumar|rovermarios|ruck: for ussuri we need to merge one https://review.opendev.org/#/c/746533/09:44
marios|ruckmaster promoting/will promote09:44
marios|ruck        * https://review.rdoproject.org/zuul/buildset/2b8df6b02eb14d4196a6e6f916337f0309:44
chkumar|roverthen all will be green09:44
chkumar|rovermarios|ruck: cool09:45
marios|ruckchkumar|rover: yeah must be the minor update fix09:45
marios|rucklooking09:45
marios|ruckack09:45
marios|ruckchkumar|rover: so maybe we get lucky and it promotes ussuri tonight/tomorrow morning09:45
chkumar|roverhttps://review.opendev.org/#/c/725782/ introduced regression which brought https://bugs.launchpad.net/tripleo/+bug/189207809:46
openstackLaunchpad bug 1892078 in tripleo "Train standalone jobs are failing check/gate - unable to resolve tripleo_deploy_control_virtual_ip" [Critical,In progress] - Assigned to chandan kumar (chkumar246)09:46
*** ysandeep|lunch is now known as ysandeep09:55
marios|ruckchkumar|rover: fyi new         * https://bugs.launchpad.net/tripleo/+bug/1892169 but not a blocker (commented out in criteria)10:03
openstackLaunchpad bug 1892169 in tripleo "periodic centos 8 train FS20 tempest fails test_server_connectivity_resize" [Undecided,Triaged]10:03
chkumar|rovermarios|ruck: ack, will take a look10:03
*** ysandeep is now known as ysandeep|brb10:27
chkumar|rovermarios|ruck: so rate limit has arrived it is seen in most of the check jobs now10:43
chkumar|roverespecially c8 multinode jobs10:43
marios|ruckchkumar|rover: yeah https://bugs.launchpad.net/tripleo/+bug/1889122/comments/4410:44
openstackLaunchpad bug 1889122 in tripleo "mirror timeouts in upstream causing undercloud and standalone failures" [Critical,Triaged] - Assigned to Alex Schultz (alex-schultz)10:44
marios|ruckchkumar|rover: we still have it on our hackmd anyway10:44
marios|ruckchkumar|rover: you still working on https://bugs.launchpad.net/tripleo/+bug/1891372 ? i added a comment there it affects stein too looks like10:46
openstackLaunchpad bug 1891372 in tripleo "rocky periodic jobs are failing with " Error: image tripleorocky/centos-binary-tempest:9801dc7461cbd6cbd73868e72e74d21d586c6708_fbb4de96-updated-20200812131025 not found"" [Critical,In progress] - Assigned to chandan kumar (chkumar246)10:46
chkumar|rovermarios|ruck: checking10:48
*** ysandeep|brb is now known as ysandeep10:49
zbrchkumar|rover: weshay https://review.opendev.org/#/c/730389/ (easy)11:02
*** jpena is now known as jpena|lunch11:31
*** apetrich has joined #oooq11:46
*** apetrich has quit IRC11:52
*** apetrich has joined #oooq11:52
weshaymarios|ruck, fyi.. for next sprint https://review.opendev.org/#/c/731100/511:54
weshaychkumar|rover, I think we can cancel todays retro11:55
weshayand just hold it tomorrow11:55
weshaythat's what rfolco and I discussed anyway11:55
chkumar|roverweshay: so tomorrow we have retro and planning both?11:58
weshaychkumar|rover, since there is not that much to plan.. I think we can do it11:58
weshaychkumar|rover, wdyt?11:58
chkumar|roverweshay: sounds ok to me11:59
chkumar|rovermarios|ruck: bhagyashris arxcruz sshnaidm ysandeep svyas retro cancelled today11:59
ysandeepchkumar|rover, ack12:00
chkumar|roverwe will be doing retro and planning together tomorrow12:00
marios|ruckchkumar|rover: ack12:01
marios|ruckchkumar|rover: i will add couple cards am out tomorrow12:01
chkumar|rovermarios|ruck: sure thanks!12:01
*** rlandy has joined #oooq12:02
marios|ruckweshay: looking (in prog call)12:02
marios|ruckweshay: ack ok12:02
*** rfolco has joined #oooq12:03
rlandyysandeep: hello12:04
ysandeeprlandy, weshay \o/ good news we got our first 16.2 Standalone Green run :)12:04
rlandyysandeep: that's great .. more worried about the control_virtual_ip change12:04
ysandeeprlandy chkumar|rover fixed that12:04
rlandychkumar|rover++12:06
rlandyysandeep: that's great news12:07
rlandywell done12:07
rlandyysandeep: ok - so I am closing https://bugzilla.redhat.com/show_bug.cgi?id=186937912:08
openstackbugzilla.redhat.com bug 1869379 in rabbitmq-server "[OSP 16.2] Containers fail to start for step 2 - mysql_init_bundle error" [High,New] - Assigned to plemenko12:08
ysandeeprlandy, go ahead12:09
ysandeeprlandy, for standalone - with latest podman i faced some issues(details in email i sent to you), I used older podman to get green run. But from changelog it looks like latest and older podman don't have much difference.. May be we are hitting random issues - I am retriggering more jobs to confirm if podman is the issue.12:09
arxcruzrfolco: don't we have sync today ?12:09
rfolconope arxcruz12:09
ysandeeprlandy, https://code.engineering.redhat.com/gerrit/#/c/209162/ had to come put with this dirty hack to install older version of podman12:10
ysandeeptemporary ^^12:10
rlandyysandeep: that should be set in container tools - let me check that12:12
ysandeeps/put/up12:13
ysandeeprlandy, ack12:13
rlandyoh - the update to rhel 8.312:13
rlandyrhos-release12:13
rlandyappstream12:13
rlandythis should be locked in container tools12:14
ysandeepyes we are getting new rpm from rhosp-rhel-8.3-appstream12:15
ysandeepi have run two job 1 with older version and 1 with new version - https://code.engineering.redhat.com/gerrit/#/c/208575/ & https://code.engineering.redhat.com/gerrit/#/c/201382/ lets see how this goes12:16
rlandyysandeep: so currently youhave two jobs running, right?12:16
rlandyyep12:16
ysandeepyes12:16
rlandylooks like 1 just failed12:16
rlandylet's check the versions12:16
rlandyI think we have to lock container tools12:17
rlandyI though rhos-release was doing that12:17
ysandeepfailed one was with newer podman version (so in total 4/4 jobs failed with newer podman)12:17
rlandybuildah.x86_64                                1.11.6-8.module+el8.3.0+7385+02cfa547           @rhosp-rhel-8.3-appstream12:18
chkumar|rovermarios|ruck: https://review.opendev.org/746894 will fix rocky stein tempest container issue12:18
rlandyshoot - they upgraded buildah as well12:18
rlandybut at leats not to 1.1412:18
rlandypodman.x86_64                                 1.6.4-13.module+el8.3.0+7385+02cfa547           @rhosp-rhel-8.3-appstream12:19
ysandeeprlandy, i got success with podman.x86_64 1.6.4-12.module+el8.2.0+6670+014d0ff812:19
ysandeephttps://sf.hosted.upshift.rdu2.redhat.com/logs/75/208575/33/check/periodic-tripleo-ci-rhel-8-standalone-rhos-16-2/953874d/logs/undercloud/var/log/extra/package-list-installed.txt12:20
rlandyysandeep: so we definitely need to lock it12:20
rlandythe question is hwy is rhos-release not12:20
ysandeeprlandy, yup12:20
ysandeeprlandy, i think this started after https://code.engineering.redhat.com/gerrit/#/c/209089/12:21
rfolcoweshay, pls cancel today's retro12:21
rfolcoI 'll send a new invite for tomorrow12:21
ysandeeprfolco, chandan canceled today's invite12:22
rlandyysandeep: yes it did12:22
rlandy# ACTION: dnf -y module disable container-tools:rhel812:22
rlandy# ACTION: dnf -y module enable container-tools:2.012:22
rlandyI need to find out how this all works12:22
rfolcoysandeep, ok thx12:23
rfolcoinvite sent for tomorrow chkumar|rover12:23
ysandeeprlandy, me too otherwise this thing will keep on banging our heads12:23
rlandyykarel: hi - saw your discussion container-tools on #rhos-delivery12:23
rlandyI thought that rhos-release locked us to container-tools 2.0?12:24
rlandyper lines above12:24
rlandymaybe we need to execute that again?12:24
chkumar|roverrfolco: thanks!12:25
rlandyshreshtha> ykarel, it coming from module-container-tools-rhel8-8020120200601155013-ffd2803a12:25
rlandyysandeep: ^^ we should be 2.012:25
rlandyysandeep: let's try another test patch ...12:26
*** sshnaidm is now known as sshnaidm|afk12:26
rlandyin the release file, in stead of taking the old bets repo,12:26
rlandytry this ...12:26
rlandyhttps://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/CentOS-8/master.yml#L16212:28
ykarelrlandy, hi12:28
ykarelrlandy, execute again, what?12:28
rlandyykarel: so ... rhos-release prints out ...12:29
ykarelin rhos-delivery we were discussing compose failures12:29
rlandy# ACTION: dnf -y module disable container-tools:rhel812:29
rlandy# ACTION: dnf -y module enable container-tools:2.012:29
rlandyso I thought it locked the container-tools to 2.012:29
ykarelyes it's locking to container-tools:2.012:29
rlandyok - so it is ...12:30
rlandyysandeep's job picked up podman 1.6.4-13.module+el8.3.0+7385+02cfa54712:30
rlandywhich fails12:30
ykarel^ part of container-tools:2.012:31
rlandysince rhos-release was updated to follow latest ... https://code.engineering.redhat.com/gerrit/#/c/209089/2/repos/rhos-release-rhel-8.3.repo12:31
rlandyso version are not locked even in container-tools:2.012:31
ykarelso stream can ship multiple versions12:32
ykarelnewer wins12:32
ykarelunless and until it's excluded12:32
rlandyykarel: what's the right way to go about this? log a bug to podman?12:33
rlandyhow would we exclude a version?12:33
rlandyI see buildah updated as well but that seems ok12:34
ykarelrlandy, yes i think bug needs to be created, but you can discuss with Emilien, he used to deal with podman12:35
ykarelrlandy, wrt exclude, i just checked rhel8.3 repo, only 1 version is shipped in container-tools:2.012:35
ykarelso nothing to exclude12:36
rlandyykarel: last question ... how do version get included in container-tools? is there a qualification process?12:36
*** jpena|lunch is now known as jpena12:37
rlandyhttps://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_podman/vars/redhat.yml12:39
rlandy^^ would it help to lock this more specifically?12:40
rlandyysandeep: we haven't done a container build since the rhos-release update12:44
rlandybetter try that12:44
ysandeeprlandy, okay, rechecking container build12:44
marios|ruckrfolco: chkumar|rover: weshay: reminder /me out tomorrow for few days (see mail, also will send status later today )12:45
rlandyysandeep: wait - it will fail12:46
rlandylet me put back a change ...12:46
ysandeeprlandy, okay12:46
chkumar|rovermarios|ruck: Happy holidays :-)12:46
chkumar|roverweshay: we need to find new ruck & rovers for next sprint12:46
marios|ruck\o/ thanks chkumar|rover i really need a break12:46
*** weshay is now known as weshay|interview12:47
*** Goneri has joined #oooq12:47
rlandyysandeep: ok - updated and rekicked12:49
ykarelrlandy, https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_podman/vars/redhat.yml will only help if enabled module stream have that version12:49
ykarelrlandy, what you mean by qualification process?12:50
ykarelyou mean qa expect? like what tests it's run before shipping12:50
rlandyykarel: run tests before updating the versions12:50
ykarelrlandy, no i am not aware of that12:50
ysandeeprlandy, ack thanks!12:51
rlandyysandeep: let's get a handle on how different the two versions are12:51
rlandyI saw your comparison in the email12:52
ysandeeprlandy, I am wondering if we locally extract rpm, we will be able to see the actual differences. I am not aware if we keep track somewhere or other easy way to do this.12:55
rlandyysandeep: short of looking through the commits in between, I don't have a better idea12:57
rlandyI want to see what happens with container builds12:57
rfolcozbr, arxcruz: would you review https://review.rdoproject.org/r/#/c/29003/ https://review.rdoproject.org/r/#/c/29004 https://review.rdoproject.org/r/#/c/28901/ and https://review.rdoproject.org/r/#/c/28902/13:00
rlandyysandeep: this  changing rhel base is going to make us all crazy!13:01
rlandyif the steam is not locked13:01
rlandyI don't see how we operate13:01
ysandeeprlandy, :(13:01
rfolcorlandy, marios|ruck chkumar|rover akahat ysandeep bhagyashris svyas pojadhav: pls add above patches to your review queue if possible. Thanks.13:02
rfolcospecially you zbr, arxcruz akahat bhagyashris ^13:02
akahatrfolco, ack13:03
bhagyashrisrfolco, ack13:03
zbrrfolco: where is the value of changing the job name for periodic ones? IMHO is bad because I loose ability to see history of a job run on chances and periodi13:05
rfolcozbr, I am replacing them with the new ones, discarding legacy ones13:05
rfolcozbr, not renaming13:05
zbrtrue and that is why i approved, but my question still stands13:06
zbrwhy another name if no special reason. in fact there was something around file filters in the past, right?13:07
zbrbut I do not remember exactly13:07
rlandyysandeep: let's ask at the prod chain meeting13:07
rlandyforwarding you the invite13:07
rfolcozbr, mol-* are the new ones... molecule-delegated-* are the legacy ones. The reason is that the old ones run a totally different workflow. The new ones were just created fresh to not break the existing ones.13:07
rfolcozbr, On our design we agreed to create new ones without touching the old ones13:08
rfolcoexcept the role itself, but that was necessary13:08
zbrsure, question is unrelated to you patch.13:08
zbrtake a look at https://review.rdoproject.org/zuul/builds?job_name=mol-promoter#13:08
zbrdo you see gate, check and periodic? right?13:09
zbrdoes it give you good indication if something went wrong, as periodic should had also fail13:09
rfolcozbr, I don't understand... trying to process your comment...13:10
zbrmy question is: why we would still need to create additional child jobs that run periodic, they prevent us from seeing a page like above13:10
*** weshay|interview is now known as weshay13:10
rfolcoaaaah13:11
rfolconame should be preserved your mean for periodic ones13:11
rfolcook, I kept the same structure, just that. But yeah, I don't see any reason for having a child job just for periodic, we should be able to use the same names.... and we should be able to define them in ci-config13:12
rfolconot in rdo-jobs13:12
weshayrlandy, good morning..13:13
rfolcozbr, if that is an impediment, please -113:13
svyasrfolco, ack13:13
zbrrfolco: nope, it is but is unrelated to your change.13:13
rfolcozbr, yeah I can propose a change now to consolidate the jobs if you agree13:14
weshayrlandy, saw your pings from yesterday.. let's chat about the ansible role path when we have a sec.. but for the prod-chain council meeting..  I think we want to get insight on which repos to use for RHEL deps pipeline right?13:14
rlandyweshay: ack ... need to ask about rhel there13:18
weshayaye13:18
rlandyweshay: ysandeep: I think I finally understand why people has money for stable versions of linux13:18
rlandypay13:18
weshaylolz13:18
rlandyI'd pay for stable rhel right now13:18
ysandeephahahaa13:18
rlandya lot13:18
rfolcochkumar|rover, https://code.engineering.redhat.com/gerrit/#/c/208524/ is green, so the only jo that still fails is fs002 train/centos7, but not my patch's fault from what I see. So, when CI runs on https://review.opendev.org/733659, please help double checking everything is ok. I also did a recheck with the depends-on at https://review.rdoproject.org/r/#/c/28948/.13:18
weshaychkumar|rover, marios|ruck so you two have one more morning as ruck/rover13:19
rfolcochkumar|rover, thx for your help13:19
weshaysorry13:19
*** ratailor_ has quit IRC13:19
marios|ruckweshay: i'm out tomorrow13:20
weshayah13:20
weshayk13:20
weshayI'll cover13:20
weshayoh man.. I have my nick back13:20
weshay\0/13:20
weshayhoooray13:21
*** weshay is now known as weshay|ruck13:21
weshay|ruckmarios|ruck, don't tell anyone but I forgot my login to my znc bouncer13:21
weshay|rucklolz13:21
* weshay|ruck is dumb13:21
weshay|ruckchkumar|rover, marios|ruck cix canceled13:23
chkumar|roverweshay|ruck: ok13:24
weshay|ruckpain.. https://review.opendev.org/#/c/746533/13:24
weshay|ruckchkumar|rover, let's chat about tomorrow for a sec13:24
chkumar|roverweshay|ruck: sure13:25
marios|ruckweshay|ruck: ack13:25
marios|ruckchkumar|rover: we can sync after paul call if you like13:25
marios|rucksorry i mean weshay|ruck ^^13:25
marios|ruck late then for chkumar|rover13:25
weshay|ruckchkumar|rover, https://meet.google.com/gnd-xirg-bvb?authuser=113:26
weshay|ruckmarios|ruck, I just need to chat w/ him re: UA stuff13:26
marios|ruckack weshay|ruck was wondering ok13:26
rlandywell container builds on 17 are completely broken13:26
ysandeeprlandy, with base os/ rhel packages?13:28
rlandyhaven't even looked as to why13:30
*** udesale_ has joined #oooq13:33
*** udesale has quit IRC13:36
*** TrevorV has joined #oooq13:46
rlandyweshay|ruck: https://review.opendev.org/#/c/745912/13:46
rlandyweshay|ruck: want to chat now?14:12
weshay|ruckrlandy, https://meet.google.com/bbs-bzse-vrj?authuser=114:13
marios|ruckchkumar|rover: rfolco: fyi there are two UniSprint31 labels on the retrospective board14:14
marios|ruckchkumar|rover: rfolco: i am adding both to my cards hope you don't miss them (out tomorrow adding cards now)14:14
rfolcomarios|ruck, if there is an empty one, please remove14:14
rfolcomarios|ruck, I mean no cards with that one yet14:14
marios|ruckrfolco: both empty afaics14:15
rfolcomarios|ruck, use one and I'll remove the other14:15
rfolcomarios|ruck, thx for letting me know14:15
marios|ruckrfolco: k feel free to remove one i added both of them14:15
rfolcomarios|ruck, k14:15
marios|ruckrfolco: chkumar|rover: fyi * https://trello.com/c/P1oVcDAd * https://trello.com/c/Hqm11FOI * https://trello.com/c/KC1VcKxY14:16
marios|ruckthanks14:16
chkumar|rovermarios|ruck: thanks!14:23
*** saneax is now known as saneax_14:26
ysandeeprlandy, with success on vanila standalone.. from tommorow I will start attempting fs001/component. see you tomorrow o/14:27
rlandyysandeep: ack - great work today!14:28
ysandeeprlandy++ :) bye o/14:28
*** ysandeep is now known as ysandeep|away14:29
chkumar|rovermarios|ruck: one more down https://review.opendev.org/#/c/746894/2 rocky issue fixed14:32
marios|ruckchkumar|rover: nice14:32
chkumar|roverit will clear stein queens and rocky all14:32
rlandyweshay|ruck: https://code.engineering.redhat.com/gerrit/#/c/209162/1/config/release/promotion-testing-hash-rhos-16.2.yml14:40
weshay|ruckysandeep|away, easy one https://code.engineering.redhat.com/gerrit/#/c/209162/114:41
*** sshnaidm|afk is now known as sshnaidm14:42
rlandyweshay|ruck: https://code.engineering.redhat.com/gerrit/#/c/19588414:44
marios|ruckweshay|ruck: chkumar|rover: so do you want to sync re ruck|rover. any ideas who will be next?15:07
*** skramaja has quit IRC15:10
*** ykarel is now known as ykarel|away15:12
chkumar|rovermarios|ruck: all ok from myside15:12
* chkumar|rover logs out now15:14
chkumar|rovermarios|ruck: once again Happy Holidays :-)15:14
chkumar|roversee ya15:14
*** chkumar|rover is now known as raukadah15:14
marios|rucko/ raukadah15:14
marios|ruckraukadah: thanks for your support sprint3 115:14
raukadahmarios|ruck: thank to you too for the same :-)15:15
weshay|ruckmarios|ruck, meh.. no not sure15:16
marios|ruckweshay|ruck: not sure?15:16
marios|ruckoh who will be next15:17
marios|ruckweshay|ruck: just go in order15:17
weshay|ruckmarios|ruck, just make sure chandan has the info15:17
marios|ruckweshay|ruck: ack yeah me and raukadah are all synced up to time.now()15:18
*** ykarel|away has quit IRC15:26
*** apetrich has quit IRC15:46
* marios|ruck shutdown sequence15:46
marios|ruckweshay|ruck: ^ going in few15:46
*** marios|ruck is now known as marios|out15:56
rlandyweshay|ruck: ysandeep|away: https://code.engineering.redhat.com/gerrit/#/c/209162/ updated15:57
*** marios|out has quit IRC16:06
*** jpena is now known as jpena|off16:07
weshay|ruckrlandy, 0/16:12
rlandyweshay|ruck: hey16:12
weshay|ruckrlandy, meant to say.. +2 ^16:13
rlandyweshay|ruck: so I think we should just keep the beta repo option and add to it as needed while rhel 8.3 is in flux16:13
weshay|ruckk16:16
*** udesale_ has quit IRC16:18
weshay|rucksshnaidm, rlandy arxcruz sorry to kill your week next week w/ interviews16:22
weshay|ruckwe have two really competitive candidates16:22
rlandythat's a good problem16:22
weshay|ruckand we can repeat the same questions for both16:22
weshay|ruckthey have similiar backgrounds..16:23
arxcruzweshay|ruck: cool16:30
arxcruzI'm having fun with panda's code it might be a good idea to change a little bit the context, that code is driving me crazy16:30
*** dtantsur is now known as dtantsur|afk17:01
rlandyweshay|ruck: https://review.rdoproject.org/r/#/c/28715/17:42
weshay|ruckrlandy, https://review.opendev.org/#/q/topic:tcib/train+(status:open+OR+status:merged)17:45
rlandyweshay|ruck: https://code.engineering.redhat.com/gerrit/#/c/207802/17:58
rlandyweshay|ruck: https://code.engineering.redhat.com/gerrit/#/c/208575/17:59
*** apetrich has joined #oooq18:21
weshay|ruckrlandy, https://opendev.org/openstack/tripleo-common/src/commit/d28fb0be8ec1421b87741615597e9033e82ba803/tripleo_common/actions/ansible.py#L74-L8218:48
weshay|ruckrlandy, same in train and master18:49
rlandyweshay|ruck: yeah - no idea why18:49
weshay|ruckhttps://opendev.org/openstack/tripleo-common/src/branch/stable/train/tripleo_common/actions/ansible.py18:49
rlandy '/usr/share/ansible/roles:'18:49
weshay|ruckrlandy, we could update /etc/ansible/ansible.cfg18:50
rlandyhttp://git.app.eng.bos.redhat.com/git/openstack-tripleo-common.git/tree/tripleo_common/actions/ansible.py?h=rhos-16.2-trunk-patches#n7718:51
rlandywhy update - it's there18:51
weshay|rucknot the virtual env18:52
weshay|ruckyes.. have you copy everything18:52
rlandyweshay|ruck: on the upside, my patch didn't break 17 ... here is the periodic job ...18:52
weshay|ruckof course it's not logged https://sf.hosted.upshift.rdu2.redhat.com/logs/84/195884/25/check/periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-16.2/a4d3785/etc/18:52
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/84/195884/26/check/periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-17/d470db1/logs/build.log18:52
weshay|ruckah good18:52
rlandy^^ legit failure on tempest18:53
rlandymaybe we can fix that18:53
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/84/195884/26/check/periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-17/d470db1/logs/buildah-builds/container-builds/4790eedd-72aa-4946-9b21-f5a946e9df54/base/os/tempest/tempest-build.log18:53
rlandyweshay|ruck: missing deps18:54
rlandy  - cannot install the best candidate for the job18:54
rlandy  - nothing provides python3.6dist(pyasn1-modules) >= 0.2.1 needed by python3-google-auth-1:1.19.1-1.el8ost.noarch18:54
rlandy  - nothing provides python3.6dist(setuptools) >= 40.3.0 needed by python3-google-auth-1:1.19.1-1.el8ost.noarch18:54
rlandyhttp://download.eng.bos.redhat.com/rel-eng/repos/rhos-17.0-rhel-8-trunk-candidate/tagged-builds.lst18:55
rlandypython-google-auth-1.19.1-1.el8ost18:56
weshay|ruckhttps://review.opendev.org/74702218:56
weshay|ruckadd ansible config to container builds18:56
weshay|ruckrlandy, so.. the move to 8.3 caused those errors?18:56
weshay|ruckapparently18:56
rlandyweshay|ruck: no - 1718:56
rlandy:)18:57
weshay|ruckya.. but 17 moved to 8.3 as well right?18:57
rlandynot yet18:57
weshay|ruckI guess we don't know18:57
rlandy latest-RHOS_TRUNK-17-RHEL-8.2/18:57
rlandy8.218:58
rlandymissing dep18:58
rlandy python3-kubernetes                      noarch  10.0.1-1.el8                          delorean-master-testing       1.1 M19:03
rlandyweshay|ruck: let's promote 1719:03
rlandythe last one with passing containers19:04
weshay|ruckk19:04
* rlandy gets19:04
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?pipeline=openstack-periodic-rhos-17&job_name=periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-1719:05
rlandy08-1419:05
rlandyaggregate_hash": "62db6bcf3201a5aa1c8632f62975c6d1"19:05
rlandy[DIR]current-tripleo/2020-07-31 12:2219:06
rlandyperiodic-tripleo-ci-rhel-8-standalone-rhos-17openstack/tripleo-cimasteropenstack-periodic-rhos-17master37 mins 6 secs2020-08-12 00:44:58FAILURE19:13
rlandyperiodic-tripleo-ci-rhel-8-standalone-rhos-17openstack/tripleo-cimasteropenstack-periodic-rhos-17master1 hr 28 mins 21 secs2020-08-11 01:50:29SUCCESS19:13
rlandydid we stop installing pacemaker?19:13
weshay|ruckI hope not.. still on a call though19:13
weshay|ruckneed a few19:13
*** jfrancoa has quit IRC19:22
*** saneax_ has quit IRC19:32
weshay|ruckrlandy, rhos-17 promotion19:58
weshay|ruckneed me?19:58
weshay|ruckrlandy, ysandeep|away fyihttps://review.opendev.org/#/q/topic:modules+(status:open+OR+status:merged)20:07
rlandyweshay|ruck: yes - pls20:08
rlandyweshay|ruck: wrt https://review.opendev.org/#/c/747024/ going to try that with a temp downstream change20:09
weshay|ruckrlandy, k20:10
weshay|ruckrlandy, so tmux on internal?20:11
rlandyweshay|ruck: yep20:11
weshay|ruckok.. might as well chat while we do it20:11
weshay|ruckhttps://meet.google.com/aox-rusm-goe?authuser=120:11
weshay|ruckrlandy, https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/62fab4171d4f40119d1bbf8885a548b720:17
weshay|ruckrlandy, d18671cafbf31d038fe085ed42850fcc20:21
weshay|ruck  commit_hash: 21a722adac9c844d33a07323d111d0fe7a888ab320:22
weshay|ruck  distro_hash: 159f4d19b142dbedd11f895f00905de93b033f1220:22
*** rfolco has quit IRC21:05
*** jmasud has quit IRC21:10
*** jmasud has joined #oooq21:19
*** jmasud has quit IRC21:30
*** jmasud has joined #oooq21:36
*** tosky has quit IRC21:47
*** jmasud has quit IRC22:14
*** jmasud has joined #oooq22:16
*** rlandy is now known as rlandy|bbl22:28
*** TrevorV has quit IRC22:31
*** apetrich has quit IRC22:47
*** Goneri has quit IRC23:27
*** jmasud has quit IRC23:53
*** jmasud has joined #oooq23:59

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!