Monday, 2019-08-05

*** d0ugal has quit IRC00:05
*** d0ugal has joined #oooq00:19
*** aakarsh has joined #oooq00:48
*** aakarsh has quit IRC03:12
*** udesale has joined #oooq04:07
*** ykarel has joined #oooq04:23
*** ratailor has joined #oooq04:36
*** ratailor_ has joined #oooq04:41
*** ratailor has quit IRC04:42
*** ksambor has joined #oooq04:42
chandankumarykarel: Hello05:01
ykarelchandankumar, hello05:01
chandankumarykarel: can you pass me the logs where RHEL8 container build exit status is not working as expected05:01
chandankumar?05:01
ykarelchandankumar, it's not just rhel8, it's affected on all distros05:02
*** ratailor__ has joined #oooq05:02
ykarelchandankumar, see my comment in https://review.opendev.org/#/c/674282/05:03
ykarelfrom that it would be more clear05:03
*** ratailor_ has quit IRC05:04
*** surpatil has joined #oooq05:18
*** skramaja has joined #oooq05:21
*** ratailor has joined #oooq05:25
*** ratailor__ has quit IRC05:25
*** ratailor_ has joined #oooq05:34
chandankumarykarel: aye05:36
*** ratailor has quit IRC05:36
*** bhagyashris has joined #oooq05:37
*** marios|rover has joined #oooq05:37
*** marios has joined #oooq05:37
*** jaosorior has joined #oooq05:41
*** soniya29 has joined #oooq05:42
*** jfrancoa has joined #oooq05:53
*** jtomasek has joined #oooq06:34
*** apetrich has joined #oooq06:48
*** kopecmartin|pto is now known as kopecmartin06:51
*** bogdando has joined #oooq06:58
*** amoralej|off is now known as amoralej07:11
*** tesseract has joined #oooq07:17
*** tosky has joined #oooq07:19
*** gkadam has joined #oooq07:26
chandankumarmarios|rover: Good morning07:29
chandankumarmarios|rover: can we merge this https://review.opendev.org/#/c/674258/ to unblock promotion job07:29
marios|roverchandankumar: sure will check it in a sec07:32
marios|roverchandankumar: we don't have a bug?07:32
marios|roverchandankumar: is this related to the ./logfile_11Jul2019.log:149:[centos@reprozuul ~]$ rsync -auvz ironic-python-agent.tar -e ssh centos@38.145.34.141:/var/www/rcm-guest/rhel8-imgs/07:33
marios|roverchandankumar: oops i meant that failed HTTPError: 504 http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario003-standalone-master/929a76f/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz07:33
chandankumarmarios|rover: nope, it is related to https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-master/11c7794/logs/undercloud/var/log/containers/stdouts/keystone_db_sync.log.txt.gz07:35
chandankumarmarios|rover: ykarel and /me seen on thursday07:36
marios|roverchandankumar: ok thanks but that is that in criteria (ah i'm probably checking centos not rhel8 sec)07:37
marioschandankumar: oh, we don't have rhel8 criteria yet?07:38
marioschandankumar: https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/dlrnapi_promoter/config07:38
marioschandankumar: so how is it promotion blocker07:38
*** ccamacho has joined #oooq07:49
*** ykarel is now known as ykarel|lunch07:59
*** brault has joined #oooq08:03
* arxcruz on the doctor, brb08:17
*** arxcruz is now known as arxcruz|grb08:18
*** arxcruz|grb is now known as arxcruz|brb08:18
*** tesseract-RH has joined #oooq08:22
*** tesseract has quit IRC08:22
*** tesseract-RH has quit IRC08:24
*** tesseract has joined #oooq08:24
zbrmorning guys! where can I help?08:28
*** sshnaidm|afk is now known as sshnaidm08:37
*** chem has joined #oooq08:44
*** jaosorior has quit IRC08:51
*** ykarel|lunch is now known as ykarel09:08
sshnaidmchandankumar, hi09:20
sshnaidmchandankumar, can you point me please to list of packages that are included in rhe8 image in rdo? All dib elements I suppose09:21
mariosykarel: chandankumar do you folks know if we have a bug for that "Error running ['podman', 'run', '--name', 'nova_wait_for_compute_service',  don't see anything on the etherpad but its a promotion blocker found it twice so far on this morning periodics09:27
mariosykarel: chandankumar just checking will file one in a sec will see if i can find more quickly09:28
ykarelmarios, i think ^^ is same that i shared with you on Thursday09:28
mariosykarel: right09:28
ykareland a bug is there09:28
mariosykarel: its already in https://etherpad.openstack.org/p/ruckroversprint1309:28
mariosykarel: but we didn't file a bug09:28
ykarelmarios, i also proposed a fix in placement09:28
mariosykarel: oh i see it was added above the crossed out thing so i missed it09:28
mariosykarel: line 4409:28
ykarelmarios, can u share log, i can check if the issue is same for which i proposed a fix and bug is there09:29
mariosykarel: http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/95d47c4/logs/undercloud/home/zuul/install-undercloud.log.txt.gz09:29
marioshttp://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/e656039/logs/undercloud/home/zuul/install-undercloud.log.txt.gz09:29
mariosykarel: it fails on undercloud install like Error running ['podman', 'run', '--name', 'nova_wait_for_compute_service09:29
mariosykarel: looks like it from the description a@ https://bugs.launchpad.net/tripleo/+bug/183863309:30
openstackLaunchpad bug 1838633 in tripleo "[nova placement] no version was discoverable" [Critical,Triaged]09:30
ykarelmarios, yes it's same and bug is ^^ https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/95d47c4/logs/undercloud/var/log/containers/nova/nova-scheduler.log.txt.gz#_2019-08-05_02_09_33_99709:30
mariosykarel: thanks i see it already has promotion-blocker09:30
mariosykarel: damn.. so we need promotion for the fix09:33
mariosykarel: [1] https://review.opendev.org/#/c/656023/09:33
ykarelmarios, ?09:33
mariosykarel: but this blocks promotion :/09:34
mariosykarel: its in openstack/nova09:34
ykarelwe just need placement patch to merge09:34
mariosykarel: where is the fix? the thing your pointing to in comment 1 is the cause right? (I thiought it was the fix)09:34
ykarelmarios, just commented the fix09:35
ykarellink of the fix09:35
mariosykarel: thx09:35
mariosykarel: so same thing though. like we need new version of openstack/placement with that fix. but we can't have it until we get promotion right?09:36
ykarelmarios, we just need the patch to be merged, and package to get build by DLRN, so periodic jobs which run after fix is available in consistent repo will pass09:38
ykarelperiodic jobs will pass09:39
marios|roverykarel: ok great. i thought for anything that isn't tripleo, like nova/placement etc etc, we are only getting a new version once we tag a new current-tripleo i.e. with promotion09:39
ykarelmarios|rover, ^^ is needed only when upstream check/gate jobs are affected09:40
marios|roverykarel: ah so periodic jobs consume built packages from consistent?09:40
marios|roverof all the things?09:41
ykarelmarios|rover, yes09:41
marios|roverykarel: thanks09:41
marios|roverykarel: well, not so much about 'periodic jobs' more like 'rdo jobs'09:41
chandankumarsshnaidm: please have a look at this https://review.opendev.org/#/c/674433/7/roles/overcloud-ssl/library/tls_tht.py@11809:43
sshnaidmchandankumar, actually it failed with rb, tried it in previous patches09:45
sshnaidmchandankumar, work with "r" only, idk why09:45
chandankumarsshnaidm: then we donot need to define 'r' there then09:46
chandankumarit will be taken by default with open method09:46
sshnaidmchandankumar, yeah, but doesn't hurt09:46
chandankumarsshnaidm: :-)09:47
marios|roversshnaidm:  wdyt can we merge it? https://review.rdoproject.org/r/#/c/21640/09:47
marios|roversshnaidm: i think the voting info is useful especially on the promotion jobs09:47
marios|roversshnaidm: snapshot in commit message09:47
marios|roversshnaidm: appreciate when you next have some time thanks09:48
marios|rover++ anyone else interested in the grafana cockpit please thanks https://review.rdoproject.org/r/#/c/21640/ Adds voting info on jobs and descriptions to metrics09:48
sshnaidmmarios|rover, sure, +w09:50
marios|roversshnaidm: thanks :D hope i don't break it ! worst case revert09:50
marios|roversshnaidm: i wasn't sure about the notification.json cos i didn't make changes there09:51
marios|roversshnaidm: it seems to flip two things between 1 and 209:51
marios|roversshnaidm: but export-grafana said i did09:51
sshnaidmmarios|rover, yeah, there are casual changes there, related to order I think, I just don't commit them09:51
marios|roversshnaidm: ack do you want me to update09:51
marios|roversshnaidm: i can remove them before merge then09:51
sshnaidmmarios|rover, no, that's fine09:51
marios|roversshnaidm: ack09:51
marios|roversshnaidm: thanks09:51
marios|roversshnaidm: do we need to restart it to see that09:52
marios|roversshnaidm: or when/how does it happen09:52
sshnaidmmarios|rover, it will be done automatically09:52
marios|roversshnaidm: ok thanks man i will stop harrassing you now09:52
sshnaidmmarios|rover, I think we have ansible pull there every 5-10 mins09:52
marios|roversshnaidm: you can tell me more about how that works on tomorrow community call? :D09:53
marios|roversshnaidm: nice09:53
sshnaidmmarios|rover, np :)09:53
*** skramaja has quit IRC09:58
*** jaosorior has joined #oooq10:05
*** bhagyashris has quit IRC10:25
*** arxcruz|brb is now known as arxcruz10:52
*** skramaja has joined #oooq10:55
zbrsshnaidm: what do you think about https://review.opendev.org/#/c/673842/ ?11:10
*** brault has quit IRC11:11
zbrregarding ansible-pull, i never liked it, so if someone wants to remove it, I will support it.11:12
sshnaidmzbr, why someone would remove it??11:14
zbrsshnaidm: i see the use of recurring ansible-pull as as design anti-pattern, where you rely on the managed host to reconfigure itself when configuration management is changed. this does not scale well. also it does not work well with flexible environments like: containers or hosts that have a limited lifespan. That is why i support pushing configuration changes from a centralized place (even CI), as this allows me to have execution history in a single11:20
zbr place. on pull based systems: if the managed host disapears, you lose execution history.11:20
zbri guess pull comes from people used to puppet/chef as that was their default mode of working.11:20
zbrpush has other benefits, as it allows you to slowly perform migrations, applying changes to only some hosts. on pull systems you do not have the same level of control.11:21
zbranyway, thats technical debate without clear winner, no need to go deeper now, we have plenty of stuff to do.11:22
*** jaosorior has quit IRC11:23
sshnaidmI'd like to use AWX instead, but it requires some effort to install and configure11:32
*** gkadam has quit IRC11:32
*** gkadam has joined #oooq11:49
zbrsshnaidm: create a card for it, sounds like a good idea. the only question would why not tower. anyway, any of them would be of good use for us.12:03
sshnaidmzbr, awx is a upstream tower actually12:05
sshnaidmzbr, not sure we can use tower.. but question to weshay12:06
*** udesale has quit IRC12:06
*** udesale has joined #oooq12:07
weshayzbr sshnaidm what's the goal?12:07
zbrto have an interface for managing execution of configuration management tasks (periodic, or on demand): example configuring the tools server, cleaning the stack leftovers,....12:08
zbrlike an interactive web-cron.12:09
*** jaosorior has joined #oooq12:10
weshayzbr ya.. tower could be considered for our inra I think.. but it would need more discussion.  I don't see a lot of time/space for such topics in the near future.. perhaps closer to the end of the cycle12:11
zbrweshay: that was the purpose: for infra management. infact I would deploy it on the same server we use now. if is tower or awx, that's only a detail (i do not care much).12:13
zbrin fact was planning to spin out one for my smart-home, now I only have playbooks I run manually.12:14
*** rfolco has joined #oooq12:16
*** rfolco is now known as rfolco|ruck12:16
weshayzbr it's a good idea to consider for sure12:18
weshaybut we don't have much room for it12:18
weshayzbr btw.. are you using rdo-ci-fx2-02-s6.v101.rdoci.lab.eng.rdu2.redhat.com12:18
zbrweshay: no idea what is that.12:20
sshnaidmweshay, but is it possible to get "free" tower for us? or we can use upstream awx only12:20
weshaysshnaidm zbr ya.. I'm not worried about getting tower or awx.. more about the time12:21
sshnaidmweshay, we can start from first steps, like set up12:22
zbri got the impression that was the only problem: cost/benefit in terms of dev-time. how much time we need to spend setting/running it and how much it saves us.12:23
zbrideally PNT should be able to run an instance for us, so we avoid the maintenance part.12:23
sshnaidminfra kills us..12:25
sshnaidmevery patch is rechecked multiple times to get through..12:26
*** rlandy has joined #oooq12:26
sshnaidmzbr, yeah, would be nice to have it maintained by someone12:27
sshnaidmbut not sure pnt even know about it12:27
*** ratailor_ has quit IRC12:33
rfolco|ruckweshay, fixing headset issues12:36
rfolco|ruck:(12:36
*** ccamacho has quit IRC12:43
*** soniya29 has quit IRC12:45
rlandysshnaidm: sorry - got involved with image upload on friday - didn't work on ovb12:46
sshnaidmrlandy, that's fine, introspection issue was casual, didn't repeat12:47
rlandygood to know12:47
sshnaidmrlandy, yeah, at least introspection works..12:47
rlandysshnaidm: what do you think about https://review.opendev.org/#/c/673891/?12:48
rlandyit's included in the test run patch12:48
rlandymay impact other ovb12:48
rlandysee the order in which the files are passed12:48
rlandyor we would have to switch case which image files are used in the ovb settings12:49
sshnaidmrlandy, yes, environment is overriding release specifics, it was intentional12:51
rlandysshnaidm: ok - so then I need to change the patch not to exclude those settings but to switch case them12:52
sshnaidmrlandy, we need to fix paths to include rhel8 images too, like http://images.rdoproject.org/{{ release }}/rdo_trunk/{{ distro }}12:52
sshnaidmrlandy, what is current path of promoted rhel8 images on the server?12:53
rlandysshnaidm: the paths in the release files are correct12:53
rlandyI think12:53
rlandywe have trying to get image upload to work12:53
sshnaidmrlandy, I mean not the temporary in release file, but uploaded from jobs12:54
sshnaidmrlandy, does it work now?12:55
rlandysshnaidm: we are missing the correct secret - will discuss at meeting - but we are close12:55
sshnaidmrlandy, ok12:57
rfolco|ruckrlandy, ping scrum13:01
rfolco|rucksorry you there13:01
rfolco|ruck:)13:01
chandankumarsshnaidm: arxcruz rdo to login registry https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build-push/976db0b/job-output.txt.gz#_2019-08-05_12_45_43_31781113:05
chandankumarbroke again13:05
chandankumarlogin was successful but failed with false condition13:05
*** skramaja has quit IRC13:15
*** ykarel is now known as ykarel|afk13:22
ykarel|afk^^ sems happening after switch to docker instead of docker_login13:24
chandankumarykarel|afk: yes, I think we need to remove become: false part from the playbook13:25
ykarel|afkhmm possibly13:25
chandankumarykarel|afk: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-ci-periodic-base/containers-push-registry-login.yaml#L3213:25
chandankumararxcruz: ^^13:26
arxcruzchandankumar:?13:26
ykarel|afkso it depends13:26
arxcruzchandankumar: is that the reason is failing ?13:26
chandankumararxcruz: might be if you see the task, login was successful13:27
ykarel|afkiirc both true|false were added earlier so token is saved for both ansible user and root13:27
arxcruzso, remove the falce ?13:27
arxcruzfalse*13:27
chandankumarand then it failed13:27
ykarel|afkso further commands(which needs token) work with ansible user and root user13:27
chandankumarneed to check the push happens with sudo or as a normal user13:28
ykarel|afkbut if you are sure if ^^ is not the case, good to go without false13:28
arxcruzok13:28
*** amoralej is now known as amoralej|lunch13:29
arxcruzchandankumar: ykarel|afk https://review.rdoproject.org/r/#/c/21714/13:30
*** vinaykns has joined #oooq13:30
chandankumarykarel|afk: do we want to run in check with disable push once it gets merged?13:31
ykarel|afkchandankumar, yes would be better as next run is in couple of hours13:32
chandankumararxcruz: ^^13:33
chandankumararxcruz: please add Co-authored-by:  in the commit message of fs01 patch13:38
*** jbadiapa has joined #oooq13:39
arxcruzchandankumar:ok13:39
chandankumarrlandy: https://review.opendev.org/#/c/673517/ and https://review.opendev.org/#/c/673757/13:41
*** surpatil has quit IRC13:43
*** jbadiapa has quit IRC13:44
*** aakarsh has joined #oooq13:47
*** aakarsh|2 has joined #oooq13:49
ykarel|afkchandankumar, i think docker ones pushes without root13:50
ykarel|afkso good to check that part13:50
*** aakarsh has quit IRC13:52
*** ykarel|afk is now known as ykarel|away13:52
rlandyweshay: https://review.rdoproject.org/r/#/c/21715/13:52
rlandyUse the same secret on rcm server as on rdo image server13:52
weshayrlandy reminder interview tomorrow at 8am ur time13:56
weshaymarios|rover do you have interest in one last interview?13:56
rlandyweshay: ack13:57
marios|roverweshay: sure13:57
chandankumarweshay: arxcruz https://twitter.com/odyssey4me/status/1158346012509032448 -> odyssey4me joined RH14:02
arxcruzchandankumar: old news :P14:02
chandankumarrfolco|ruck: in RHEL-7 jobs, how containers are pushed in RDO registry ? using docker with sudo or without sudo?14:03
rfolco|ruckchandankumar, hmm.. rhel7 we did not push I think... we just switched to rhel8 when the poc was done (container build job was working)14:05
chandankumarrfolco|ruck: so why we login to rdo registry in rhel-7 container build job then?14:06
chandankumaris it for fedora?14:06
rfolco|ruckchandankumar, what I observed is that old versions of podman require sudo...14:06
chandankumarrfolco|ruck: rhel-8 is fine14:06
chandankumarrfolco|ruck: it is about rhel-7 and fedora14:06
rlandychandankumar: wrt https://review.opendev.org/#/c/673517/ the rhel8 job is still failing here14:07
rfolco|ruckchandankumar, I think we don't need sudo for buildah/podman... maybe fedora used docker command, not buildah? need to refrash my mind on this14:08
chandankumarrlandy: as the container with that hash does not exists14:08
rfolco|ruckchandankumar, if I remember right, centos used --use-buildah and fedora didn't... might be wrong though14:09
chandankumarrfolco|ruck: refreshing https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build-push/976db0b/job-output.txt.gz#_2019-08-05_12_45_44_39411614:10
chandankumardocker login passed with become true as failed with false14:10
rfolco|ruckits because it uses docker_login module14:10
rfolco|ruckit requires sudo afaik14:11
rfolco|ruckchandankumar, ^14:11
chandankumarso we have a patch to just use docker with sudo, and then concern is docker used in pushing containers to rdo registry for fedora/centos using sudo?14:11
rfolco|ruckif not using docker_login, that is, podman/buildah, no need for become_true14:11
rfolco|ruck'become: true' I mean14:12
chandankumarrfolco|ruck: we removed docker_login today14:12
chandankumarswitched to docker14:12
rfolco|ruckdocker is docker14:12
rfolco|ruckneeds sudo afaik14:12
chandankumarykarel|away: let's merge it and test it https://review.rdoproject.org/r/#/c/21714/14:13
chandankumarrfolco|ruck: ok, then14:13
chandankumarwe are good to go14:13
*** rlandy is now known as rlandy|brb14:13
* chandankumar takes a look from home14:15
chandankumarrfolco|ruck: I might little late for pre planning meeting14:15
rfolco|ruckchandankumar, ack14:15
marios|roverweshay: er... tomorrow is a little short notice though? :)14:16
marios|roverweshay: lets discuss on the email better thanks14:18
*** rlandy|brb is now known as rlandy14:23
rlandyweshay: I think it worked!!! http://logs.rdoproject.org/83/21683/8/check/periodic-tripleo-rhel-8-buildimage-overcloud-full-master/e18b893/job-output.txt.gz14:24
rlandy2019-08-05 10:16:59.426694 | primary | tripleo-ci-testing -> 8cde536692a36cd094c337551e5e7053f861ea0e_d0e11ceb14:24
rlandydrwxrwxr-x. 2 centos centos  128 Aug  5 14:16 8cde536692a36cd094c337551e5e7053f861ea0e_d0e11ceb14:25
rlandyha14:25
rlandyneed to try run both jobs and see if it works together14:26
rlandyhmmm ...14:27
rlandyhow does it have ironic-python-agent.tar ?14:27
marios|roverweshay: can we skip our call and i'll schedule something14:30
rlandymarios|rover: ping re: image uploads14:33
marios|roverrlandy: arxcruz weshay sshnaidm sent you an email just now about a call in 1.5 hours please see inbox cos we don't have any other time14:33
marios|roverrlandy: o/14:33
marios|rover17:34 < rlandy> marios|rover: ping re: image uploads14:33
rlandymarios|rover: pls see http://logs.rdoproject.org/83/21683/8/check/periodic-tripleo-rhel-8-buildimage-overcloud-full-master/e18b893/job-output.txt.gz14:34
rlandylooks like we built and uploaded both ips and overcloud14:34
rlandyipa14:34
rlandyin eone job14:34
rlandyone14:34
rlandycorrection - somehow we uploaded the ipa image14:35
marios|roverrlandy: hmm in the build command, if you don't specify the image it builds whats in config let me point to what i  mean14:35
*** amoralej|lunch is now known as amoralej14:35
*** jaosorior has quit IRC14:36
marios|roverrlandy: no sorry, we didn't merge that14:37
rlandymarios|rover: did merge which piece?14:37
marios|roverrlandy: i was thinking of https://review.opendev.org/#/c/669845/4/roles/oooci-build-images/templates/build-images.sh.j214:37
marios|roverrlandy: i thought we would want to build all the images in one job at some point and that was the review14:38
marios|roverrlandy: so it isn't that anyway... /me looks at log again14:38
rlandymarios|rover: k - we have two jobs atm in periodic14:38
*** jaosorior has joined #oooq14:39
rlandyI had the images defined at one point14:39
rfolco|ruckzbr, can you please quick show me the pre-commit hook example ? link to a simple example is enough14:40
zbrsure. it would be my pleasure.14:41
zbr5min14:41
rfolco|ruckk14:41
rlandymarios|rover: k - I know why14:42
marios|roverrlandy: i see 2019-08-05 14:16:34.535 | Image file ./overcloud-full.qcow2 created... http://logs.rdoproject.org/83/21683/8/check/periodic-tripleo-rhel-8-buildimage-overcloud-full-master/e18b893/build.log14:42
marios|roverrlandy: what else is built there? ipa?14:42
rlandymarios|rover: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-ci-periodic-base/buildimage/post.yaml14:42
marios|roverrlandy: i don't see it14:42
rlandy^^ build them both14:43
marios|roverrlandy: ah the md5 pre thing didn't realise we were using that  hmm so maybe will have to add a when there14:43
rlandymarios|rover: weshay: why do we need two separate jobs?14:43
rlandywhy can;t we build both in one job14:43
marios|roverrlandy: "cos thats how it works upstream"14:44
rlandyI can fix the playbook with a var14:44
marios|roverrlandy: we could14:44
marios|roverrlandy: i mean it was explored lemme fetch taiga14:44
rfolco|ruckweshay, this is your request, ready for w+ https://review.opendev.org/#/c/673842/14:45
marios|roverrlandy: "proposal 2" @ reviews in https://tree.taiga.io/project/tripleo-ci-board/task/115114:45
zbrrfolco|ruck: now ok? https://bluejeans.com/265541792814:45
rfolco|ruckzbr, joining14:46
rlandymarios|rover: ok - so I just need to add a switch to https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-ci-periodic-base/buildimage/post.yaml#L914:48
weshayrfolco|ruck add a patch on top re: code comments.. and I'll +214:48
rlandyand https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-ci-periodic-base/buildimage/post.yaml#L1714:48
*** brault has joined #oooq14:50
*** brault has quit IRC14:55
rfolco|ruckweshay, code comments ? I don't see any requests in the patch for it14:55
rfolco|ruckweshay, https://review.opendev.org/#/c/673842/ - which code comment you need ?14:55
weshayrfolco|ruck sorry.. there now14:56
rfolco|ruckweshay, ack14:56
weshayrfolco|ruck chandankumar preplanning15:01
rfolco|ruckweshay, yes goin15:02
weshayrfolco|ruck chandankumar https://bluejeans.com/587845809715:02
chandankumarweshay: joined15:04
marios|roverweshay: arxcruz sshnaidm can you make the call in 22 mins?15:06
marios|roverweshay: i canceled our 1-1 for that ^ but you sound busy ^15:06
sshnaidmmarios|rover, yes15:06
marios|roversshnaidm: k thanks15:06
marios|roverweshay: rlandy sshnaidm arxcruz lets sync? in a minute? blue on the invite and links to docs15:27
sshnaidmmarios|rover, are you there?15:32
marios|roversshnaidm: yeah me and rlandy are there15:32
marios|roversshnaidm: check the calendar invite for info15:32
sshnaidmmarios|rover, what's link?15:32
marios|roversshnaidm: see pvt15:33
marios|roverarxcruz: are you able to join?15:33
zbrgabriele role is ready for review at https://review.rdoproject.org/r/#/c/21593/ -- who can help?15:55
*** amoralej is now known as amoralej|off15:56
zbr^ that should not a hard review as it cannot break anything by itself. rlandy rfolco|ruck16:00
*** tesseract has quit IRC16:04
*** marios|rover has quit IRC16:08
*** marios has quit IRC16:08
*** bogdando has quit IRC16:11
ykarel|awaychandankumar, /me was out, just back16:13
ykarel|awayi still doubt, but ok to merge as it's already broken.16:14
weshayjdennis ping.. can I take over your patch for a bit.. to consolidate the virt and baremetal workflow?16:14
weshayfor ipa16:14
ykarel|awaychandankumar, but need to revert both in case it didn't work16:14
chandankumarykarel|away: do you have link where container push happens in fedora 28 and centos 7?16:15
ykarel|awaychandankumar, it's part of kolla-build16:15
ykarel|awayin non -buildah jobs16:15
chandankumarykarel|away: let's merge it today and if it breaks, we can revert it in the morning?16:15
ykarel|awaychandankumar, +W can you push job in check16:16
ykarel|awaychandankumar, i will try to revert before next run16:16
ykarel|awayas current blocker fixes are up16:16
chandankumarykarel|away: do I need to use depends on to use the current periodic hash?16:16
ykarel|awaychandankumar, should not be needed for container build job16:17
ykarel|awayas release file is not used there16:17
chandankumarack16:17
chandankumarrfolco|ruck: https://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-master-containers-build-push/1b59fa8/logs/build.log.txt.gz#_2019-08-05_09_07_4016:22
chandankumarrfolco|ruck: it got timedout16:22
chandankumarfew containers have taken too much time to built please have a look16:22
jdennisweshay: sure, go for it. BTW, I saw your email this morning (thanks) and I had just started to look at how quickstart.sh and toci_quickstart.sh differ. I don't see in toci_quckstart.sh where PLAYBOOKS gets initialized, where does it?16:36
weshayin the zuul config16:42
weshaywhere I injected the third_party_role playbook16:43
* weshay looks at the log for the rest16:43
weshayhttps://github.com/openstack/tripleo-ci/blob/master/roles/run-test/templates/toci_quickstart.sh.j2#L9316:47
*** dsneddon has joined #oooq16:47
weshayhttp://logs.rdoproject.org/72/21672/1/check/tripleo-ci-centos-7-ovb-1ctlr_2comp_1supp-featureset039/99f58d1/zuul-info/inventory.yaml16:48
weshayhttps://github.com/openstack/tripleo-ci/blob/master/roles/run-test/templates/toci_quickstart.sh.j2#L12816:48
rfolco|ruckzbr, awesome ! I like the fs include solution :)16:58
zbrrfolco|ruck: i am glad you like it. once you realise how is it is, you can imagine other use cases.16:59
zbrat some point i wanted to add a zuul-linter but never had time to work on it.16:59
*** kopecmartin is now known as kopecmartin|off16:59
rfolco|ruckzbr, ;)16:59
zbrrfolco|ruck: as you can imagine in few minutes we can add a check that rejects commits without taiga links.17:00
zbrfor me the best advantage: avoiding the need to document the process for making changes to that file.17:00
rfolco|ruckzbr, right, need to socialize and get through open decision workflow for this kind of changes17:01
chandankumarzbr: how you are automating it? can we add something if code is contributed by two people, hightlight them to include co-authors17:02
*** jfrancoa has quit IRC17:03
*** udesale has quit IRC17:07
rfolco|ruckchandankumar, I did not ignore you, still looking at container build timeout17:13
rfolco|ruckchandankumar, looking for some additional debugging traps to add there17:13
*** Vorrtex has joined #oooq17:21
rlandyweshay: we have a problem ...17:25
rlandy primary |   "url": "http://38.145.34.141/rcm-guest/images/rhel8/master/rdo_trunk/tripleo-ci-testing/rhel-8.0-guest-rhui.qcow2"17:25
rlandywhat uploads that?17:25
rlandyit's in /var/www/rcm-guest/images/redhat8/master/rdo_trunk/0000000000000000000000000000000000000000_0000000017:25
ykarel|awayarxcruz, chandankumar, unauthorized: authentication required in container build job17:25
weshayrlandy we uploaded that by hand..17:25
weshayit shouldn't be under a hash17:26
rlandybut not in the latest hash uploaded17:26
weshayit's used by image build as the rhel seed image17:26
rlandywhere should it be?17:26
weshayrlandy in it's own dir17:26
chandankumarykarel|away: if we run the job somewhere, I donot think secrets will be shared17:26
ykarel|awaychandankumar, arxcruz it would be good to revert and reiterate, as current master fixes are in place,17:26
chandankumar?17:26
rlandyby what name?17:26
weshaysome.. like..  rhel_base17:26
weshayor redhat_base17:26
weshaydon't care17:26
ykarel|awaychandankumar, elsewhere?17:26
weshaywhy someone would put that under a hash is beyond me17:27
rlandyvar/www/rcm-guest/images/redhat8/master/rdo_trunk/rhel-8-base?17:27
rlandyweshay: ^^ ok?17:27
weshayrlandy no17:27
chandankumarykarel|away: we are running the job from rdo-jobs but the promotion job is defined in config repo17:27
weshaywhy would it be under master?17:27
ykarel|awaychandankumar, ^^ doesn't matter17:28
weshayshould be.. images/rhel_base/rhel-8.0-base17:28
chandankumarykarel|away: let's revert it17:28
rlandyk17:28
ykarel|awaychandankumar, ack preparing patch for reverting both together17:28
chandankumarykarel|away: we might switch to podman for pushing the stuff?17:29
chandankumaras RHEL-8 is working fine17:29
ykarel|awaychandankumar, yes as a seperate task can be done17:30
ykarel|awaylast time when it was enabled there were some issues17:30
ykarel|awaybut good to reiterate and use same tooling17:30
chandankumarykarel|away: with both centos/fedora both?17:31
chandankumarykarel|away: we have staging server coming up, we might test the stuff there in next sprint17:31
ykarel|awaychandankumar, i saw recently fedora jobs are removed17:31
ykarel|awayso not sure about fedora what's the plan for tripleo-ci17:31
ykarel|awayi guess it will go soon17:31
chandankumarrfolco|ruck: weshay ^^17:31
weshayykarel|away fedora is getting removed17:32
weshayno longer valuable now that we have rhel817:32
ykarel|awayweshay, yup i guessed that , Thanks for confirming17:33
weshaynp17:33
*** ykarel|away has quit IRC17:56
sshnaidmrlandy, did something change with images in http://38.145.34.141 ?18:10
sshnaidmrlandy, there is new error in ovb: Failed to find handler for "/home/zuul/images-cache/1276481102f218c981e0324180bafd9f.tar". Make sure the required command to extract the file is installed. Command "/usr/bin/unzip" could not handle archive. Command "/usr/bin/gtar" could not handle archive. Command "/usr/bin/gtar" found no files in archive.18:12
rlandysshnaidm: ugh sorry18:15
rlandysshnaidm: I moved the images18:16
sshnaidm;(18:16
rlandysshnaidm: <weshay> rlandy in it's own dir18:16
rlandywe manually pushed the images18:17
rlandythe rhel images18:17
rlandywhich ones are you taking about?18:17
rlandythe rhui gust image moved18:18
rlandyguest18:18
sshnaidmrlandy, I think it's overcloud and ironic images18:18
rlandysshnaidm: what link are you using?18:18
*** dsneddon has quit IRC18:19
rlandyhttp://logs.rdoproject.org/83/21683/10/check/periodic-tripleo-rhel-8-buildimage-ironic-python-agent-master/952434e/job-output.txt.gz18:19
rlandyjust completed18:19
sshnaidmrlandy, usual link from release file18:19
rlandyand this http://logs.rdoproject.org/83/21683/10/check/periodic-tripleo-rhel-8-buildimage-overcloud-full-master/4df8d1b/job-output.txt.gz18:19
rlandyif it's tripleo-ci-testing18:19
sshnaidmrlandy, I don't use these images18:19
weshaycopy them over to what ever you need man18:20
weshaysymlink rather18:20
sshnaidmrlandy, https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/RedHat-8/master.yml#L31-L3218:20
rlandyI never deleted any images18:20
rlandysshnaidm: so those images are being built now by the jobs18:20
rlandythe tripleo-ci-testing link goes to the images uploaded18:21
sshnaidmrlandy, well, currently I don't think I can use them18:21
rlandyif they are failing ,we have a problem18:21
rlandywe can relink to the old images18:21
sshnaidmrlandy, do we have containers for this tripleo-ci-testing hash?18:21
rlandybut that's a problem with the jobs then18:21
rlandyprobably not18:22
rlandylet me check18:22
sshnaidmrlandy, than I can't use it18:22
sshnaidmwell, something definitely changed with old images, because nothing changed in job18:22
rlandysshnaidm: what changed is where the tripleo-ci-testing link is pointing18:23
rlandywe can change the link18:24
rlandybut eventually we need to move on from using the hand uploaded images18:24
sshnaidmrlandy, I'm testing in this patch https://review.rdoproject.org/r/#/c/21649/18:24
sshnaidmrlandy, I don't use this link, but specific dlrn hash that I have containers for it18:24
rlandysshnaidm: ok - I was testing the upload jobs18:25
rlandywill relink back to old hash18:25
sshnaidmrlandy, I mean I don't use tripleo-ci-testing18:26
rlandyif your jobs is periodic18:26
sshnaidmrlandy, https://review.rdoproject.org/r/#/c/21649/21/zuul.d/projects.yaml18:26
sshnaidmfeatureset_override:18:27
sshnaidm                dlrn_hash_tag: c17d2177ce6a05e4290224d99126a0f39c601b4e_23df877318:27
sshnaidmnevermind, I'll look tomorrow18:27
sshnaidmrlandy, what is path for new images?18:27
rlandysshnaidm: I did two things18:27
rlandyadded a new path for rhui image18:27
rlandyretagged tripleo-ci-testing18:28
rlandythat is all18:28
rlandyI did not move the old rhui image18:28
rlandy2019-08-05 12:10:54.989424 | primary | + [[ redhat == \f\e\d\o\r\a ]]18:29
rlandy2019-08-05 12:10:54.989436 | primary | + HASHES_URL=https://trunk.rdoproject.org/redhat8-master/tripleo-ci-testing/commit.yaml18:29
rlandy2019-08-05 12:10:54.989482 | primary | + curl -sLo /home/zuul/workspace/commit.yaml https://trunk.rdoproject.org/redhat8-master/tripleo-ci-testing/commit.yaml18:29
rlandy2019-08-05 12:10:55.252710 | primary | ++ shyaml get-value commits.0.commit_hash18:29
rlandy2019-08-05 12:10:55.316480 | primary | + COMMIT_HASH=8cde536692a36cd094c337551e5e7053f861ea0e18:29
rlandy2019-08-05 12:10:55.316903 | primary | ++ shyaml get-value commits.0.distro_hash18:29
rlandy2019-08-05 12:10:55.367584 | primary | + DISTRO_HASH=d0e11ceb3684ffd1daa672e98c5a82697f73073318:29
rlandy2019-08-05 12:10:55.367633 | primary | + FULL_HASH=8cde536692a36cd094c337551e5e7053f861ea0e_d0e11ceb18:29
rlandy2019-08-05 12:10:55.367642 | primary | + cat18:29
rlandy^^ picking up the new hash18:29
*** tosky has quit IRC18:32
rlandysshnaidm: I linked it back to the old hash18:33
rlandyweshay: I think the image uploade jobs are working now ...18:34
rlandyhttps://review.rdoproject.org/r/#/c/21683/18:34
weshayNICE!!!!18:34
rlandywill see with next periodic18:34
* weshay looks18:35
rlandythe the retagging will work with containers18:35
rlandyweshay: pls check me18:35
weshayrlandy 2019-08-05 13:57:58.479260 | primary | Identity added: /tmp/ansible.ydvshuox (/tmp/ansible.ydvshuox)18:35
weshaythat's good18:35
weshayso it was the key?18:35
weshayswitching to the centos-7 image server key worked?18:36
rlandyweshay: used same uploader key18:36
rlandyworked18:36
weshayya.. cool...18:36
rlandyI just don;t know if the image build is good now18:36
rlandywill have to see when periodic runs18:36
rlandyand there are matching images18:36
rlandyI retagged back to old hash to unblock sshnaidm18:37
weshayrlandy on the file server18:38
weshaylooks good to me18:38
rlandyweshay: k - let's see what happens on next rub18:38
rlandyrun18:38
rlandyhopefully I didn't hose anything else18:38
weshayrlandy++18:42
rlandyweshay: let's wait on all the pluses until we see what else I broke18:44
*** dsneddon has joined #oooq18:49
rlandyweshay: ok if I merge https://review.rdoproject.org/r/#/c/21579/ - will add buildimage jobs to a lot of repos19:07
weshayis the job marked non-voting19:08
*** Goneri has joined #oooq19:08
* rlandy checks19:10
rlandyweshay: yes - https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/buildimage.yaml19:11
weshayrlandy ack to merge then19:12
rlandyk19:13
*** Goneri has quit IRC19:42
*** Goneri has joined #oooq19:43
rlandyzbr: what's still left to merge on your side?20:00
rlandyhttps://review.rdoproject.org/r/#/c/21593/?20:01
zbrrlandy: that is the one. i can sync with panda tomorrow morning.20:02
rlandyzbr: looking at that review20:03
*** Goneri has quit IRC20:07
*** Goneri has joined #oooq20:13
weshayrlandy need anything?20:14
rlandyweshay: maybe - checking registry from current promotion run20:15
weshayzbr qq since you are here.. I was wonder how much certainty that molecule playbook is providing w/o actually pulling a container or something like that20:15
rlandyweshay: I think we're ok now20:16
rlandyhash has containers and images20:16
zbrweshay: quite good, afaik, docker login  command does not succeed if credentials are not valid.20:16
weshay2019-08-05 14:47:52.360096 | primary | + rsync --verbose --archive --delay-updates --relative 305e280cab53c284da05dfbb3e7485db696d541c_ee467b40 centos@38.145.34.141:/var/www/rcm-guest/images/redhat8/master/rdo_trunk20:16
rlandy:305e280cab53c284da05dfbb3e7485db696d541c_ee467b4020:16
weshaylatest looks good20:16
rlandyweshay: ack20:17
rlandyI need to update     https://review.opendev.org/#/c/673891/ DNM: Remove image path definitions from OVB settings file for oVB20:17
weshayhttps://trunk.rdoproject.org/rhel8-master/tripleo-ci-testing/delorean.repo20:17
weshayis 30520:17
weshaylooking GOOD!!20:17
rlandythat is all matching20:17
rlandyneed a little party20:17
weshayI think I heard discussion on openvswitch20:19
weshayhttp://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-master/81fcdfd/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz20:19
weshayprobably can allow earasing20:19
weshayI hate that re: upstream playbooks.. installing openvswitch20:19
weshayrlandy did you see20:20
weshayhttp://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/fac7dc1/job-output.txt.gz20:20
weshaymismatch on the checksum20:20
weshayhttp://images.rdoproject.org/master/rdo_trunk/305e280cab53c284da05dfbb3e7485db696d541c_ee467b40/overcloud-full.tar.md520:21
rlandylooking20:21
weshayI forgot to check for that when I looked at the images20:21
rlandyhmmm20:22
rlandyweshay: let's check on server20:22
rlandyweshay: checksum is there20:23
weshayrlandy aye.. it's there20:24
weshaycat overcloud-full.tar.md520:24
weshayaebfe1ae1df38b34ac8819868a25637e  overcloud-full.tar20:24
rlandyweshay: maybe there is a lag20:24
weshayrlandy HA20:25
weshayrlandy wrong server20:25
weshayrlandy 2019-08-05 15:43:22.929003 | primary |         "http://images.rdoproject.org/master/rdo_trunk/305e280cab53c284da05dfbb3e7485db696d541c_ee467b40/overcloud-full.tar.md5"20:25
weshayrlandy need to update the release file20:25
rlandyoh - lol20:25
rlandyweshay: needs patches in https://review.rdoproject.org/r/#/c/21649/20:26
rlandythe problem is that there are images links in the ovb settings20:26
rlandythat are overriding the settings in the release file20:26
rlandyweshay: see https://review.opendev.org/#/c/673891/20:26
rlandyI need to talk to sshnaidm about those20:27
rlandy<rlandy> or we would have to switch case which image files are used in the ovb settings20:27
rlandy<sshnaidm> rlandy, yes, environment is overriding release specifics, it was intentional20:27
rlandyI ma confused about which review is going forward here20:34
rlandy    https://review.rdoproject.org/r/#/c/21593/  Add login role -  is this ready for review??20:34
rlandy    <panda> rlandy: we're probably going to use https://review.opendev.org/673481 instead of 2159320:34
rlandy    https://review.opendev.org/#/c/673481/   [WIP] expand login subtasks so it can be used in all rdo jobs20:34
rlandy^^ zbr20:34
rlandyweshay: pls vote on these two ... https://review.opendev.org/#/c/673517/ and https://review.opendev.org/#/c/673757/20:35
zbrwell, nobody told me about the upstream one. but i added myself and I can take care of it tomorrow. now is too late.20:36
rlandyzbr: np - pls discuss with panda  and let us know20:37
rlandyw +'ed20:40
rlandyweshay: k - anything else we need done here?20:44
weshayrlandy I guess sagi was testing w/ centos images20:44
rlandyweshay: ack - I ma modifying that patch20:44
rlandyidk what's with the hardened image20:45
rlandyif we need that or not20:45
weshayrlandy no we do not20:49
weshayin check is fine20:49
weshaybut not in promotion20:49
*** Goneri has quit IRC20:53
*** ade_lee has quit IRC21:08
*** weshay is now known as weshay_dentist21:09
*** vinaykns has quit IRC21:45
*** jtomasek has quit IRC22:04
*** aakarsh|2 has quit IRC22:11
*** tosky has joined #oooq22:26
*** rlandy is now known as rlandy|bbl22:42
*** tosky has quit IRC22:47
*** Vorrtex has quit IRC22:58
*** dsneddon has quit IRC23:18
*** dsneddon has joined #oooq23:36
*** dsneddon has quit IRC23:43
*** ade_lee has joined #oooq23:50
*** chem has quit IRC23:55
*** dsneddon has joined #oooq23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!