Monday, 2020-02-17

*** dsneddon has joined #rdo00:05
rdogerritrdo-trunk proposed rdoinfo master: Bump rdoinfo ussuri-uc tags to latest upper-constraints  https://review.rdoproject.org/r/2479600:08
*** dsneddon has quit IRC00:10
*** jmlowe has joined #rdo00:16
*** jmlowe has quit IRC00:23
*** jmlowe has joined #rdo00:32
*** jmlowe has quit IRC00:55
*** dsneddon has joined #rdo01:13
*** dsneddon has quit IRC01:18
*** dsneddon has joined #rdo01:21
*** dsneddon has quit IRC01:26
*** rlandy has joined #rdo01:32
*** rlandy has quit IRC01:33
*** dsneddon has joined #rdo01:55
*** dsneddon has quit IRC02:00
*** fultonj has joined #rdo02:48
*** paragn has joined #rdo02:58
*** paragn has joined #rdo03:01
rdogerritMerged openstack/metalsmith-distgit train-rdo: python-metalsmith-0.15.1-1  https://review.rdoproject.org/r/2496903:33
rdogerritMerged openstack/sushy-distgit rocky-rdo: python-sushy-1.6.1-1  https://review.rdoproject.org/r/2497003:34
rdogerritMerged rdoinfo master: Promote CBS tags update for stein-release  https://review.rdoproject.org/r/2490903:35
*** jdandrea has joined #rdo03:50
*** jdandrea has quit IRC03:51
*** dsneddon has joined #rdo03:55
*** dsneddon has quit IRC04:00
*** ykarel|away is now known as ykarel04:05
*** udesale has joined #rdo04:10
*** ramishra has joined #rdo04:40
*** dsneddon has joined #rdo04:45
*** skramaja has joined #rdo04:50
*** dsneddon has quit IRC04:51
*** dsneddon has joined #rdo05:00
*** dpawlik has joined #rdo05:21
*** dsneddon has quit IRC05:24
*** dsneddon has joined #rdo05:51
*** ramishra_ has joined #rdo06:03
*** ramishra has quit IRC06:05
*** lmiccini has joined #rdo06:16
rdogerritYatin Karel proposed openstack/cinder-tempest-plugin-distgit stein-rdo: python-cinder-tests-tempest-0.3.1-1  https://review.rdoproject.org/r/2494906:17
*** chasmo77 has quit IRC06:23
*** marios|ruck has joined #rdo06:23
*** dpawlik has quit IRC06:31
*** dpawlik has joined #rdo06:31
rdogerritMerged rdoinfo master: Promote CBS tags update for rocky-release  https://review.rdoproject.org/r/2490806:36
*** aufi has joined #rdo06:37
*** bandini has joined #rdo06:41
*** dpawlik has quit IRC06:42
*** dpawlik has joined #rdo06:48
*** jfrancoa has joined #rdo06:54
*** iurygregory has joined #rdo06:57
rdogerritYatin Karel created rdoinfo master: Pin zaqar-tempest-plugin to workig commit  https://review.rdoproject.org/r/2497506:58
*** ykarel is now known as ykarel|afk07:01
rdogerritChandan Kumar created config master: Move rhel-8 build base image to new image server  https://review.rdoproject.org/r/2497607:02
*** jbadiapa has joined #rdo07:05
*** ykarel|afk is now known as ykarel07:14
*** jcapitao has joined #rdo07:35
*** Jeffrey4l has quit IRC07:41
*** Jeffrey4l has joined #rdo07:41
*** apetrich has joined #rdo07:46
*** ralonsoh has joined #rdo07:52
*** maciejjozefczyk has joined #rdo07:57
*** slaweq_ has joined #rdo07:58
*** slaweq_ is now known as slaweq07:59
*** ccamposr has joined #rdo08:00
*** jlibosva has joined #rdo08:02
*** rpittau|afk is now known as rpittau08:10
*** tkajinam has quit IRC08:15
*** apetrich has quit IRC08:19
*** ykarel is now known as ykarel|lunch08:22
*** tesseract has joined #rdo08:30
*** amoralej|off is now known as amoralej08:37
*** ccamacho has joined #rdo08:37
soniya29arxcruz, I didn't get your comment in this patch- https://review.opendev.org/#/c/706218/. What am I supposed to do?08:39
*** dtantsur|afk is now known as dtantsur08:41
*** apetrich has joined #rdo08:45
rdogerritChandan Kumar proposed rdo-jobs master: [DNM] master periodic movement to new image server  https://review.rdoproject.org/r/2492908:49
*** jfrancoa has left #rdo08:51
*** tosky has joined #rdo08:51
*** xek has joined #rdo08:52
*** jpena|off is now known as jpena08:52
*** migarcia|away is now known as migarcia09:00
*** jpich has joined #rdo09:01
rdogerritMerged openstack/ironicclient-distgit rocky-rdo: python-ironicclient-2.5.4-1  https://review.rdoproject.org/r/2494009:06
rdogerritSagi Shnaidman created config master: Add rhel8 ovb job to tripleo-ci repo  https://review.rdoproject.org/r/2497709:06
sshnaidmchandankumar, ^^09:07
*** apevec has joined #rdo09:16
*** shardy has joined #rdo09:19
rdogerritJavier Peña created rdoinfo master: Pin rally on stein and train to last py2-compatible commit  https://review.rdoproject.org/r/2497809:22
shardyHi all, are there known issues with using the current pin for master atm?09:23
*** paramite|clone has joined #rdo09:23
shardyI see the master promotion hasn't happened for 7 days, but the version (6efcfd8) for ironic seems to be from last December?09:23
rdogerritMerged config master: Move rhel-8 build base image to new image server  https://review.rdoproject.org/r/2497609:27
*** alphacc has quit IRC09:27
*** derekh has joined #rdo09:33
*** chem has joined #rdo09:40
*** adellam has joined #rdo09:41
amoralejshardy, we pinned it for compabitility with python209:42
amoralejuntil we can move out of centos709:42
shardyamoralej: Aha - is there any other pin we can consume on RHEL/CentOS8 to get the latest version?09:42
shardywe need a bugfix that landed last month09:42
amoralejcurrently not09:43
shardy:(09:43
amoralejshardy, we'd like to get a centos8 promotion running09:43
amoralejand then get rid of cento709:43
amoralejwe need to check with oooq09:43
shardyrpittau: ^^ sounds like we may need to build our own ironic packages or install from source09:43
amoralejshardy, iiuc you need it for tripleo jobs?09:44
amoralejor where exactly you need it?09:44
amoralejfor metal3?09:44
shardyamoralej: yeah it's for metal3 container images09:44
amoralejshardy, centos7 packages are fine?09:44
shardyNo we moved to centos809:45
shardyhttps://github.com/metal3-io/ironic-image/blob/master/Dockerfile09:45
amoralejjpena, may we start creating centos8-master-head?09:45
amoralejthat would follow master branches without pins09:46
jpenaamoralej: sure, we can09:46
shardyand switched to master so we can test with the latest Redfish patches, and https://review.opendev.org/#/c/701765/ which is the bugfix I mentioned09:46
jpenaamoralej: it's probably going to be quite broken, since we have many pinned packages, but it will help09:46
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: creating execution path split  https://review.rdoproject.org/r/2467609:46
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace legacy promoter function  https://review.rdoproject.org/r/2467709:46
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace promote_all_links legacy function  https://review.rdoproject.org/r/2467809:46
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace setup_logging  https://review.rdoproject.org/r/2468409:46
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace fetch_hashes  https://review.rdoproject.org/r/2468709:46
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace fetch_jobs  https://review.rdoproject.org/r/2469909:46
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace get_latest_hashes  https://review.rdoproject.org/r/2469409:46
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace tag_qcow_images  https://review.rdoproject.org/r/2471409:46
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace check named hashes function  https://review.rdoproject.org/r/2471509:46
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace tag_containers  https://review.rdoproject.org/r/2471609:46
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace promote_link  https://review.rdoproject.org/r/2472109:46
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Compoment pipeline: Implement aggregate hashes for promoter workflow  https://review.rdoproject.org/r/2477409:46
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Adapt staging environment to DlrnHash and add component pipeline scenario  https://review.rdoproject.org/r/2478209:46
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: [WIP] Adapt staging environment for the component pipeline  https://review.rdoproject.org/r/2481809:46
shardyamoralej, jpena: thanks - if we can figure out a workaround that would be great09:48
shardyany ETA on moving to centos8 and removing the pins?09:48
amoralejjpena, but the pinned packages should be fine in centos809:49
amoralejwith python309:49
jpenaamoralej: but we're only pinning on ussuri-uc09:49
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: creating execution path split  https://review.rdoproject.org/r/2467609:50
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace legacy promoter function  https://review.rdoproject.org/r/2467709:50
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace promote_all_links legacy function  https://review.rdoproject.org/r/2467809:50
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace setup_logging  https://review.rdoproject.org/r/2468409:50
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace fetch_hashes  https://review.rdoproject.org/r/2468709:50
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace fetch_jobs  https://review.rdoproject.org/r/2469909:50
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace get_latest_hashes  https://review.rdoproject.org/r/2469409:50
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace tag_qcow_images  https://review.rdoproject.org/r/2471409:50
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace check named hashes function  https://review.rdoproject.org/r/2471509:50
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace tag_containers  https://review.rdoproject.org/r/2471609:50
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace promote_link  https://review.rdoproject.org/r/2472109:50
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Compoment pipeline: Implement aggregate hashes for promoter workflow  https://review.rdoproject.org/r/2477409:50
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Adapt staging environment to DlrnHash and add component pipeline scenario  https://review.rdoproject.org/r/2478209:50
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: [WIP] Adapt staging environment for the component pipeline  https://review.rdoproject.org/r/2481809:50
rdogerritMerged config master: Add rhel8 ovb job to tripleo-ci repo  https://review.rdoproject.org/r/2497709:53
*** gfidente has joined #rdo09:54
rdogerritJavier Peña created rdo-infra/puppet-dlrn master: Add centos8-master-head worker  https://review.rdoproject.org/r/2498009:54
*** udesale_ has joined #rdo10:00
*** udesale_ has quit IRC10:01
rdogerritJavier Peña proposed rdo-infra/puppet-dlrn master: Add centos8-master-head worker  https://review.rdoproject.org/r/2498010:01
*** udesale has quit IRC10:01
*** udesale has joined #rdo10:01
*** ykarel|lunch is now known as ykarel10:05
*** ccamposr has quit IRC10:06
*** ccamposr has joined #rdo10:06
rdogerritMerged config master: Added secrets for images-vexxhost zuul configuration  https://review.rdoproject.org/r/2477010:14
chandankumardpawlik, ^^, let'see how the migration went10:14
chandankumardpawlik, let's turnoff image server tomorrow?10:15
dpawlikchandankumar, probably I will disable uploader user access today, if everything goes well10:15
dpawlikthen I sync images that were created to new host.10:15
dpawlikbut before that, we will change the domain from images-vexxhost to normal images name10:16
chandankumardpawlik, yes, until it lands https://review.opendev.org/#/c/707359/ and https://review.opendev.org/#/c/707574/ lands10:16
dpawlikand if all will be fine, we can stop old server10:16
dpawlikchandankumar, sure. I'm waiting for gate tests10:17
dpawlikI will notify on #rdo and #oooq about hour when we will change the DNS record10:17
*** ccamposr has quit IRC10:30
rdogerritChandan Kumar proposed rdo-jobs master: Get rhui version in podman jobs  https://review.rdoproject.org/r/2494410:35
*** udesale has quit IRC10:38
rdogerritMerged rdoinfo master: Pin rally on stein and train to last py2-compatible commit  https://review.rdoproject.org/r/2497810:52
*** leanderthal has joined #rdo10:55
rdogerritMerged rdoinfo master: Pin zaqar-tempest-plugin to workig commit  https://review.rdoproject.org/r/2497510:58
*** rpittau is now known as rpittau|bbl11:03
rdogerritYatin Karel created openstack/zaqar-tempest-plugin-distgit rpm-master: Dummy commit to rebuild zaqar-tempest-plugin  https://review.rdoproject.org/r/2498111:05
*** avivgt has joined #rdo11:06
rdogerritMerged rdo-infra/puppet-dlrn master: Add centos8-master-head worker  https://review.rdoproject.org/r/2498011:09
amoralejjpena, the cert for https://codesearch.rdoproject.org/ has expired11:15
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: creating execution path split  https://review.rdoproject.org/r/2467611:17
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace legacy promoter function  https://review.rdoproject.org/r/2467711:17
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace promote_all_links legacy function  https://review.rdoproject.org/r/2467811:17
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace setup_logging  https://review.rdoproject.org/r/2468411:17
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace fetch_hashes  https://review.rdoproject.org/r/2468711:17
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace fetch_jobs  https://review.rdoproject.org/r/2469911:17
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace get_latest_hashes  https://review.rdoproject.org/r/2469411:17
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace tag_qcow_images  https://review.rdoproject.org/r/2471411:17
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace check named hashes function  https://review.rdoproject.org/r/2471511:17
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace tag_containers  https://review.rdoproject.org/r/2471611:17
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Modularize Promoter: replace promote_link  https://review.rdoproject.org/r/2472111:17
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Compoment pipeline: Implement aggregate hashes for promoter workflow  https://review.rdoproject.org/r/2477411:17
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: Adapt staging environment to DlrnHash and add component pipeline scenario  https://review.rdoproject.org/r/2478211:17
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: [WIP] Adapt staging environment for the component pipeline  https://review.rdoproject.org/r/2481811:17
pandatoday is going to be flood over flood ...11:17
*** jcapitao is now known as jcapitao_adk11:21
*** jcapitao_adk is now known as jcapitao_afk11:21
rdogerritYatin Karel created rdo-jobs master: [DNM] Test c8 cbs job  https://review.rdoproject.org/r/2498211:23
rdogerritYatin Karel proposed openstack/diskimage-builder-distgit train-rdo: [DNM] Test cbs build centos8  https://review.rdoproject.org/r/2493511:27
ykarelamoralej, can u clear dummy commit https://review.rdoproject.org/r/#/c/24981/11:44
ykarelzaqar-tempest11:44
amoralejmerging11:44
*** raildo has joined #rdo11:45
ykarelack11:45
*** gfidente has quit IRC11:47
*** jpich has quit IRC11:50
*** jpich has joined #rdo11:51
rdogerritMerged openstack/zaqar-tempest-plugin-distgit rpm-master: Dummy commit to rebuild zaqar-tempest-plugin  https://review.rdoproject.org/r/2498111:51
*** ramishra_ has quit IRC11:52
*** ykarel is now known as ykarel|afk11:54
*** marios|ruck is now known as marios|ruck|food11:55
*** ramishra has joined #rdo11:55
arxcruzsoniya29: we are not using validate-tempest anymore iirc11:58
*** rfolco has joined #rdo12:04
*** mvkr has joined #rdo12:10
rdogerritYatin Karel proposed rdo-jobs master: [DNM] Test c8 cbs job  https://review.rdoproject.org/r/2498212:14
*** marios|ruck|food is now known as marios|ruck12:14
*** jpich has quit IRC12:27
*** jpich has joined #rdo12:28
*** udesale has joined #rdo12:31
*** jpena is now known as jpena|lunch12:35
*** fuzzball81 is now known as jjoyce12:36
*** rpittau|bbl is now known as rpittau12:37
*** jaosorior has joined #rdo12:37
*** rh-jelabarre has joined #rdo12:43
apevecamoralej, I've added added FTR comment in https://bugs.centos.org/view.php?id=16968 to proceed with exclusivearches12:45
amoralejapevec, given that we will build rabbitmq in messaging sig12:46
apevecalso replied on rdo list advocating any workarounds needed to get c8 deps ready for oooci12:46
amoraleji was thinking in doing exclusivearch messaging sig12:46
amoralejand keep cloudsig with aarch6412:46
amoralejeven if it's not really usable until 8.212:46
apevecamoralej, ok, that works for me, mrunge wdyt^12:46
*** maciejjozefczyk has quit IRC12:47
amoralejapevec, i wrote a bit about rabbit in messagin in https://review.rdoproject.org/etherpad/p/rabbitmq3.8-messaging-sig12:47
apevecamoralej, but we want be able to publish aarch64 cloudsig either12:47
apevecerr12:47
apevecwe won't*12:47
apevecsince we'll depend on msg sig12:47
apevecw/o it12:47
amoralejbut we could skip it at publish time instead of at build time?12:48
amoralejalso by ussuri, there may be 8.212:48
apevecyeah, although we might hit missing aarch64 rhel stuff somewhere else12:48
amoraleji'd like to do full rebuild12:48
amoralejsorry12:48
amoraleji'd like to avoid full rebuild12:48
apevecah yes, that's good point12:48
apevecyeah, keep what we have12:48
apevecfor publish, we need to do our own stuff on trunk.rdo for now anyway12:49
apevecand we'll see what new publishing/signing looks like when we get it (ETA was not solid as of last week)12:49
amoralejyes12:50
*** rlandy has joined #rdo12:55
*** marios|ruck is now known as marios|ruck|call13:01
*** morazi has joined #rdo13:04
*** jcapitao_afk is now known as jcapitao13:08
*** maciejjozefczyk has joined #rdo13:09
*** amoralej is now known as amoralej|lunch13:12
dpawlikchandankumar, merging those 2 PSs took almost half of the day xD13:15
dpawlikchandankumar, and its still not merged13:15
chandankumardpawlik, it's tripleo reviews13:15
*** ccamposr has joined #rdo13:19
*** gfidente has joined #rdo13:25
*** jpena|lunch is now known as jpena13:34
*** ykarel|afk is now known as ykarel13:46
rdogerritChandan Kumar created config master: Switch to images.rdoproject.org to distro images  https://review.rdoproject.org/r/2498413:52
rdogerritYatin Karel proposed rdo-jobs master: [DNM] Test c8 cbs job  https://review.rdoproject.org/r/2498213:53
*** amoralej|lunch is now known as amoralej13:53
*** Goneri has joined #rdo13:53
*** marios|ruck|call is now known as marios|ruck13:54
*** dmacpher has joined #rdo13:57
*** ekultails has joined #rdo13:59
*** marios|ruck is now known as marios|ruck|call14:00
*** marios|ruck|call is now known as marios|ruck14:00
rdogerritRafael Folco proposed config master: Switch to the new get_hash role for promoting hashes  https://review.rdoproject.org/r/2484714:13
*** lbragstad__ is now known as lbragstad14:15
*** jdandrea has joined #rdo14:19
*** lbragstad has quit IRC14:21
*** eharney has quit IRC14:22
rdogerritYatin Karel proposed rdo-jobs master: [DNM] Test c8 cbs job  https://review.rdoproject.org/r/2498214:27
rdogerritRonelle Landy proposed config master: get_hash role requires that the promote_source be defined  https://review.rdoproject.org/r/2496614:27
rdogerritRonelle Landy proposed config master: Change component promotion labels  https://review.rdoproject.org/r/2451214:30
rdogerritRonelle Landy proposed config master: centos8: promoted-components to tripleo-ci-testing  https://review.rdoproject.org/r/2451114:31
dpawlikchandankumar, https://review.rdoproject.org/r/#/c/24984/ => during the migration, both domains: images.rdoproject.org and imagex-vexxhost.rdoproject.org will be set to the vexxhost host (old server will be change to images-old) THat will be merged after edit DNS record:  https://softwarefactory-project.io/r/#/c/17428/3/playbooks/site.yaml14:38
*** nweinber has joined #rdo14:42
rdogerritrdo-trunk created openstack/tempest-distgit rpm-master: openstack-tempest: failed to build c674736  https://review.rdoproject.org/r/2498514:43
rdogerritRafael Folco proposed config master: integration: promoted-components to tripleo-ci-testing  https://review.rdoproject.org/r/2451114:43
rdogerritRafael Folco proposed config master: integration: promoted-components to tripleo-ci-testing  https://review.rdoproject.org/r/2451114:43
*** jdandrea has quit IRC14:44
*** jdandrea has joined #rdo14:53
*** artom has joined #rdo14:56
*** TrevorV has joined #rdo14:58
*** leanderthal has quit IRC15:00
*** eharney has joined #rdo15:05
*** gfidente has quit IRC15:06
*** jpich has quit IRC15:09
*** jpich has joined #rdo15:10
*** gfidente has joined #rdo15:19
rdogerritYatin Karel created gating_scripts master: [DNM] Add support for CentOS8 CBS builds  https://review.rdoproject.org/r/2498615:20
*** ykarel is now known as ykarel|away15:20
rdogerritRafael Folco created config master: Switch to new get_hash role in promote hash jobs  https://review.rdoproject.org/r/2498715:21
rdogerritYatin Karel proposed openstack/diskimage-builder-distgit train-rdo: [DNM] Test cbs build centos8  https://review.rdoproject.org/r/2493515:21
rdogerritYatin Karel proposed gating_scripts master: [DNM] Add support for CentOS8 CBS builds  https://review.rdoproject.org/r/2498615:32
*** skramaja has quit IRC15:33
*** chandankumar is now known as raukadah15:33
rdogerritRafael Folco proposed config master: integration: promoted-components to tripleo-ci-testing  https://review.rdoproject.org/r/2451115:40
*** ccamposr has quit IRC15:46
*** ccamposr has joined #rdo15:46
*** jcapitao is now known as jcapitao_afk15:49
*** ccamacho has quit IRC15:58
*** ccamacho has joined #rdo15:58
mjturekI'm having trouble ssh'ing through ansible from cico jenkins to cico ppc64le nodes http://paste.openstack.org/show/789647/16:01
mjturekanyone seen anything like this before?16:01
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: [WIP] Adapt staging environment for the component pipeline  https://review.rdoproject.org/r/2481816:03
mjturekI think I figured it out, StrictHostKeyChecking=no vanished, not sure why yet16:05
*** ccamacho has quit IRC16:05
*** jcapitao_afk is now known as jcapitao16:06
*** jdandrea_ has joined #rdo16:08
*** jdandrea has quit IRC16:08
*** aufi has quit IRC16:09
*** maciejjozefczyk has quit IRC16:09
rdogerritMerged config master: get_hash role requires that the promote_source be defined  https://review.rdoproject.org/r/2496616:09
*** d0ugal has quit IRC16:10
rdogerritRafael Folco proposed config master: integration: promoted-components to tripleo-ci-testing  https://review.rdoproject.org/r/2451116:11
rdogerritRafael Folco proposed config master: integration: promoted-components to tripleo-ci-testing  https://review.rdoproject.org/r/2451116:11
amoralejykarel|away, i've created new builds forcing ruby-2.516:13
amoralejtesting it in https://review.rdoproject.org/r/#/c/24935/16:13
*** jdandrea_ has quit IRC16:15
amoralejmmm, wrong review16:16
ykarel|awayamoralej, cool16:16
ykarel|awayhmm i saw ruby-shadow build some time back16:16
*** jdandrea has joined #rdo16:17
amoralejykarel|away, which review were we using to test cbs repos?16:17
ykarel|awayamoralej, upstream?16:18
ykarel|awaypoi?16:18
amoralejpoi16:18
amoralejwhere did we catch the issue with puppet?16:18
ykarel|awayamoralej, https://review.opendev.org/#/c/706175/16:18
ykarel|awayin packstack we catched ^^16:18
ykarel|awayso u can update delorean-deps-cbs.repo16:19
ykarel|awayin mock server16:19
amoralejah, that was the trick16:19
ykarel|awayamoralej, just disable delorean-master-testing-rabbit16:20
amoralejyeah16:20
ykarel|awayamoralej, so u build all including rabbitmq?16:20
amoralejnop16:20
amoralejneed to use rabbit from copr16:21
ykarel|awayokk so u adjust that repo as per need16:21
ykarel|awayjust removing ruby thing could do the trick16:21
ykarel|awayrubygem-ruby-shadow ruby-augeas16:21
amoralejyeah, just did it16:22
ykarel|awayack16:22
rdogerritwes hayutin proposed config master: Change component promotion labels  https://review.rdoproject.org/r/2451216:22
ykarel|awayamoralej, u might have already checked i had started testing with wip patches https://review.rdoproject.org/r/#/c/24935/16:24
ykarel|awaywill focus tomorrow more on it,16:24
amoralejykarel|away, ack16:24
ykarel|awayso if u have suggestions regarding the direction, please comment16:24
rdogerritMerged config master: integration: promoted-components to tripleo-ci-testing  https://review.rdoproject.org/r/2451116:28
rdogerritRonelle Landy proposed config master: Change component promotion labels  https://review.rdoproject.org/r/2451216:29
rdogerritJoel Capitao proposed openstack/tempest-distgit rpm-master: Remove deprecated verify-tempest-config binary  https://review.rdoproject.org/r/2498516:30
mjturekjpena: http://paste.openstack.org/show/789656/ I can't seem to connect to cico nodes from the jenkins nodes.. Has there been a key change?16:37
jpenamjturek: we had to change the ssh key for the new logserver. dpawlik can you help ^^ ?16:37
mjturekjpena wait, I don't think that's the issue16:38
mjturekit's ssh'ing from jenkins to cico, which is probably a centos problem16:38
* dpawlik reading16:38
dpawlikmjturek, where did you get such log?16:40
mjturekdpawlik: https://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/2026/console16:40
mjturekdpawlik: I don't think it's related to the logserver16:40
mjturekI'm pretty sure it's an issue with cico only16:41
dpawlikmjturek, worst thing in such message is: you don't know if the key is wrong or maybe the priv key is not set to mode 0600 or authorized_keys16:43
mjturekpretty much16:43
mjturekdpawlik last working build https://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/2005/consoleFull16:43
* dpawlik checking container build workflow16:43
dpawlikmjturek, so its on 10th of Feb16:44
rdogerritMerged rdo-infra/ci-config master: Modularize Promoter: creating execution path split  https://review.rdoproject.org/r/2467616:45
dpawliklet me check when we switch to the new host16:45
mjtureksure16:45
*** ccamposr has quit IRC16:46
dpawlikmjturek, https://review.rdoproject.org/r/#/q/topic:rdo-logs according to this, build should fail on 12th not on 1016:46
*** ccamposr has joined #rdo16:46
mjturekdpawlik: actually that lines up16:49
mjturekdpawlik: the first couple of failures could ssh16:49
mjturekdpawlik: the first one that couldn't was this one https://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/2011/console16:49
mjturekon the 12th16:49
mjturekdpawlik: we then tried to fix this by setting "-o UserKnownHostsFile=/dev/null" but then we got the authentication failures seen in the paste http://paste.openstack.org/show/789656/16:51
bahaIf you look at the timestamps, that's the very first job run after the patch merged16:51
bahaFWIW16:52
dpawlikyup, so problem is with the key16:52
*** jmlowe has joined #rdo16:52
dpawlikbaha, mjturek: did you change the key in known_hosts?16:53
dpawlikAFAIK ykarel|away was rebuilding our images to put the pub key there16:53
bahaWe don't have access to the jenkins environment setup, so we haven't adjusted anything permanent16:53
bahaLike the known_hosts file16:54
*** rpittau is now known as rpittau|afk16:54
dpawlikbaha, ack16:54
*** marios|ruck is now known as marios|ruck|call16:55
*** jlibosva has quit IRC17:00
*** dtantsur is now known as dtantsur|afk17:03
dpawlikso if I understand correctly, it fails on this step https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/scripts/tripleo-upstream-collect-logs.sh#L3617:03
*** marios|ruck|call is now known as marios|out17:04
bahaIt does fail there, but it fails earlier in, too. It also fails at https://github.com/rdo-infra/ci-config/blob/master/jenkins/jobs/tripleo-upstream-builders.yml#L75 , which is another attempt to SSH into the cico node.17:07
baha(for reference, the cico node is defined in openstack_hosts, so any step that attempts to use it will fail)17:08
*** migarcia is now known as migarcia|away17:11
*** ccamposr has quit IRC17:12
*** jmlowe has quit IRC17:13
*** ccamposr has joined #rdo17:13
*** jmlowe has joined #rdo17:15
*** lmiccini has quit IRC17:15
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: [WIP] Adapt staging environment for the component pipeline  https://review.rdoproject.org/r/2481817:16
*** ccamposr has quit IRC17:17
*** ccamposr has joined #rdo17:17
*** udesale has quit IRC17:17
rdogerritrdo-trunk created openstack/tempest-distgit rpm-master: openstack-tempest: failed to build c674736  https://review.rdoproject.org/r/2499017:18
*** gfidente has quit IRC17:18
*** ccamposr has quit IRC17:18
*** ccamacho has joined #rdo17:20
*** jcapitao is now known as jcapitao_off17:21
*** dpawlik has quit IRC17:22
*** jmlowe has quit IRC17:22
*** danpawlik has joined #rdo17:24
*** jlibosva has joined #rdo17:28
rdogerritMerged openstack/tempest-distgit rpm-master: Remove deprecated verify-tempest-config binary  https://review.rdoproject.org/r/2498517:29
*** adellam has quit IRC17:31
danpawlikbaha, its weird17:31
*** marios|out has quit IRC17:32
danpawlikbaha, is possible to login to the container to check?17:32
*** jlibosva has quit IRC17:32
rdogerritAlfredo Moralejo proposed gating_scripts master: [DNM] Add support for CentOS8 CBS builds  https://review.rdoproject.org/r/2498617:34
*** jpich has quit IRC17:38
rdogerritRonelle Landy created config master: Add role for get_hash to  the base definition.  https://review.rdoproject.org/r/2499117:38
*** bandini has quit IRC17:38
rlandyrfolco: ^^17:39
bahadanpawlik: I'm uncertain. We have SSH access to the jobs while they're running, but that's all that I'm aware of17:46
bahaSomeone here may be better equipped to check17:46
rdogerritMerged config master: Add role for get_hash to  the base definition.  https://review.rdoproject.org/r/2499117:47
bahajpena: Would you or dmsimard|off be able to login to the container, or is there someone else we could ask?17:48
danpawlikbaha, imho, problem is on jenkins node17:49
danpawlikjpena, you were restarting service on all jenkins slave yes?17:49
rdogerritRafael Folco proposed config master: Change component promotion labels  https://review.rdoproject.org/r/2451217:50
rdogerritRafael Folco proposed config master: Change component promotion labels  https://review.rdoproject.org/r/2451217:51
*** mvkr has quit IRC17:52
rdogerritAlfredo Moralejo proposed gating_scripts master: [DNM] Add support for CentOS8 CBS builds  https://review.rdoproject.org/r/2498617:52
danpawlikbaha, on last success build https://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/2005/consoleFull17:52
danpawlikit also have problem17:52
jpenadanpawlik, baha: based on the output, it looks like the node created by cico does not recognize the jenkins ssh key (which would make sense after we changed it). My doubt is how is that key assigned to the node17:54
*** jlibosva has joined #rdo17:55
rdogerritRafael Folco proposed config master: Deprecate get-hash task  https://review.rdoproject.org/r/2498717:57
rdogerritRafael Folco proposed config master: Deprecate get-hash task  https://review.rdoproject.org/r/2498717:57
bahadanpawlik: I think the "man in the middle" error is a red herring in this instance - we've been getting it off and on for a while because known_hosts isn't cleared between runs and the cico nodes have dynamic IP addresses17:59
*** jlibosva has quit IRC17:59
bahaThe real error that was causing all the failures was exposed after we started playing with the SSH options, which is that the cico nodes are rejecting our key, as jpena is suggesting17:59
*** derekh has quit IRC18:00
bahaWe have strict host key checking disabled, so up until that SSH key was rejected, we would just get man-in-the-middle warnings but it would connect and run the job anyway18:01
*** tesseract has quit IRC18:02
*** avivgt has quit IRC18:02
*** chasmo77 has joined #rdo18:02
bahaSo it sounds like the solution may be that someone from RDO needs to get in touch with someone from Centos to get the new keys up on the cico nodes?18:04
jpenaI'd say so. I need to run now, but I'll sync with danpawlik tomorrow about it. We'll probably need to open a bug in bugs.centos.org to get the ssh key changed (don't have the details)18:06
danpawlikbaha, normally it could be me, but I don't have accesss18:06
*** rcernin has quit IRC18:06
*** shardy has quit IRC18:06
*** jlibosva has joined #rdo18:06
bahaAlright! Well, thank you guys. The time/effort is very much appreciated18:06
danpawlikbaha, in the worst way, I can add old pub key and check if it will be ok18:07
danpawlikbaha, normally container should be able to push, so if it do that18:07
*** shardy has joined #rdo18:07
danpawlikwhat do you think?18:07
bahaThat might be a good test, just to see if it fixes the issue. Our next run is in 8 minutes18:07
bahaThe one after that is about 5 hours later18:08
danpawliklet me copy18:08
*** jmlowe has joined #rdo18:09
*** jpena is now known as jpena|off18:09
danpawlikbaha, added all keys. I will wait for the result and then I need to go :|18:10
ykarel|awaydanpawlik, u added keys on slaves?18:11
bahaThank you! Fortunately trying to connect to the cico node is one of the very first things the job does, so it shouldn't be too long of a wait!18:11
ykarel|awayjust reading above18:11
danpawlikykarel|away, no, just on logserver18:11
*** jlibosva has quit IRC18:11
danpawlikaaaaa18:11
ykarel|awaymmm18:11
ykarel|awaynot sure how logserver will solve that,18:11
ykarel|awayissue looks on slave sides, but me didn't looked much18:12
* ykarel|away checks logs18:12
danpawlikykarel|away, I think the same18:12
ykarel|awayso as per logs, ansible commands on cico nodes from slave nodes failing18:13
bahaOhh, I thought you meant the slaves! Sorry, I misunderstood.18:14
bahaWe're actually connecting to the logserver fine already in quite a few of the failed runs18:14
danpawlikbaha, I don't have access to jenkins slaves :/18:14
bahaIt doesn't seem to have the folder for our job exposed so the URL it outputs doesn't work, but we connect to it just fine18:14
danpawlikso only jpena|off, ykarel|away, amoralej can help18:15
bahadanpawlik: Gotcha! NP then, it can wait until jpena is back around tomorrow18:15
amoralejhow can i help?18:15
ykarel|awayme too also don't have access18:15
bahaamoralej: The short versions is that the cico nodes don't seem to have the new SSH keys for RDO jobs, we were considering putting the old key on the jenkins slaves as a test to see if it would fix our authentication issues to confirm that's what's going on18:16
ykarel|awayme also not sure about the history, how/why keys were changed for slaves, and how it used to be managed in cico nodes in past when keys are changed for slave nodes18:17
ykarel|awayi think answer lies in ^^ how to handle this now18:17
amoralejwhere can i see logs?18:17
ykarel|awayhttps://ci.centos.org/job/tripleo-upstream-containers-build-master-ppc64le/2026/consoleText18:17
*** shardy has quit IRC18:18
*** shardy has joined #rdo18:18
*** jmlowe has quit IRC18:19
*** alphacc has joined #rdo18:19
amoralejykarel|away, baha it's only with ppc nodes?18:20
*** rcernin has joined #rdo18:21
ykarel|awayamoralej, possibly yes, as tripleo quickstart jobs which require cico nodes are not seeing this issue18:21
bahaI have only worked with the ppc nodes so I can't really add more than that18:23
danpawlikbaha, so it may be the problem18:24
ykarel|awayamoralej, sorry i might be wrong, quickstart jobs didn't ran much due to no promotion in last couple of days18:24
amoraleji'll launch one now18:24
ykarel|awayamoralej, currently those are broken18:25
ykarel|awaydue to virtualenv hacks18:25
amoralejmmm18:25
amoralejlet's see how it goes18:25
ykarel|awayamoralej, https://bugs.launchpad.net/tripleo/+bug/186363118:25
openstackLaunchpad bug 1863631 in tripleo "lastest virtenv workarounds broke native tripleo-quickstart install" [Critical,Triaged]18:25
*** jmlowe has joined #rdo18:25
amoralejyeah, seen that18:26
*** jmlowe has quit IRC18:27
ykarel|awayi see last success from 11th, on 12th the virtualenv patch broked these jobs18:28
*** jmlowe has joined #rdo18:29
danpawlikykarel|away, in the same time was merged secrets for jenkins and logserver, but its coincidence18:35
ykarel|awaydanpawlik, what was the reasoning to change jenkins slaves?18:38
ykarel|awayi got the logserver part18:38
amoralejbaha, ykarel|away i see it's trying to connect using root user?18:38
amoralejinstead of jenkins18:38
ykarel|awaybut didn't understood the binding between the two keys18:38
rdogerritRafael Folco proposed config master: Deprecate get-hash task  https://review.rdoproject.org/r/2498718:38
ykarel|awayamoralej, earlier also it used root, i don't know the reasoning, but it used too18:38
*** ramishra has quit IRC18:38
ykarel|awayand was passing18:38
amoralejykarel|away, baha i've checked that i can log in as jenkins but not as root18:38
danpawlikamoralej, but jenkins slave tries to sync between container and the host or? So root is possible that is correct18:39
ykarel|awaymay be root login disabled now?18:39
amoralejyeah18:39
amoralejthat's what i was thinking18:39
amoralejmaybe, some change in cico nodes provisioning has disabled root access?18:39
ykarel|away^^ possible18:39
danpawlikbut... it want to pull from the container, not push yes?18:40
danpawlikI need to go, see you tomorrow18:41
ykarel|awaydanpawlik, it didn't reached the container part yet, gather facts is failing at ssh to nodes18:41
bahaFor reference, we build as root to avoid permissions issues18:41
*** jmlowe has quit IRC18:41
danpawlikykarel|away, ah!18:42
danpawlikamoralej, could you check permissions and owner of the ssh priv key and authorized keys if is ok?18:42
danpawliksame error is when its incorrect18:42
amoralejthere is something weird18:44
amoralejit seems sometimes works, some doesn't18:46
amoralejwe'll need help from centos infra guys18:46
amoraleji need to leave now18:46
*** jlibosva has joined #rdo18:46
ykarel|awayme too, let's catchup tomorrow18:46
ykarel|awaybaha, mjturek if u will be still around, can u check if there was some changes in cico nodes after 10th18:47
ykarel|awaylike some updates,18:47
ykarel|awayand ssh with root etc18:47
ykarel|awayon #centos-devel18:48
rdogerritRonelle Landy proposed rdo-jobs master: Add centos 8 standalone component jobs  https://review.rdoproject.org/r/2493818:48
amoralejykarel|away, baha i'm not sure if it works with jenkins18:49
amoralejwe need to reserve a cico node18:49
amoralejand check18:49
amoraleji did it but got weird results18:49
amoralejwhen i could log in i see an x86_64 node18:49
amoralejso, i'm not sure if i'm testing it fine18:50
ykarel|awayok let's do that tomorrow then and get more details18:50
*** mvkr has joined #rdo18:51
*** amoralej is now known as amoralej|off18:52
*** ralonsoh has quit IRC18:55
mjturekykarel|away: I'll ask in centos-devel but they seem a bit busy today18:56
*** sshnaidm is now known as sshnaidm|afk18:57
*** jlibosva has quit IRC18:58
*** jmlowe has joined #rdo19:00
*** Tengu has quit IRC19:07
*** Tengu has joined #rdo19:09
*** Tengu has quit IRC19:14
*** Tengu has joined #rdo19:15
rdogerritRonelle Landy created rdo-jobs master: Add centos 8 jobs for standalone.  https://review.rdoproject.org/r/2499219:18
*** Tengu has quit IRC19:21
*** maciejjozefczyk has joined #rdo19:22
*** Tengu has joined #rdo19:22
*** mvkr has quit IRC19:32
*** maciejjozefczyk_ has joined #rdo19:32
*** maciejjozefczyk has quit IRC19:33
*** jmlowe has quit IRC19:37
*** maciejjozefczyk has joined #rdo19:38
rdogerritRafael Folco proposed config master: Deprecate get-hash task  https://review.rdoproject.org/r/2498719:39
*** maciejjozefczyk_ has quit IRC19:40
*** jlibosva has joined #rdo19:41
*** Tengu has quit IRC19:44
*** jlibosva has quit IRC19:45
*** Tengu has joined #rdo19:46
*** openstackstatus has quit IRC19:49
*** Tengu has quit IRC19:51
*** maciejjozefczyk_ has joined #rdo19:52
*** jlibosva has joined #rdo19:53
rdogerritGabriele Cerami proposed rdo-infra/ci-config master: [WIP] Adapt staging environment for the component pipeline  https://review.rdoproject.org/r/2481819:53
*** maciejjozefczyk has quit IRC19:54
*** Tengu has joined #rdo19:55
*** jdandrea has quit IRC19:59
*** jlibosva has quit IRC20:01
*** jaosorior has quit IRC20:02
*** jdandrea has joined #rdo20:04
rdogerritHarry Rybacki created rdoinfo master: Add urllib-gssapi package to RDO as of Ussuri  https://review.rdoproject.org/r/2499320:09
*** maciejjozefczyk_ has quit IRC20:36
*** jlibosva has joined #rdo20:37
*** maciejjozefczyk has joined #rdo20:37
*** eharney has quit IRC20:43
*** morazi has quit IRC20:47
*** jlibosva has quit IRC20:48
rdogerritRonelle Landy created rdo-jobs master: Add centos 8 nodeset for multinode  https://review.rdoproject.org/r/2499420:59
rlandyweshay|ruck: rfolco: ^^20:59
*** Goneri has quit IRC20:59
*** jdandrea has quit IRC21:06
*** morazi has joined #rdo21:06
*** jdandrea has joined #rdo21:06
*** ccamposr has joined #rdo21:08
*** apevec has quit IRC21:09
*** nweinber has quit IRC21:09
*** jmlowe has joined #rdo21:15
*** ccamposr has quit IRC21:16
*** maciejjozefczyk has quit IRC21:17
rdogerritMerged rdo-jobs master: Add centos 8 nodeset for multinode  https://review.rdoproject.org/r/2499421:19
*** artom has quit IRC21:20
*** paramite|clone has quit IRC21:25
*** jlibosva has joined #rdo21:32
*** TrevorV has quit IRC21:33
*** jlibosva has quit IRC21:37
*** mvkr has joined #rdo21:38
*** eharney has joined #rdo21:39
*** danpawlik has quit IRC21:39
*** jdandrea has quit IRC21:39
*** xek has quit IRC21:45
cyberpearI see that RPMs have been built for RDO Train on CentOS 8, but not published yet? https://cbs.centos.org/koji/builds?tagID=195621:46
*** ccamposr has joined #rdo22:01
*** ccamposr has quit IRC22:02
cyberpearokay, found it here: https://cbs.centos.org/repos/cloud8-openstack-train-candidate/22:03
*** morazi has quit IRC22:04
*** Goneri has joined #rdo22:04
*** jlibosva has joined #rdo22:08
*** jlibosva has quit IRC22:12
*** lennyb has quit IRC22:17
*** slaweq has quit IRC22:17
*** lennyb has joined #rdo22:20
*** nweinber has joined #rdo22:23
*** jmlowe has quit IRC22:27
*** raildo has quit IRC22:29
*** nweinber has quit IRC22:34
*** jmlowe has joined #rdo22:36
rdogerritRonelle Landy created config master: Add misisng workspace var for base promote hash playbook  https://review.rdoproject.org/r/2499622:39
*** rh-jelabarre has quit IRC22:44
rdogerritMerged config master: Add misisng workspace var for base promote hash playbook  https://review.rdoproject.org/r/2499622:47
*** jmlowe has quit IRC22:53
*** tkajinam has joined #rdo22:59
*** tkajinam has quit IRC22:59
*** tkajinam has joined #rdo23:01
*** jlibosva has joined #rdo23:02
*** openstackstatus has joined #rdo23:02
*** ChanServ sets mode: +v openstackstatus23:02
*** igordc has joined #rdo23:03
*** ekultails has quit IRC23:06
*** jbrooks has quit IRC23:06
*** jlibosva has quit IRC23:08
*** jbrooks has joined #rdo23:09
*** slaweq has joined #rdo23:11
*** slaweq has quit IRC23:16
*** mvkr has quit IRC23:38
*** tosky has quit IRC23:49
*** artom has joined #rdo23:53
*** jlibosva has joined #rdo23:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!