| rdogerrit | rdo-trunk proposed rdoinfo master: Promote CBS tags update for ocata-testing https://review.rdoproject.org/r/10263 | 00:07 |
|---|---|---|
| rdogerrit | rdo-trunk proposed rdoinfo master: Promote CBS tags update for newton-testing https://review.rdoproject.org/r/10262 | 00:07 |
| rdogerrit | rdo-trunk created openstack/neutron-vpnaas-distgit rpm-master: openstack-neutron-vpnaas: failed to build 99d2687 https://review.rdoproject.org/r/10348 | 00:10 |
| rdobot | [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-master-current @ http://tinyurl.com/y8hxewm2 |#| Build failure on centos7-master/current: neutron-vpnaas: http://trunk.rdoproject.org/centos7-master/report.html | 00:14 |
| *** newmember has joined #rdo | 00:21 | |
| rdogerrit | rdo-trunk created openstack/neutron-vpnaas-distgit rpm-master: openstack-neutron-vpnaas: failed to build 99d2687 https://review.rdoproject.org/r/10349 | 00:22 |
| *** chasmo77 has joined #rdo | 00:28 | |
| *** newmember has quit IRC | 00:32 | |
| *** newmember has joined #rdo | 00:33 | |
| *** sgordon has quit IRC | 00:36 | |
| *** owalsh_pto has quit IRC | 00:36 | |
| *** noslzzp has quit IRC | 00:36 | |
| *** scorcoran_afk has quit IRC | 00:38 | |
| *** sgordon has joined #rdo | 00:45 | |
| *** owalsh_ has joined #rdo | 00:49 | |
| *** newmember has quit IRC | 00:51 | |
| *** owalsh has joined #rdo | 00:51 | |
| *** newmember has joined #rdo | 00:51 | |
| *** owalsh- has joined #rdo | 00:53 | |
| *** owalsh_ has quit IRC | 00:54 | |
| *** owalsh_ has joined #rdo | 00:55 | |
| *** newmember has quit IRC | 00:55 | |
| *** owalsh has quit IRC | 00:56 | |
| *** newmember has joined #rdo | 00:56 | |
| *** owalsh has joined #rdo | 00:58 | |
| *** owalsh- has quit IRC | 00:59 | |
| *** newmember has quit IRC | 01:01 | |
| *** owalsh- has joined #rdo | 01:01 | |
| *** owalsh_ has quit IRC | 01:01 | |
| *** newmember has joined #rdo | 01:02 | |
| *** owalsh_ has joined #rdo | 01:04 | |
| *** owalsh has quit IRC | 01:04 | |
| *** Duck has joined #rdo | 01:06 | |
| *** Duck has joined #rdo | 01:06 | |
| *** rhallisey has quit IRC | 01:07 | |
| *** owalsh has joined #rdo | 01:07 | |
| *** owalsh- has quit IRC | 01:08 | |
| *** owalsh- has joined #rdo | 01:10 | |
| *** owalsh_ has quit IRC | 01:11 | |
| *** owalsh has quit IRC | 01:14 | |
| *** julim has joined #rdo | 01:15 | |
| *** sgordon has quit IRC | 01:15 | |
| *** julim has quit IRC | 01:18 | |
| *** sgordon has joined #rdo | 01:20 | |
| *** rlandy has quit IRC | 01:20 | |
| *** vnogin has quit IRC | 01:35 | |
| *** Son_Goku has quit IRC | 01:38 | |
| *** Son_Goku has joined #rdo | 01:54 | |
| *** fragatina has quit IRC | 02:01 | |
| *** apetrich has quit IRC | 02:03 | |
| *** apetrich has joined #rdo | 02:04 | |
| *** threestrands has joined #rdo | 02:11 | |
| *** threestrands has quit IRC | 02:11 | |
| *** threestrands has joined #rdo | 02:11 | |
| *** julim has joined #rdo | 02:16 | |
| *** rwsu has joined #rdo | 02:22 | |
| *** catintheroof has joined #rdo | 02:24 | |
| *** gvrangan has joined #rdo | 02:31 | |
| *** gvrangan has quit IRC | 02:33 | |
| *** gvrangan has joined #rdo | 02:33 | |
| *** gvrangan has quit IRC | 02:38 | |
| *** catintheroof has quit IRC | 02:43 | |
| *** axisys has joined #rdo | 02:43 | |
| *** axisys has left #rdo | 02:45 | |
| *** dneary has quit IRC | 02:48 | |
| *** shreshtha has joined #rdo | 02:50 | |
| *** gildub has joined #rdo | 02:54 | |
| *** paragan has joined #rdo | 02:55 | |
| *** cshastri has joined #rdo | 02:57 | |
| *** gvrangan has joined #rdo | 03:04 | |
| rdobot | [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-master-current @ http://tinyurl.com/y8hxewm2 |#| Build failure on centos7-master/current: neutron-vpnaas: http://trunk.rdoproject.org/centos7-master/report.html | 03:11 |
| *** gvrangan has quit IRC | 03:18 | |
| *** newmember has quit IRC | 03:22 | |
| *** Son_Goku has quit IRC | 03:23 | |
| *** jhershbe has joined #rdo | 03:24 | |
| *** liujiong has joined #rdo | 03:26 | |
| *** vkmc has quit IRC | 03:29 | |
| *** vkmc has joined #rdo | 03:35 | |
| *** shreshtha has quit IRC | 03:36 | |
| *** jatanmalde has joined #rdo | 03:44 | |
| *** fragatina has joined #rdo | 03:49 | |
| *** fragatina has quit IRC | 03:49 | |
| *** fragatina has joined #rdo | 03:50 | |
| *** fragatina has quit IRC | 03:50 | |
| *** fragatina has joined #rdo | 03:51 | |
| *** ykarel has joined #rdo | 04:06 | |
| *** udesale has joined #rdo | 04:06 | |
| *** jatanmalde has quit IRC | 04:10 | |
| *** jatanmalde has joined #rdo | 04:14 | |
| *** dyasny__ has joined #rdo | 04:21 | |
| *** xaeth has quit IRC | 04:21 | |
| *** fragatina has quit IRC | 04:22 | |
| *** jpena|off has quit IRC | 04:23 | |
| *** lhinds has quit IRC | 04:23 | |
| *** tumblew has quit IRC | 04:23 | |
| *** tumblew has joined #rdo | 04:24 | |
| *** sc` has quit IRC | 04:24 | |
| *** lon has quit IRC | 04:24 | |
| *** sc` has joined #rdo | 04:25 | |
| *** lhinds has joined #rdo | 04:25 | |
| *** jpena|off has joined #rdo | 04:25 | |
| *** nhicher has quit IRC | 04:25 | |
| *** dyasny_ has quit IRC | 04:25 | |
| *** lon has joined #rdo | 04:25 | |
| *** nhicher has joined #rdo | 04:25 | |
| *** sasha21 has quit IRC | 04:26 | |
| *** fragatina has joined #rdo | 04:27 | |
| *** sasha2 has joined #rdo | 04:30 | |
| *** xaeth has joined #rdo | 04:30 | |
| *** sasha2 has quit IRC | 04:30 | |
| *** cshastri has quit IRC | 04:31 | |
| *** anilvenkata has joined #rdo | 04:39 | |
| *** ratailor has joined #rdo | 04:47 | |
| *** mercolino has joined #rdo | 05:02 | |
| *** janki has joined #rdo | 05:06 | |
| *** jatanmalde has quit IRC | 05:20 | |
| *** jatanmalde has joined #rdo | 05:20 | |
| *** jatanmalde has quit IRC | 05:25 | |
| *** gildub has quit IRC | 05:25 | |
| *** jatanmalde has joined #rdo | 05:25 | |
| *** jhershbe has quit IRC | 05:26 | |
| *** janki has quit IRC | 05:27 | |
| *** janki has joined #rdo | 05:27 | |
| *** iranzo has joined #rdo | 05:31 | |
| *** gildub has joined #rdo | 05:37 | |
| *** jhershbe has joined #rdo | 05:40 | |
| *** gildub has quit IRC | 05:46 | |
| *** threestrands has quit IRC | 05:46 | |
| *** fragatina has quit IRC | 05:47 | |
| *** jparrill has quit IRC | 05:50 | |
| *** gkadam has joined #rdo | 05:57 | |
| *** cshastri has joined #rdo | 05:58 | |
| *** paragan has quit IRC | 05:58 | |
| *** threestrands has joined #rdo | 05:58 | |
| *** threestrands has quit IRC | 05:58 | |
| *** threestrands has joined #rdo | 05:58 | |
| rdogerrit | Yatin Karel created config master: Use RDO dev user/email for updating changelog https://review.rdoproject.org/r/10350 | 06:01 |
| *** lkuchlan has joined #rdo | 06:01 | |
| *** threestrands has quit IRC | 06:03 | |
| rdobot | [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-master-current @ http://tinyurl.com/y8hxewm2 |#| Build failure on centos7-master/current: neutron-vpnaas: http://trunk.rdoproject.org/centos7-master/report.html | 06:11 |
| *** gildub has joined #rdo | 06:12 | |
| *** paragan has joined #rdo | 06:15 | |
| *** abregman has joined #rdo | 06:16 | |
| *** jackNemrod has joined #rdo | 06:18 | |
| *** sbonazzo has joined #rdo | 06:27 | |
| *** agurenko has quit IRC | 06:27 | |
| *** iranzo has quit IRC | 06:42 | |
| *** emalikov has joined #rdo | 06:45 | |
| rdogerrit | Yatin Karel proposed openstack/neutron-vpnaas-distgit rpm-master: Remove neutron-vpn-agent service and binary from package https://review.rdoproject.org/r/10349 | 06:46 |
| ykarel | dmsimard, jpena|off any idea why jobs are in queued state from long in: https://review.rdoproject.org/zuul/ | 06:50 |
| *** tosky has joined #rdo | 06:53 | |
| *** iranzo has joined #rdo | 06:55 | |
| *** iranzo has quit IRC | 06:55 | |
| *** iranzo has joined #rdo | 06:55 | |
| *** jtomasek has joined #rdo | 06:57 | |
| *** rcernin has quit IRC | 06:59 | |
| *** jackNemrod has quit IRC | 06:59 | |
| *** jtomasek has quit IRC | 07:00 | |
| *** gkadam has quit IRC | 07:01 | |
| *** brault has joined #rdo | 07:02 | |
| *** spectr has joined #rdo | 07:08 | |
| *** agurenko has joined #rdo | 07:09 | |
| *** egonzalez has joined #rdo | 07:14 | |
| *** anshul has joined #rdo | 07:17 | |
| *** jprovazn has joined #rdo | 07:17 | |
| *** pcaruana has joined #rdo | 07:17 | |
| *** tosky has quit IRC | 07:19 | |
| *** dsneddon has joined #rdo | 07:20 | |
| *** vnogin has joined #rdo | 07:20 | |
| *** gildub has quit IRC | 07:21 | |
| *** gildub has joined #rdo | 07:22 | |
| adarazs|ruck | any idea why we have jobs for 20 hours in the zuul queue? https://review.rdoproject.org/zuul/ | 07:24 |
| adarazs|ruck | is there something with zuul/rdocloud? | 07:24 |
| adarazs|ruck | or it's still that slowness that dmsimard talked about? | 07:24 |
| adarazs|ruck | panda|ruck|off: this rdocloud slowness is getting bad beyond usable level. | 07:25 |
| *** lebauce has joined #rdo | 07:25 | |
| *** paramite has joined #rdo | 07:26 | |
| *** jhershbe has quit IRC | 07:32 | |
| *** jtomasek has joined #rdo | 07:33 | |
| *** racedo_ has joined #rdo | 07:34 | |
| *** mercolino has quit IRC | 07:34 | |
| *** vnogin has quit IRC | 07:36 | |
| *** paragan has quit IRC | 07:37 | |
| rdogerrit | rdo-trunk created openstack/congress-distgit rpm-master: openstack-congress: failed to build 5526f76 https://review.rdoproject.org/r/10351 | 07:39 |
| *** adarazs|ruck is now known as adarazs|rover | 07:40 | |
| *** jackNemrod has joined #rdo | 07:43 | |
| *** jackNemrod has quit IRC | 07:43 | |
| *** jackNemrod has joined #rdo | 07:44 | |
| *** rcernin has joined #rdo | 07:44 | |
| *** paragan has joined #rdo | 07:44 | |
| *** shardy has joined #rdo | 07:45 | |
| *** vnogin has joined #rdo | 07:46 | |
| rdogerrit | Juan Badia Payno created config master: Remove rubygem-psych https://review.rdoproject.org/r/10352 | 07:47 |
| *** shardy has joined #rdo | 07:49 | |
| *** gildub has quit IRC | 07:50 | |
| *** mosulica has joined #rdo | 07:50 | |
| *** jhershbe has joined #rdo | 07:55 | |
| *** aufi has joined #rdo | 07:58 | |
| *** ragiman has joined #rdo | 08:01 | |
| *** ykarel is now known as ykarel|lunch | 08:03 | |
| *** lkuchlan has quit IRC | 08:03 | |
| *** gkadam has joined #rdo | 08:05 | |
| adarazs|rover | jpena|off, amoralej|off: when you get in, please check what's up with zuul, see my previous messages above. | 08:07 |
| *** sbulage has joined #rdo | 08:13 | |
| *** tesseract has joined #rdo | 08:16 | |
| *** ccamacho has joined #rdo | 08:17 | |
| *** tshefi has joined #rdo | 08:22 | |
| *** josecastroleon has joined #rdo | 08:24 | |
| *** gfidente has joined #rdo | 08:24 | |
| *** gfidente has quit IRC | 08:24 | |
| *** gfidente has joined #rdo | 08:24 | |
| *** pgadiya has joined #rdo | 08:27 | |
| *** vnogin has quit IRC | 08:27 | |
| *** d0ugal has quit IRC | 08:29 | |
| *** d0ugal has joined #rdo | 08:29 | |
| number80 | adarazs|rover: likely on RDO cloud side | 08:30 |
| number80 | It looked like nodepool was starved but that's not the case | 08:31 |
| *** rbartal has joined #rdo | 08:31 | |
| *** rbartal_ has joined #rdo | 08:31 | |
| *** rbartal__ has joined #rdo | 08:31 | |
| *** gfidente has quit IRC | 08:32 | |
| *** jpena|off is now known as jpena | 08:33 | |
| *** gfidente has joined #rdo | 08:34 | |
| *** gfidente has quit IRC | 08:34 | |
| *** gfidente has joined #rdo | 08:34 | |
| jpena | adarazs|rover: the queue looks ok now | 08:35 |
| number80 | jpena: I restarted zuul that's why | 08:35 |
| adarazs|rover | :) | 08:35 |
| jpena | number80: ah ok | 08:35 |
| adarazs|rover | number80: thanks. | 08:35 |
| number80 | still has to clean the stalled nodes | 08:36 |
| *** hewbrocca_afk is now known as hewbrocca | 08:37 | |
| *** josecastroleon has quit IRC | 08:37 | |
| adarazs|rover | good timing, we're going to have a periodic run at 0900UTC | 08:41 |
| adarazs|rover | we really need that master promotion. | 08:41 |
| number80 | looks like nodepool is also stalled | 08:42 |
| *** ykarel|lunch is now known as ykarel | 08:42 | |
| number80 | instances were deleted according nova but nodepool still shows them ... | 08:42 |
| *** amoralej|off is now known as amoralej | 08:44 | |
| *** vnogin has joined #rdo | 08:44 | |
| *** jpich has joined #rdo | 08:44 | |
| amoralej | zuul queue has been lost on restart, right? | 08:45 |
| *** oidgar has joined #rdo | 08:46 | |
| amoralej | there is procedure to backup/restore the queue when restarting it | 08:46 |
| number80 | amoralej: nodes were completely unusable | 08:47 |
| jpena | number80: wait, you restarted zuul, not nodepool? | 08:47 |
| * jpena needs caffeine | 08:47 | |
| amoralej | number80, yeah, restarting is not a problem | 08:47 |
| number80 | jpena: both | 08:47 |
| jpena | number80: nodepool should've been enough | 08:48 |
| *** MrAbaddon has quit IRC | 08:48 | |
| amoralej | but doing the backup/restore of the queue jobs are re-launched without needing to recheck jobs | 08:48 |
| jpena | restarting zuul without https://softwarefactory-project.io/docs/faqs.html#how-to-restart-zuul-without-losing-builds-in-progress means we lose in-progress jobs | 08:48 |
| number80 | amoralej: if you have doc for restoring queue jobs, I'm taking it | 08:48 |
| *** vnogin has quit IRC | 08:48 | |
| number80 | jpena: all were completely stalled | 08:49 |
| number80 | some nodes were still working like an hour ago and stopped | 08:49 |
| number80 | I suspect again another storage issue on cloud | 08:49 |
| jpena | number80: yes, that's usually a nodepool issue. Soon after restarting it, it should start respawning instances | 08:49 |
| amoralej | number80, https://softwarefactory-project.io/docs/faqs.html#how-to-restart-zuul-without-losing-builds-in-progress for next time | 08:49 |
| jpena | anyway | 08:49 |
| jpena | let me have a look | 08:49 |
| *** gfidente has quit IRC | 08:49 | |
| number80 | amoralej: thanks :) | 08:49 |
| number80 | jpena: ack | 08:50 |
| amoralej | adarazs|rover, i'm crossing fingers for next periodic run | 08:50 |
| adarazs|rover | me too... | 08:50 |
| number80 | amoralej: do you know where I can send systemd unit for zuul? | 08:50 |
| number80 | This can be handled by systemd actually | 08:51 |
| amoralej | number80, that'd be nice | 08:52 |
| amoralej | maybe you can propose to add pre and post actions in systemd for zuul | 08:52 |
| number80 | tristanC: if you're still up ^ | 08:52 |
| number80 | amoralej: yep, exactly! | 08:52 |
| amoralej | i'm not sure if it's safe to do it always | 08:52 |
| amoralej | but sounds good | 08:52 |
| jpena | number80: https://softwarefactory-project.io/r/software-factory/zuul-distgit | 08:52 |
| number80 | jpena: thanks! | 08:53 |
| amoralej | btw, can't the queue be done persistent instead of in memory only? | 08:53 |
| * amoralej has no idea about zuul.... | 08:53 | |
| *** lkuchlan has joined #rdo | 08:54 | |
| amoralej | jpena, jobs are failint with NOT_REGISTERED | 08:54 |
| jpena | amoralej: give me an example to test | 08:55 |
| jpena | I'm cleaning up nodepool vms | 08:55 |
| number80 | https://review.rdoproject.org/r/#/c/10263/4 | 08:55 |
| number80 | ah difference is that you're using --now | 08:56 |
| jpena | ok, zuul seems to be in a better shape now | 08:57 |
| number80 | Yeah, jobs are running again | 08:58 |
| amoralej | jpena, look at https://review.openstack.org/#/c/512239/ | 08:59 |
| *** vnogin has joined #rdo | 09:00 | |
| jpena | amoralej: yes, we'll have to recheck all commits with issues | 09:00 |
| *** flepied has quit IRC | 09:01 | |
| rdogerrit | Alfredo Moralejo proposed rdo-infra/puppet-dlrn master: Switch deps repo for master builder https://review.rdoproject.org/r/10342 | 09:03 |
| number80 | https://softwarefactory-project.io/r/#/c/10052/ | 09:06 |
| number80 | ^ zuul-server-safe unit | 09:06 |
| tristanC | number80: i'm not entirely sure this is safe to do automatically, it's more of on op thing to do on purpose | 09:08 |
| rdogerrit | rdo-trunk created openstack/congress-distgit rpm-master: openstack-congress: failed to build 5526f76 https://review.rdoproject.org/r/10353 | 09:08 |
| number80 | tristanC: that's why it's a separate unit | 09:08 |
| oanson | number80, hi. Did you have a chance to look at https://bugzilla.redhat.com/show_bug.cgi?id=1494979 ? | 09:08 |
| openstack | bugzilla.redhat.com bug 1494979 in Package Review "Review Request: python-jsonmodels - Models to make easier to deal with structures that are converted to, or read from JSON in python" [Medium,Assigned] - Assigned to karlthered | 09:08 |
| number80 | oanson: not yet, I came back from PTO y-day | 09:08 |
| oanson | Sorry, don't mean to rush. I just want to make sure it isn't forgotten :) | 09:09 |
| rdogerrit | Merged rdo-infra/puppet-dlrn master: Switch deps repo for master builder https://review.rdoproject.org/r/10342 | 09:09 |
| oanson | Hope you enjoyed your PTO. | 09:09 |
| amoralej | number80, ykarel jpena i've applied https://review.rdoproject.org/r/#/c/10342 manually | 09:09 |
| number80 | tristanC: alternative, it could be a separate script rather than a unit | 09:10 |
| tristanC | number80: hum, interesting... but is it ok to have two unit managing a single process? | 09:10 |
| amoralej | let me know if you see some related error | 09:10 |
| number80 | amoralej: ack | 09:10 |
| ykarel | amoralej, ack | 09:10 |
| number80 | tristanC: good question | 09:10 |
| number80 | I can't try that on prod | 09:11 |
| rdobot | [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-master-current @ http://tinyurl.com/y8hxewm2 |#| Build failure on centos7-master/current: neutron-vpnaas, congress: http://trunk.rdoproject.org/centos7-master/report.html | 09:11 |
| *** gfidente has joined #rdo | 09:13 | |
| *** mcornea has joined #rdo | 09:14 | |
| *** vnogin has quit IRC | 09:15 | |
| *** martinkopec has joined #rdo | 09:15 | |
| tristanC | number80: perhaps adding a "restart-service" action to the zuul-changes.py script would do the trick | 09:15 |
| tristanC | number80: though not sure that's worth the effort, iirc this behavior is planned in stock zuul-v3 | 09:16 |
| *** vnogin has joined #rdo | 09:17 | |
| *** lucas-afk is now known as lucasagomes | 09:19 | |
| *** aufi has quit IRC | 09:20 | |
| rdogerrit | Alfredo Moralejo proposed openstack/neutron-vpnaas-distgit rpm-master: Remove neutron-vpn-agent binary and service https://review.rdoproject.org/r/10348 | 09:22 |
| amoralej | that will need changes in deployment tools ^ | 09:23 |
| amoralej | ajo, ^ does someones takes care of vpnaas? | 09:24 |
| *** mvk has quit IRC | 09:26 | |
| *** jistr|mtgs is now known as jistr | 09:29 | |
| *** Son_Goku has joined #rdo | 09:32 | |
| *** jlibosva has joined #rdo | 09:32 | |
| number80 | tristanC: well, this is to be discussed, atm, I changed the change to suggest a separate script inspired from how postgres handle specific actions | 09:32 |
| *** DV has quit IRC | 09:33 | |
| rdogerrit | Alfredo Moralejo proposed openstack/congress-distgit rpm-master: Remove duplicated config files under /usr/etc/congress https://review.rdoproject.org/r/10351 | 09:34 |
| number80 | I'm just trying to find idiotproof ways to use zuul when needed :) <== from the idiot :) | 09:34 |
| *** aufi has joined #rdo | 09:34 | |
| *** liujiong_lj has joined #rdo | 09:35 | |
| *** jatanmalde has quit IRC | 09:35 | |
| *** liujiong has quit IRC | 09:35 | |
| *** wolverineav has joined #rdo | 09:38 | |
| *** paragan has quit IRC | 09:38 | |
| *** ushkalim has joined #rdo | 09:39 | |
| amoralej | adarazs|rover, periodic still failing creating the image | 09:41 |
| amoralej | i'll check the logs | 09:41 |
| *** shardy has quit IRC | 09:42 | |
| *** shardy has joined #rdo | 09:42 | |
| adarazs|rover | amoralej: what? same error?? | 09:42 |
| adarazs|rover | 2017-10-31 09:31:04 | 2017-10-31 09:31:04.062 | Error: python-neutron-lbaas conflicts with 1:python-neutron-12.0.0-0.20171030214124.76ec5ca.el7.centos.noarch 2017-10-31 09:31:04 | 2017-10-31 09:31:04.062 | Error: openstack-neutron-lbaas conflicts with 1:openstack-neutron-12.0.0-0.20171030214124.76ec5ca.el7.centos.noarch | 09:42 |
| adarazs|rover | https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/5a8d998/undercloud/home/jenkins/overcloud_image_build.log.txt.gz#_2017-10-31_09_31_04 | 09:43 |
| amoralej | adarazs|rover, let me check the repo config | 09:43 |
| rdogerrit | Javier Peña proposed openstack/congress-distgit rpm-master: Remove duplicated config files under /usr/etc/congress https://review.rdoproject.org/r/10351 | 09:43 |
| *** vnogin has quit IRC | 09:45 | |
| *** jatanmalde has joined #rdo | 09:46 | |
| *** wolverineav has quit IRC | 09:46 | |
| *** vnogin has joined #rdo | 09:47 | |
| *** wolverineav has joined #rdo | 09:47 | |
| *** owalsh- is now known as owalsh | 09:47 | |
| *** julim_ has joined #rdo | 09:49 | |
| amoralej | adarazs|rover, jobs downloaded dependencies repo from http://mirror.dfw.rax.openstack.org:8080/rdo/centos7-master/delorean-deps.repo | 09:50 |
| amoralej | at 09:08 | 09:51 |
| amoralej | i updated https://trunk.rdoproject.org/centos7-master/delorean-deps.repo at 09:08 | 09:51 |
| *** wolverineav has quit IRC | 09:51 | |
| amoralej | so it probably still got old one | 09:51 |
| adarazs|rover | :( | 09:52 |
| *** julim has quit IRC | 09:52 | |
| adarazs|rover | oh well, we can check again at 1400UTC :( | 09:52 |
| *** pgadiya has quit IRC | 09:52 | |
| adarazs|rover | any week now and we'll have a promotion. | 09:53 |
| amoralej | adarazs|rover, we could abort all jobs in the pipeline and change pipeline schedule.. | 09:53 |
| amoralej | i'm not sure how clean abort jobs is | 09:53 |
| adarazs|rover | amoralej: it might be worth it at this point. | 09:53 |
| *** MrAbaddon has joined #rdo | 09:53 | |
| *** mandre has quit IRC | 09:55 | |
| rdogerrit | Yatin Karel proposed openstack/neutron-vpnaas-distgit rpm-master: Remove neutron-vpn-agent service and binary from package https://review.rdoproject.org/r/10349 | 09:55 |
| *** mandre_ has joined #rdo | 09:56 | |
| *** mandre_ is now known as mandre | 09:56 | |
| *** mvk has joined #rdo | 09:56 | |
| amoralej | adarazs|rover, aborting jobs | 09:56 |
| *** brault has quit IRC | 09:56 | |
| amoralej | adarazs|rover, i need to abort pike ones also | 09:57 |
| adarazs|rover | well, not much is lost at this point. | 09:57 |
| adarazs|rover | amoralej: can't we just trigger that pipeline btw? | 09:57 |
| amoralej | not afaik | 09:57 |
| adarazs|rover | instead of changing the timing? | 09:57 |
| adarazs|rover | damn. | 09:57 |
| amoralej | tristanC, jpena is there any way to manually trigger a periodic pipeline other that changing schedule? | 09:58 |
| jpena | amoralej: I read someone it was possible, but very cumbersome and not worth it | 09:58 |
| jpena | s/someone/somewhere/ | 09:58 |
| *** leseb_ has joined #rdo | 10:00 | |
| *** rickflare has joined #rdo | 10:00 | |
| amoralej | adarazs|rover, do you know if aborting jobs is clean for ovb? | 10:00 |
| *** higgins` has joined #rdo | 10:00 | |
| *** jruzicka_ has joined #rdo | 10:00 | |
| *** jistr_ has joined #rdo | 10:00 | |
| amoralej | multinodes is safe because nodepool will remove nodes | 10:00 |
| *** agurenko_ has joined #rdo | 10:00 | |
| amoralej | but i'm not sure about ovb | 10:00 |
| *** leseb has quit IRC | 10:01 | |
| *** higgins has quit IRC | 10:01 | |
| *** jistr has quit IRC | 10:01 | |
| *** jschlueter has quit IRC | 10:01 | |
| *** agurenko has quit IRC | 10:01 | |
| *** thrash|g0ne has quit IRC | 10:01 | |
| *** jruzicka has quit IRC | 10:01 | |
| *** apevec has joined #rdo | 10:01 | |
| *** rwsu has quit IRC | 10:01 | |
| *** hrw has quit IRC | 10:01 | |
| *** zerick has quit IRC | 10:01 | |
| *** migi has quit IRC | 10:01 | |
| *** rickflare2 has quit IRC | 10:01 | |
| *** apevec has quit IRC | 10:01 | |
| *** apevec has joined #rdo | 10:01 | |
| *** hrww has joined #rdo | 10:01 | |
| *** zerick has joined #rdo | 10:01 | |
| *** hrww is now known as hrw | 10:01 | |
| *** rwsu has joined #rdo | 10:02 | |
| *** jruzicka_ is now known as jruzicka | 10:02 | |
| adarazs|rover | amoralej: I'm not sure if something will clean things up or not :/ | 10:02 |
| adarazs|rover | let me check | 10:02 |
| amoralej | adarazs|rover, i've just aborted https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024-pike/281/console | 10:02 |
| *** migi has joined #rdo | 10:02 | |
| *** thrash has joined #rdo | 10:03 | |
| *** thrash has quit IRC | 10:03 | |
| *** thrash has joined #rdo | 10:03 | |
| *** bitchecker has quit IRC | 10:03 | |
| *** ryansb has quit IRC | 10:03 | |
| *** Jeffrey4l has quit IRC | 10:04 | |
| *** zoli has quit IRC | 10:04 | |
| *** jlibosva1 has joined #rdo | 10:04 | |
| *** jlibosva is now known as Guest55036 | 10:04 | |
| *** jlibosva1 is now known as jlibosva | 10:04 | |
| *** gfidente has quit IRC | 10:05 | |
| *** pgadiya has joined #rdo | 10:06 | |
| rdogerrit | Alfredo Moralejo created config master: Temporarily change timing for openstack-periodic https://review.rdoproject.org/r/10354 | 10:06 |
| amoralej | adarazs|rover, jpena ^ | 10:06 |
| *** bitchecker has joined #rdo | 10:06 | |
| *** pblaho has joined #rdo | 10:06 | |
| *** zoli has joined #rdo | 10:06 | |
| *** jatanmalde has quit IRC | 10:07 | |
| adarazs|rover | amoralej: so I thin the OVB nodes got cleaned up. | 10:07 |
| *** ryansb has joined #rdo | 10:07 | |
| *** ryansb has joined #rdo | 10:07 | |
| *** Jeffrey4l has joined #rdo | 10:07 | |
| amoralej | adarazs|rover, cool | 10:07 |
| adarazs|rover | amoralej: I tried to look for "name": "baremetal-4497-0", in our tenant | 10:07 |
| rdogerrit | Merged config master: Temporarily change timing for openstack-periodic https://review.rdoproject.org/r/10354 | 10:07 |
| adarazs|rover | or anything with 4497 in its name | 10:08 |
| adarazs|rover | and there was nothing. | 10:08 |
| *** iranzo has quit IRC | 10:08 | |
| *** gfidente has joined #rdo | 10:08 | |
| *** gfidente has quit IRC | 10:08 | |
| *** gfidente has joined #rdo | 10:08 | |
| adarazs|rover | and I think that's the instance that the te-broker created | 10:08 |
| amoralej | adarazs|rover, it uses heat to deploy ovb, right? | 10:09 |
| *** pblaho has quit IRC | 10:09 | |
| amoralej | there should be something there | 10:09 |
| *** brault has joined #rdo | 10:09 | |
| *** pblaho has joined #rdo | 10:09 | |
| adarazs|rover | amoralej: yep. that heat stack is gone | 10:09 |
| amoralej | then i guess it's ok | 10:09 |
| adarazs|rover | it should be baremetal-4497 and it's not in the list :) | 10:09 |
| *** agurenko_ is now known as agurenko | 10:10 | |
| amoralej | adarazs|rover, if i didn't mess it up, jobs will start in 5 minutes | 10:10 |
| adarazs|rover | yep \o/ | 10:11 |
| adarazs|rover | thanks for hacking it :) | 10:11 |
| *** ratailor has quit IRC | 10:11 | |
| amoralej | hopefully there are nothing else breaking us | 10:11 |
| tristanC | amoralej: it should be possible to trigger a periodic buildset using "zuul enqueue-ref" command | 10:12 |
| adarazs|rover | that sounds useful. let's try it next time. | 10:13 |
| *** gfidente has quit IRC | 10:13 | |
| *** gfidente has joined #rdo | 10:16 | |
| *** gfidente has quit IRC | 10:16 | |
| *** gfidente has joined #rdo | 10:16 | |
| *** liujiong_lj has quit IRC | 10:18 | |
| *** mosulica has quit IRC | 10:18 | |
| adarazs|rover | amoralej: I'm going to relocate to the office meanwhile and hope for the best :) | 10:18 |
| amoralej | adarazs|rover, ok, i'll keep an eye on the jobs | 10:19 |
| *** nstrug has joined #rdo | 10:21 | |
| *** jhershbe has quit IRC | 10:22 | |
| *** jatanmalde has joined #rdo | 10:23 | |
| *** pgadiya has quit IRC | 10:24 | |
| *** brault has quit IRC | 10:24 | |
| rdogerrit | Merged rdoinfo master: Promote CBS tags update for ocata-testing https://review.rdoproject.org/r/10263 | 10:26 |
| *** udesale has quit IRC | 10:26 | |
| *** nstrug has quit IRC | 10:26 | |
| *** iranzo has joined #rdo | 10:27 | |
| number80 | Looks like there's an exceptional holiday in Germany | 10:27 |
| rdogerrit | Alfredo Moralejo proposed rdoinfo master: Promote CBS tags update for pike-testing https://review.rdoproject.org/r/10264 | 10:28 |
| *** gfidente has quit IRC | 10:29 | |
| *** mvk has quit IRC | 10:30 | |
| *** gfidente has joined #rdo | 10:32 | |
| *** gfidente has quit IRC | 10:32 | |
| *** gfidente has joined #rdo | 10:32 | |
| *** aufi has quit IRC | 10:32 | |
| *** jhershbe has joined #rdo | 10:32 | |
| *** paragan has joined #rdo | 10:34 | |
| *** jschlueter|znc has joined #rdo | 10:35 | |
| *** brault has joined #rdo | 10:38 | |
| *** pgadiya has joined #rdo | 10:38 | |
| *** mvk has joined #rdo | 10:43 | |
| *** brault has quit IRC | 10:43 | |
| *** nstrug has joined #rdo | 10:46 | |
| *** aufi has joined #rdo | 10:46 | |
| rdogerrit | Merged config master: Use RDO dev user/email for updating changelog https://review.rdoproject.org/r/10350 | 10:47 |
| *** mosulica has joined #rdo | 10:49 | |
| *** nstrug has quit IRC | 10:51 | |
| *** jhershbe has quit IRC | 10:56 | |
| *** jhershbe has joined #rdo | 10:56 | |
| *** atoth has joined #rdo | 11:08 | |
| adarazs|rover | amoralej: wooo, no failure yet! | 11:10 |
| amoralej | yeah | 11:10 |
| adarazs|rover | let's see what else can fudge up. :) | 11:10 |
| amoralej | i think at least we passed previous issue | 11:11 |
| amoralej | let's see now | 11:11 |
| amoralej | adarazs|rover, wrt the issue pushing container images | 11:11 |
| amoralej | did you find something? | 11:11 |
| adarazs|rover | amoralej: no. there's no good output on that job apart from it failing, so no idea what is the real problem... timeout or what. | 11:12 |
| amoralej | the strange thing is that it looks like it's failing to push to localhost | 11:13 |
| adarazs|rover | if we have such a low disk read/write speed, it might as well just time out on that... | 11:14 |
| amoralej | adarazs|rover, https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-centos-7-master-containers-build/aa325ac/kolla/logs/redis-sentinel.log | 11:14 |
| amoralej | ReadTimeoutError: UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. | 11:14 |
| amoralej | yeah, maybe | 11:14 |
| amoralej | adarazs|rover, image build succeeded in https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/282/console | 11:16 |
| amoralej | \o/ | 11:16 |
| adarazs|rover | yaaaay \o/ | 11:16 |
| *** rcernin has quit IRC | 11:18 | |
| adarazs|rover | amoralej: but yeah, now we can root for the container builder jobs to not mess up :) | 11:22 |
| rdogerrit | Alfredo Moralejo created rdo-infra/puppet-dlrn master: Use latest symlink for queens-testing deps repo https://review.rdoproject.org/r/10355 | 11:24 |
| adarazs|rover | amoralej: it took 2h10m for the last successful master container build job to go over the "Build and push images" stage. | 11:24 |
| adarazs|rover | and we're only 30m in. | 11:24 |
| *** panda|ruck|off is now known as panda|ruck | 11:24 | |
| amoralej | yes, let's cross fingers | 11:24 |
| adarazs|rover | panda|ruck: good news! | 11:24 |
| panda|ruck | adarazs|rover: no promotion, no good news | 11:25 |
| *** agurenko is now known as agurenko_ | 11:25 | |
| panda|ruck | :) | 11:25 |
| adarazs|rover | panda|ruck: limited good news. | 11:25 |
| panda|ruck | medium news | 11:26 |
| adarazs|rover | hehe | 11:26 |
| panda|ruck | I monitored the chat for a while, zuul is ok now, and the image building succeded | 11:26 |
| panda|ruck | pike phase2 is stll busted though | 11:26 |
| adarazs|rover | panda|ruck: yeah, we need a promotion to really fix things I think. | 11:27 |
| *** ratailor has joined #rdo | 11:27 | |
| *** gfidente has quit IRC | 11:27 | |
| adarazs|rover | panda|ruck: we don't have a good image for that. David made a symlink for the previous one, but I'm not sure it's good enough. | 11:27 |
| *** wolverineav has joined #rdo | 11:29 | |
| *** gfidente has joined #rdo | 11:30 | |
| rdogerrit | Merged rdo-infra/puppet-dlrn master: Use latest symlink for queens-testing deps repo https://review.rdoproject.org/r/10355 | 11:31 |
| rdogerrit | Merged openstack/neutron-vpnaas-distgit rpm-master: Remove neutron-vpn-agent service and binary from package https://review.rdoproject.org/r/10349 | 11:32 |
| *** nstrug has joined #rdo | 11:36 | |
| *** noslzzp has joined #rdo | 11:38 | |
| apevec | number80, it is indeed exceptional Reformation day for 500. anniversary https://de.wikipedia.org/wiki/Reformationstag#Gesetzliche_Stellung_als_Feiertag_oder_schulfreier_Tag | 11:39 |
| rdogerrit | Javier Peña proposed rdoinfo master: Pin branchless projects for stable releases https://review.rdoproject.org/r/10328 | 11:40 |
| *** agurenko_ is now known as agurenko | 11:41 | |
| number80 | apevec: shows maturity after all those religious war | 11:41 |
| apevec | yeah takes only 500 years... | 11:41 |
| *** nstrug has quit IRC | 11:41 | |
| *** nstrug has joined #rdo | 11:42 | |
| amoralej | adarazs|rover, containers failed https://review.rdoproject.org/jenkins/job/periodic-tripleo-centos-7-master-containers-build/244/console | 11:43 |
| amoralej | looks like legit issue | 11:47 |
| apevec | read timeouts? | 11:47 |
| amoralej | gimme a while | 11:47 |
| amoralej | good point is that ovb jobs are progressing well | 11:47 |
| apevec | ERROR:kolla.image.build.sensu-client:Unknown error when pushing | 11:48 |
| amoralej | apevec, it seems deps error | 11:48 |
| *** iranzo has quit IRC | 11:48 | |
| amoralej | https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-centos-7-master-containers-build/c17d3a1/kolla/logs/openstack-base.log | 11:48 |
| amoralej | No package python2-cffi available | 11:48 |
| amoralej | INFO:kolla.image.build.openstack-base:1 packages excluded due to repository priority protections | 11:48 |
| amoralej | INFO:kolla.image.build.openstack-base:Package libyaml-0.1.4-11.el7_0.x86_64 already installed and latest version | 11:48 |
| amoralej | INFO:kolla.image.build.openstack-base:Package pyOpenSSL is obsoleted by python2-pyOpenSSL, trying to install python2-pyOpenSSL-16.2.0-3.el7.noarch instead | 11:48 |
| amoralej | INFO:kolla.image.build.openstack-base:No package python2-cffi available. | 11:48 |
| amoralej | INFO:kolla.image.build.openstack-base:[91mError: Not tolerating missing names on install, stopping. | 11:48 |
| apevec | from pyopenssl | 11:48 |
| *** brault has joined #rdo | 11:49 | |
| apevec | indeed http://cbs.centos.org/koji/buildinfo?buildID=10145 is not tagged for queens | 11:49 |
| *** pchavva has joined #rdo | 11:49 | |
| amoralej | we didn't add python-cffi to queens-testing | 11:49 |
| amoralej | :( | 11:49 |
| *** iranzo has joined #rdo | 11:50 | |
| *** iranzo has quit IRC | 11:50 | |
| *** iranzo has joined #rdo | 11:50 | |
| amoralej | let me check if it was in the blacklist for bootstraping or what | 11:50 |
| apevec | repoclosure didn't report it? | 11:50 |
| *** ratailor has quit IRC | 11:50 | |
| apevec | panda|ruck, adarazs|rover, so now with dlrnapi reporting, we should be able to re-run just this containers job? | 11:50 |
| *** pgadiya has quit IRC | 11:51 | |
| amoralej | maybe we can | 11:51 |
| amoralej | using sfmanager | 11:51 |
| *** egonzalez has quit IRC | 11:51 | |
| *** egonzalez has joined #rdo | 11:51 | |
| panda|ruck | apevec: we are not able to run a single job | 11:51 |
| apevec | so why not? that was the whole point ... | 11:52 |
| apevec | vs re-running the whole pipeline like before | 11:52 |
| *** jprovazn has quit IRC | 11:52 | |
| *** brault has quit IRC | 11:53 | |
| amoralej | apevec, according to https://review.rdoproject.org/etherpad/p/pike-testing-bootstrap we didn't need to update python-cffi | 11:53 |
| amoralej | there must be a newer package in base | 11:53 |
| amoralej | oh | 11:54 |
| *** udesale has joined #rdo | 11:54 | |
| amoralej | it searchs for python2-cffi instead of python-cffi | 11:54 |
| panda|ruck | apevec: if we rebuild the containers, we'll have to test them anyway, runing the other jobs too | 11:54 |
| number80 | again? | 11:54 |
| apevec | panda|ruck, right, whole dep-tree of jobs | 11:54 |
| *** jprovazn has joined #rdo | 11:55 | |
| apevec | but if that's almost == the whole set, yeah not much point | 11:55 |
| amoralej | does kolla doesn't support provides? | 11:55 |
| amoralej | http://git.openstack.org/cgit/openstack/kolla/tree/docker/openstack-base/Dockerfile.j2 | 11:55 |
| amoralej | we must be breaking kolla jobs in gate changing the repos | 11:55 |
| apevec | amoralej, python-cffi 1.6.0-5.el7 in base doesn't have python2-* | 11:55 |
| amoralej | yes | 11:56 |
| apevec | but wth they have python2-* in the dockerfile?! | 11:56 |
| apevec | and wth is it an explicit dep?! | 11:56 |
| rdogerrit | Yatin Karel proposed gating_scripts master: List packages which are not installed if rpm -q $LIST fails https://review.rdoproject.org/r/10298 | 11:56 |
| apevec | can we fix that in ooo override? | 11:57 |
| apevec | this openstack_base_packages is just bad | 11:57 |
| amoralej | override was in tripleo-common iirc | 11:57 |
| apevec | it should be left to packages to list their deps... | 11:57 |
| apevec | damn containers | 11:57 |
| amoralej | maybe kolla do a yum list so it doesn't work with "provides"? | 11:58 |
| apevec | if it does, that's even worse | 11:58 |
| amoralej | no | 11:58 |
| panda|ruck | apevec: we decided we didin't need manual triggers if we were running the periodic pipeline frequently enough. Maybe we 'll have to rediscuss this decision at least for master. | 11:58 |
| amoralej | it does yum install | 11:58 |
| amoralej | according to logs | 11:58 |
| amoralej | https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-centos-7-master-containers-build/c17d3a1/kolla/logs/openstack-base.log | 11:58 |
| apevec | panda|ruck, it's about re-running single jobs which might have failed due to e.g. infra | 11:59 |
| apevec | amoralej, right, so fix is to just remove explicit python2-cffi from that list | 11:59 |
| amoralej | apevec, i think the do a base openstack-base with only libraries | 11:59 |
| amoralej | for reusability | 11:59 |
| apevec | and let pyopenssl pull it in | 11:59 |
| apevec | amoralej, again, no point, RPMs need to have their deps | 11:59 |
| panda|ruck | apevec: I'll add it as agenda item for the next CI meeting | 12:00 |
| apevec | it would make sense for "weak" deps for opt. functionality | 12:00 |
| apevec | but that's not the case, it's base libs! | 12:00 |
| *** dougbtv has joined #rdo | 12:00 | |
| amoralej | apevec, i think cffi message is unrelated to the pyopenssl one | 12:00 |
| amoralej | anyway, maybe we could add a kolla job to gate changes in deps | 12:01 |
| apevec | amoralej, it's pyopenssl -> python-cryptography -> python-cffi | 12:02 |
| apevec | amoralej, which kolla job? just to build all? | 12:02 |
| amoralej | btw python-cryptography is not in queens-testing | 12:02 |
| apevec | we don't build all | 12:02 |
| *** abregman has quit IRC | 12:03 | |
| amoralej | so maybe we have the same issue | 12:03 |
| apevec | it's in base | 12:03 |
| amoralej | apevec, i think best option is to just send a patch 's/python2-/python-/g' | 12:03 |
| amoralej | on that file | 12:03 |
| apevec | but strangly enough, it's python2-cryptography (with correct provides) | 12:03 |
| amoralej | wdyt | 12:03 |
| amoralej | apevec, in base? | 12:03 |
| apevec | I'd rather remove it | 12:03 |
| apevec | amoralej, yes in base | 12:03 |
| apevec | sometimes RHEL is not quite consistent wrt their own rules... | 12:04 |
| amoralej | right | 12:04 |
| apevec | I just don't see justification for explicit cffi in that kolla list | 12:04 |
| amoralej | my concern is that it may be not only cffi | 12:04 |
| *** paragan has quit IRC | 12:05 | |
| amoralej | although i guess we would have it in logs | 12:05 |
| apevec | yes, all of that list is suspect as unecessary | 12:05 |
| apevec | amoralej, ok, since it should be less resistence and faster merge, let's go with python-cffi in kolla dockerfile | 12:06 |
| *** gszasz has joined #rdo | 12:06 | |
| apevec | wdyt which one would merge faster kolla or override in tripleo-common? | 12:06 |
| apevec | amoralej, also please open LP and escalate it | 12:07 |
| honza | apevec: amoralej: adarazs|rover: and others: thanks for your hard work on the queens deps patch yesterday! much appreciated | 12:07 |
| apevec | since this is now a blocker :( | 12:07 |
| amoralej | apevec, ok | 12:07 |
| *** shardy is now known as shardy_lunch | 12:07 | |
| *** julim_ has quit IRC | 12:07 | |
| dmsimard | apevec: there's nothing preventing us from running "adhoc" voting jobs against a previous hash to "retry" part of a promotion that failed. It just wouldn't be in Zuul. It could be in ci.centos for example. | 12:08 |
| rdobot | [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-master-current @ http://tinyurl.com/y8hxewm2 |#| Build failure on centos7-master/current: congress: http://trunk.rdoproject.org/centos7-master/report.html | 12:11 |
| *** dprince has joined #rdo | 12:13 | |
| amoralej | apevec https://review.openstack.org/#/c/516643/ | 12:13 |
| *** jayg|g0n3 is now known as jayg | 12:16 | |
| *** agurenko is now known as agurenko_ | 12:17 | |
| *** paragan has joined #rdo | 12:21 | |
| *** martinkopec has quit IRC | 12:24 | |
| *** martinkopec has joined #rdo | 12:25 | |
| *** flepied has joined #rdo | 12:27 | |
| *** rhallisey has joined #rdo | 12:28 | |
| apevec | mandre, please review https://review.openstack.org/516643 | 12:29 |
| *** jackNemrod has quit IRC | 12:30 | |
| rdogerrit | Merged openstack/congress-distgit rpm-master: Remove duplicated config files under /usr/etc/congress https://review.rdoproject.org/r/10351 | 12:30 |
| *** vnogin has quit IRC | 12:31 | |
| *** catintheroof has joined #rdo | 12:31 | |
| *** dustins has joined #rdo | 12:31 | |
| *** catintheroof has quit IRC | 12:32 | |
| *** catintheroof has joined #rdo | 12:32 | |
| *** vnogin has joined #rdo | 12:32 | |
| *** morazi has joined #rdo | 12:32 | |
| *** rlandy has joined #rdo | 12:34 | |
| *** trown|outtypewww is now known as trown | 12:34 | |
| adarazs|rover | oh, the container build failed, that's not good for my digestion after lunch :/ | 12:35 |
| *** vnogin has quit IRC | 12:36 | |
| mandre | apevec: ack, /me looks | 12:37 |
| *** dneary has joined #rdo | 12:37 | |
| *** gkadam has quit IRC | 12:37 | |
| mandre | apevec: what about all the other python2-* packages listed above? | 12:38 |
| *** jmlowe_ has joined #rdo | 12:38 | |
| mandre | amoralej: ^^ | 12:38 |
| apevec | mandre, yeah, they all need closer look, but for now this one is a blocker | 12:38 |
| amoralej | mandre, i just did minimal change to unblock | 12:39 |
| apevec | we'll have a followup, this is a quickfix | 12:39 |
| mandre | apevec: ack | 12:39 |
| *** jcoufal has joined #rdo | 12:39 | |
| amoralej | mandre, thx for fixing commit msg | 12:39 |
| *** jmlowe has quit IRC | 12:40 | |
| amoralej | adarazs|rover, apevec good news is that ovb jobs are passing | 12:41 |
| adarazs|rover | yes, we have to count our blessings. | 12:42 |
| amoralej | but change in vpnaas may break things again | 12:42 |
| apevec | is vpnaas configured in ooo ? | 12:42 |
| amoralej | adarazs|rover, do you know if vpnaas is deployed in any tripleo scenario? | 12:42 |
| adarazs|rover | amoralej: no, but I can check.. | 12:42 |
| *** scorcoran_afk has joined #rdo | 12:43 | |
| amoralej | i think vpnaas is not deployed in p-o-i | 12:43 |
| *** tosky has joined #rdo | 12:44 | |
| adarazs|rover | amoralej: I don't see it in any of the scenario files here: https://github.com/openstack/tripleo-heat-templates/tree/master/ci/environments | 12:44 |
| amoralej | ack | 12:44 |
| dmsimard | adarazs|rover, panda|ruck: fyi replied to https://lists.rdoproject.org/pipermail/dev/2017-October/008373.html | 12:45 |
| *** jmelvin has joined #rdo | 12:45 | |
| dmsimard | please let me know if I should schedule a maintenance to increase the storage, we keep going above 85% usage with 500GB. I can bump to 750GB. | 12:45 |
| rdobot | [sensu] RESOLVED: master.monitoring.rdoproject.org - check-delorean-master-current @ http://tinyurl.com/y8hxewm2 |#| No build failures detected: http://trunk.rdoproject.org/centos7-master/report.html | 12:46 |
| adarazs|rover | dmsimard: I'll talk about it with panda|ruck in a few minutes and get back to you. | 12:46 |
| adarazs|rover | dmsimard: I think bumping it up a bit can't hurt though. | 12:47 |
| adarazs|rover | :) | 12:47 |
| *** lucasagomes is now known as lucas-hungry | 12:47 | |
| *** rbowen has joined #rdo | 12:47 | |
| *** ChanServ sets mode: +o rbowen | 12:47 | |
| dmsimard | adarazs|rover: sure, but weshay|PTO was like "eh but we can keep just like 3 images" or something | 12:47 |
| *** toure_biab is now known as toure | 12:47 | |
| adarazs|rover | dmsimard: hmm, 3 might be a bit of stretch, but it's true we don't need this many. | 12:47 |
| dmsimard | adarazs|rover: so I wanted to wait and see, once we bump the volume up we can't shrink it so we'd be wasting that space | 12:47 |
| adarazs|rover | dmsimard: I can make a script that will spit out the last few images that we need to keep, would that help? | 12:48 |
| adarazs|rover | dmsimard: query DLRN API for the history and based on that select the ones we need. | 12:48 |
| panda|ruck | dmsimard: one thing that I didn't understand in your first email was if you wanted to create the cron script, or if just preferred to host it locally | 12:48 |
| *** udesale has quit IRC | 12:49 | |
| dmsimard | adarazs|rover: I'm not sure a large amount of logic is required, we can probably just rotate N images out, excluding images that are symlinked and have a dumb cron run thast | 12:49 |
| *** vnogin has joined #rdo | 12:49 | |
| *** udesale has joined #rdo | 12:49 | |
| *** shardy_lunch is now known as shardy | 12:49 | |
| adarazs|rover | dmsimard: I'm not sure. I want to keep the last two images, let's say, for each link. so we have something to revert to if the latest turns out to be bad. and let's say phase2 promotes rarely enough that the last image there is 2 week old, the second one is 4 week. | 12:50 |
| adarazs|rover | dmsimard: that would mean to keep *all* the images for 4 weeks? doesn't sound reasonable. | 12:51 |
| *** jpena is now known as jpena|lunch | 12:51 | |
| dmsimard | oh you want to keep like N images per symlink ? | 12:51 |
| amoralej | mandre, btw, i've seen we are still building mongodb container. We are planning to retire mongodb from RDO as it's not longer used by any service | 12:51 |
| dmsimard | so for example you would have 9 images at any given time for 3 symlinks | 12:52 |
| adarazs|rover | dmsimard: that was something in the back of my mind, it would be prudent to do it that way I think. | 12:52 |
| adarazs|rover | dmsimard: well, potentially less if some point to the same hash, or the history is the same, etc. | 12:52 |
| panda|ruck | when the symlink point to the same hah, it would be just three | 12:52 |
| mandre | amoralej: let me just double check we've stopped using mongodb in pike | 12:53 |
| dmsimard | also, I was curious.. is the pike image really 6.9GB relative to 2.8GB in newton ? that's 4GB worth of data, that's a lot.. what's on there ? Shouldn't it just be packages and config ? | 12:53 |
| amoralej | mandre, we'll remove only in queens, it'll stay in pike | 12:53 |
| adarazs|rover | dmsimard: there's something wrong with how we build the undercloud image I think. that's what I wanted to talk about with panda|ruck. I remembet trown saying we need to shrink the undercloud image or somehting. | 12:53 |
| panda|ruck | dmsimawith virt-sparsify | 12:54 |
| adarazs|rover | I don't think it's reasonable that it got so big suddenly. | 12:54 |
| mandre | amoralej: then go for it! | 12:54 |
| amoralej | but it's still building it in gate jobs for master | 12:54 |
| amoralej | queens == current master | 12:54 |
| dmsimard | adarazs|rover, panda|ruck: I don't know what it is, but it sounds like an obvious bloat that's impacting storage and download/upload durations | 12:55 |
| dmsimard | master is 7.0GB | 12:56 |
| *** udesale has quit IRC | 12:56 | |
| mandre | amoralej: I see we still have the zaqar environment file in tripleo-heat-templates that references mongodb but I think it's a bug, I'll fix it on my side | 12:56 |
| dmsimard | ocata is 6.9GB too | 12:56 |
| *** udesale has joined #rdo | 12:56 | |
| amoralej | mandre, i think zaqar was moved to swift iirc | 12:56 |
| *** gfidente has quit IRC | 12:56 | |
| *** mandre is now known as mandre_afk | 12:56 | |
| dmsimard | adarazs|rover, panda|ruck: so I mean, it's maybe no wonder we're running out of space on the image server if we're using 4GB extra *per* image | 12:56 |
| amoralej | mandre_afk, it's still to be merged https://review.openstack.org/#/c/514727/ | 12:57 |
| adarazs|rover | yeah... | 12:57 |
| mandre_afk | amoralej: exact, looks like the environment file was left with mongodg | 12:57 |
| *** janki has quit IRC | 12:58 | |
| mandre_afk | amoralej: I was talking about https://github.com/openstack/tripleo-heat-templates/blob/master/environments/services-docker/zaqar.yaml and https://github.com/openstack/tripleo-heat-templates/blob/master/environments/services/zaqar.yaml | 12:58 |
| mandre_afk | added to my todo list | 12:58 |
| amoralej | mandre_afk, ok | 12:59 |
| *** sasha2 has joined #rdo | 13:02 | |
| trown | dmsimard: adarazs|rover panda|ruck it is not really all of a sudden | 13:02 |
| trown | dmsimard: amoralej panda|ruck it is because we started doing the convert overcloud method rather than building in a pre-job | 13:03 |
| dmsimard | trown: sudden relative from newton to ocata | 13:03 |
| *** martinkopec has quit IRC | 13:03 | |
| trown | whoops unping amoralej tabbed too many times :P | 13:03 |
| amoralej | :) | 13:03 |
| trown | dmsimard: right, it is sudden in that sense, but I am saying there is a clear root cause and it is our fault :P | 13:04 |
| trown | panda|ruck: adarazs|rover, we need to virt-sparsify after converting | 13:04 |
| *** martinkopec has joined #rdo | 13:04 | |
| *** gfidente has joined #rdo | 13:04 | |
| dmsimard | trown: I don't have the full picture but I thought you didn't need to do that with DIB ? | 13:06 |
| dmsimard | especially if you build qcow (e.g, not raw) | 13:07 |
| * dmsimard reads virt-sparsify man | 13:07 | |
| trown | dmsimard: ya the undercloud is not created with DIB... it is created from an overcloud image that was created with DIB | 13:07 |
| trown | dmsimard: doing the virt-customize stuff on the DIB image blows it back up | 13:07 |
| *** jackNemrod has joined #rdo | 13:08 | |
| dmsimard | trown: oh... | 13:08 |
| dmsimard | trown: so somewhere around here then ? http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/roles/convert-image/tasks/main.yml#n27 | 13:08 |
| trown | dmsimard: it was never a problem, because we were previously just using this convert on the fly, and not storing the converted image | 13:09 |
| number80 | dmsimard: wow, undercloud image x3 | 13:10 |
| dmsimard | number80: it's thankfully an easy fix.. I think ? | 13:10 |
| *** vnogin has quit IRC | 13:11 | |
| dmsimard | trown: I'll try and send a patch but http://my1.fr/files/emilien-right-now.jpg | 13:11 |
| *** jprovazn has quit IRC | 13:11 | |
| number80 | I hope so, otherwise for S release, it'll be something like 27GB | 13:11 |
| rdogerrit | Merged rdoinfo master: Promote CBS tags update for pike-testing https://review.rdoproject.org/r/10264 | 13:12 |
| *** jprovazn has joined #rdo | 13:12 | |
| *** jatanmalde has quit IRC | 13:12 | |
| trown | dmsimard: we need something like https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-images/tasks/main.yml#L120-L125 in https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/tripleo-upstream/convert-upload-undercloud.sh#L12-L41 | 13:13 |
| *** vnogin has joined #rdo | 13:13 | |
| trown | dmsimard: I can probably get a patch for it up quickly | 13:13 |
| dmsimard | trown: oh, so not just.. like.. virt-sparsify --in-place ? | 13:13 |
| *** kgiusti has joined #rdo | 13:13 | |
| dmsimard | trown: what you linked doesn't use sparsify though | 13:14 |
| *** vnogin has quit IRC | 13:14 | |
| trown | dmsimard: ya... we switched I guess :P | 13:14 |
| trown | dmsimard: but what I linked is what we used to do in the undercloud image build job | 13:14 |
| *** pkovar has joined #rdo | 13:14 | |
| *** vnogin has joined #rdo | 13:15 | |
| dmsimard | so you need to fix every place where you use virt-customize after DIB then | 13:15 |
| trown | dmsimard: nah just that one spot... I will put up a patch. | 13:15 |
| trown | dmsimard: I think it is simple copy/pasta | 13:15 |
| dmsimard | trown: the convert-image role is okay ? | 13:15 |
| trown | dmsimard: ya, we only care about the image we are storing being large | 13:16 |
| *** vnogin has quit IRC | 13:16 | |
| *** imcleod has joined #rdo | 13:16 | |
| trown | dmsimard: so we can do the shrink right before md5/upload | 13:16 |
| *** jcoufal_ has joined #rdo | 13:16 | |
| dmsimard | trown: what's the cost of sparsifying it ? it's probably not a premature optimization to keep the storage down even if we're not uploading it | 13:16 |
| *** jcoufal has quit IRC | 13:18 | |
| dmsimard | trown: anyway, thanks for looking into this | 13:19 |
| *** newmember has joined #rdo | 13:19 | |
| *** vnogin has joined #rdo | 13:20 | |
| rdogerrit | trown created rdo-infra/ci-config master: Compress undercloud image before upload https://review.rdoproject.org/r/10356 | 13:22 |
| trown | dmsimard: not sure really on what the cost is... it is non-zero though and we timeout jobs already :P ... in any case ^ should make the undercloud smaller | 13:22 |
| *** vnogin has quit IRC | 13:31 | |
| *** jaosorior has quit IRC | 13:33 | |
| chandankumar | number80: can we cancel today's office hour, got stuck with some work, i need to send annoucement email for time change also | 13:36 |
| chandankumar | ? | 13:36 |
| *** abregman_ has joined #rdo | 13:38 | |
| dmsimard | trown: looks ok, +2 | 13:38 |
| *** lucas-hungry is now known as lucasagomes | 13:41 | |
| *** eharney has joined #rdo | 13:41 | |
| *** julim has joined #rdo | 13:44 | |
| *** Amita has quit IRC | 13:45 | |
| *** ratailor has joined #rdo | 13:45 | |
| *** Amita has joined #rdo | 13:46 | |
| *** Amita has quit IRC | 13:46 | |
| chandankumar | number80: amoralej i have updated rdo office hour meeting invite as well as communicated the same on RDO list, next office hour is on 14th Nov. | 13:47 |
| amoralej | chandankumar, cool | 13:47 |
| *** scorcoran_afk is now known as scorcoran_biab | 13:48 | |
| *** amoralej is now known as amoralej|lunch | 13:48 | |
| *** ratailor is now known as rtailor | 13:49 | |
| *** brault has joined #rdo | 13:51 | |
| *** jpena|lunch is now known as jpena | 13:52 | |
| *** vnogin has joined #rdo | 13:54 | |
| *** vnogin has quit IRC | 13:56 | |
| *** vnogin has joined #rdo | 13:56 | |
| *** oanson has quit IRC | 13:57 | |
| *** oanson has joined #rdo | 13:57 | |
| *** agurenko_ is now known as agurenko | 14:00 | |
| number80 | ack | 14:01 |
| *** janki has joined #rdo | 14:01 | |
| dmsimard | adarazs|rover, panda|ruck: fyi https://review.rdoproject.org/r/#/c/10356/ | 14:02 |
| *** scorcoran_biab is now known as scorcoran_mtg | 14:03 | |
| *** mandre_afk is now known as mandre | 14:05 | |
| *** ykarel has quit IRC | 14:06 | |
| rdogerrit | Javier Peña created puppet/puppet-qdr-distgit pike-rdo: Set upstream_name to openstack-qdr if built under DLRN https://review.rdoproject.org/r/10357 | 14:08 |
| rdogerrit | Javier Peña created puppet/puppet-pacemaker-distgit pike-rdo: Set upstream_name to openstack-pacemaker if built under DLRN https://review.rdoproject.org/r/10358 | 14:09 |
| rdogerrit | Javier Peña proposed rdoinfo master: Pin branchless projects for stable releases https://review.rdoproject.org/r/10328 | 14:10 |
| rdogerrit | Javier Peña proposed rdoinfo master: Pin branchless projects for stable releases https://review.rdoproject.org/r/10328 | 14:10 |
| *** gszasz has quit IRC | 14:11 | |
| *** catintheroof has quit IRC | 14:12 | |
| *** noslzzp has quit IRC | 14:13 | |
| *** gszasz has joined #rdo | 14:13 | |
| *** catintheroof has joined #rdo | 14:14 | |
| rdogerrit | Javier Peña proposed puppet/puppet-pacemaker-distgit pike-rdo: Set upstream_name to openstack-pacemaker if built under DLRN https://review.rdoproject.org/r/10358 | 14:17 |
| *** jaosorior has joined #rdo | 14:20 | |
| *** Amita has joined #rdo | 14:21 | |
| *** fzdarsky has joined #rdo | 14:24 | |
| *** amoralej|lunch is now known as amoralej | 14:27 | |
| adarazs|rover | dmsimard: reviewed. also, we're trying to include the image cleanup in the promoter script, as it should know what should be kept. | 14:27 |
| adarazs|rover | dmsimard: panda|ruck has this WIP change for it: https://review.rdoproject.org/r/#/c/10257/ | 14:28 |
| adarazs|rover | which I will pick up | 14:28 |
| dmsimard | adarazs|rover: ok | 14:28 |
| dmsimard | adarazs|rover: I can also tell you how to delete images on the RDO registry, it's easy | 14:29 |
| chandankumar | number80: and rally is moving from away from pbr to setuptools_scm https://review.openstack.org/#/c/516685/ | 14:29 |
| *** catintheroof has quit IRC | 14:30 | |
| *** Son_Goku has quit IRC | 14:31 | |
| *** eharney has quit IRC | 14:31 | |
| *** catintheroof has joined #rdo | 14:35 | |
| amoralej | apevec, p-o-i is passing with new deps repo | 14:36 |
| amoralej | https://ci.centos.org/view/rdo/view/weirdo-pipelines/view/weirdo-promote-puppet/ | 14:36 |
| apevec | yay! | 14:36 |
| *** ykarel has joined #rdo | 14:36 | |
| amoralej | and vpnaas didn't break it | 14:36 |
| amoralej | we have vpnaas in one of the packstack scenarios iirc, | 14:37 |
| *** spectr has quit IRC | 14:37 | |
| apevec | so that's NOT in weirdo-promote-puppet | 14:37 |
| *** spectr has joined #rdo | 14:39 | |
| amoralej | yes | 14:39 |
| amoralej | weirdo promote-puppet is p-o-i only | 14:39 |
| amoralej | testing it i https://ci.centos.org/view/rdo/view/all/job/weirdo-generic-packstack-scenario002/1666/console | 14:41 |
| rdogerrit | Alfredo Moralejo created config master: Revert "Temporarily change timing for openstack-periodic" https://review.rdoproject.org/r/10359 | 14:42 |
| amoralej | mwhahaha, apevec we need to fix the openstack-selinux thing to promote ovs-2.8 once we are able to promote | 14:42 |
| amoralej | so, we have two options addint it as Requires for tripleoclient | 14:42 |
| amoralej | or fixing it in oooq | 14:43 |
| apevec | you said oooclient requires ovs ? | 14:43 |
| amoralej | apevec, kind of | 14:43 |
| apevec | if so, let's add openstack-selinux dep there | 14:43 |
| amoralej | oooclient requires os-net-config which requires ovs | 14:43 |
| amoralej | ok | 14:43 |
| apevec | ah then os-net-config | 14:44 |
| amoralej | undercloud always requires selinux enabled | 14:44 |
| apevec | whatever requires ovs explicitly | 14:44 |
| amoralej | so adding it as requirement in tripleoclient can be good | 14:44 |
| *** cshastri has quit IRC | 14:44 | |
| amoralej | in fact openstack-selinux brings policies not only for ovs | 14:44 |
| apevec | that's true | 14:44 |
| amoralej | so i think requires in ooooclient is better | 14:44 |
| apevec | so if oooclient is effectively UC installer, that's the place | 14:45 |
| amoralej | that's my understanding although some ooo maintainer could have a more accurate idea | 14:45 |
| amoralej | i'll send the patch to ooooclient and ask for reviews | 14:45 |
| *** eharney has joined #rdo | 14:46 | |
| *** rpioso|afk is now known as rpioso | 14:46 | |
| rdogerrit | Merged config master: Revert "Temporarily change timing for openstack-periodic" https://review.rdoproject.org/r/10359 | 14:47 |
| *** udesale has quit IRC | 14:47 | |
| *** noslzzp has joined #rdo | 14:49 | |
| apevec | mandre, amoralej, ugh, so now is oraclelinux blocking us? https://review.openstack.org/#/c/516643/ | 14:51 |
| amoralej | job timed out | 14:52 |
| apevec | if this repeats, I'd suggest temp rpm patch in rpm-master | 14:52 |
| amoralej | but seems unrelated | 14:53 |
| amoralej | we could add python2-cffi to the deps repo also | 14:53 |
| amoralej | until merged | 14:53 |
| amoralej | apevec, in http://logs.openstack.org/43/516643/2/check/kolla-build-oraclelinux-binary/6abe948/job-output.txt.gz it failed to install elasticsearch | 14:54 |
| amoralej | looks infra issue | 14:54 |
| apevec | it seems to be using external repos w/o infra mirror | 14:56 |
| apevec | I see Copr repo | 14:56 |
| rdogerrit | Alfredo Moralejo created openstack/tripleoclient-distgit rpm-master: Add openstack-selinux as requirement https://review.rdoproject.org/r/10360 | 14:57 |
| *** abregman_ is now known as abregman|mtg | 14:58 | |
| *** dtantsur|afk is now known as dtantsur | 14:58 | |
| *** vnogin has quit IRC | 14:59 | |
| apevec | amoralej, ES looks fine? http://logs.openstack.org/43/516643/2/check/kolla-build-oraclelinux-binary/6abe948/logs/build/elasticsearch.txt.gz | 14:59 |
| apevec | it's just slow I guess | 14:59 |
| *** dmarlin has joined #rdo | 15:00 | |
| amoralej | apevec, http://logs.openstack.org/43/516643/2/check/kolla-build-oraclelinux-binary/6abe948/job-output.txt.gz#_2017-10-31_13_04_56_453527 | 15:00 |
| amoralej | it's grafana container | 15:00 |
| amoralej | apevec, i'll push patch for kolla in distgit | 15:01 |
| amoralej | send review, i mean | 15:01 |
| remix_tj | Duck: hi, in rdo dev ml i see there are still a reference to the old unsubscribe email address on the mail footers. Is it right? | 15:01 |
| apevec | ack | 15:01 |
| apevec | so yeah, it's using elastic.co repo w/o infra mirror... | 15:01 |
| *** abregman|mtg has quit IRC | 15:04 | |
| rdogerrit | Alfredo Moralejo created openstack/kolla-distgit rpm-master: Switch to python-cffi for rpm based distros https://review.rdoproject.org/r/10361 | 15:04 |
| *** vnogin has joined #rdo | 15:05 | |
| *** nstrug has quit IRC | 15:06 | |
| *** agurenko is now known as agurenko_ | 15:07 | |
| rdogerrit | Alfredo Moralejo proposed openstack/kolla-distgit rpm-master: Switch to python-cffi for rpm based distros https://review.rdoproject.org/r/10361 | 15:08 |
| *** jhershbe has quit IRC | 15:08 | |
| *** oidgar has quit IRC | 15:09 | |
| rdogerrit | Merged openstack/heatclient-distgit pike-rdo: Update to 1.11.1 https://review.rdoproject.org/r/10346 | 15:10 |
| *** ykarel has quit IRC | 15:11 | |
| *** jprovazn has quit IRC | 15:14 | |
| amoralej | mandre, can we apply the patch in the package until it's merge to unblock promotions https://review.rdoproject.org/r/#/c/10361/? | 15:14 |
| *** anshul has quit IRC | 15:15 | |
| *** dtantsur is now known as dtantsur|afk | 15:15 | |
| *** newmember has quit IRC | 15:16 | |
| amoralej | jpena, packstack fails after vpn change https://ci.centos.org/view/rdo/view/all/job/weirdo-generic-packstack-scenario002/1666/console | 15:18 |
| *** jbadiapa has quit IRC | 15:18 | |
| amoralej | wdyt about removing vpnaas from scenario0002 until it's fixed in puppet-neutron? | 15:19 |
| *** jcoufal_ has quit IRC | 15:19 | |
| *** jcoufal has joined #rdo | 15:20 | |
| jpena | amoralej: ok, let's do it in the meantime | 15:20 |
| amoralej | yeah, jpena i'll send the patch | 15:20 |
| *** mosulica has quit IRC | 15:21 | |
| *** martinkopec has quit IRC | 15:23 | |
| openstackgerrit | Alfredo Moralejo proposed openstack/packstack master: Disable vpnaas from scenario002 https://review.openstack.org/516714 | 15:23 |
| *** mosulica has joined #rdo | 15:24 | |
| *** noslzzp has quit IRC | 15:25 | |
| *** noslzzp has joined #rdo | 15:26 | |
| mandre | amoralej: just so that I understand, once the kolla fix lands -- including the backports -- the resulting diff will be empty and the patch in the rpm packaging will still apply without failing, right? | 15:28 |
| amoralej | mandre, nop | 15:28 |
| amoralej | we need to revert it | 15:28 |
| amoralej | it will fail and notify us | 15:28 |
| mandre | amoralej: ack, so potentially in an hour or so? | 15:29 |
| amoralej | yeah | 15:29 |
| *** spectr has quit IRC | 15:29 | |
| mandre | ok, at least that forces us to keep things clean | 15:29 |
| amoralej | mandre, yes | 15:29 |
| *** jhershbe has joined #rdo | 15:29 | |
| amoralej | it's a good think imo | 15:30 |
| rdogerrit | Merged openstack/kolla-distgit rpm-master: Switch to python-cffi for rpm based distros https://review.rdoproject.org/r/10361 | 15:30 |
| mandre | amoralej: absolutely, that was just for my own curiosity | 15:31 |
| *** catintheroof has quit IRC | 15:34 | |
| *** Pharaoh_Atem has quit IRC | 15:34 | |
| *** brault has quit IRC | 15:38 | |
| *** brault has joined #rdo | 15:38 | |
| *** pblaho has quit IRC | 15:40 | |
| *** catintheroof has joined #rdo | 15:40 | |
| *** pblaho has joined #rdo | 15:40 | |
| amoralej | adarazs|rover, apevec re-running periodic jobs | 15:40 |
| amoralej | with patched kolla | 15:40 |
| adarazs|rover | amoralej: with the magic zuul command that triggers the pipeline? | 15:41 |
| amoralej | adarazs|rover, | 15:41 |
| amoralej | yes | 15:41 |
| dmsimard | yeah it turns out it's possible | 15:41 |
| adarazs|rover | amazing :) | 15:41 |
| amoralej | for the record | 15:41 |
| amoralej | "zuul enqueue-ref --trigger timer --pipeline openstack-periodic --project openstack-infra/tripleo-ci --ref refs/heads/master --newrev 82ea679c63284336806d19f2c821177c0046e11b" | 15:41 |
| *** spectr has joined #rdo | 15:41 | |
| *** brault has quit IRC | 15:41 | |
| dmsimard | amoralej: that triggers the whole set of jobs though | 15:42 |
| dmsimard | I suppose you can't trigger just one if you wanted to | 15:42 |
| adarazs|rover | I don't think so. however we can make multiple pipelines... | 15:42 |
| adarazs|rover | at least per release. | 15:42 |
| adarazs|rover | if necessary | 15:42 |
| *** catintheroof has quit IRC | 15:42 | |
| panda|ruck | didn't we had the next run scheduled in 15 minutes ? | 15:43 |
| adarazs|rover | panda|ruck: no, it was at 1400UTC | 15:44 |
| adarazs|rover | maybe you're confused by the daylight saving change? | 15:44 |
| amoralej | next run was 19:00 UTC | 15:45 |
| *** sbulage has quit IRC | 15:45 | |
| panda|ruck | ok, now we have 9 10:30 15:45 runs. | 15:47 |
| adarazs|rover | panda|ruck: these are exceptional times :) | 15:48 |
| panda|ruck | nothing exceptional in them, I like 11:11 more for example. | 15:49 |
| *** jparrill has joined #rdo | 15:50 | |
| *** jbadiapa has joined #rdo | 15:55 | |
| *** rtailor has quit IRC | 15:55 | |
| *** dyasny__ has quit IRC | 15:56 | |
| *** brault has joined #rdo | 15:56 | |
| *** ihrachys has joined #rdo | 15:57 | |
| *** dyasny has joined #rdo | 15:57 | |
| *** sbonazzo has quit IRC | 15:58 | |
| *** julim_ has joined #rdo | 15:59 | |
| *** janki has quit IRC | 16:00 | |
| *** bnemec has quit IRC | 16:00 | |
| *** tshefi has quit IRC | 16:00 | |
| *** julim has quit IRC | 16:01 | |
| *** jaosorior has quit IRC | 16:03 | |
| *** lkuchlan has quit IRC | 16:06 | |
| *** egonzalez has quit IRC | 16:06 | |
| *** ushkalim has quit IRC | 16:06 | |
| *** mvk has quit IRC | 16:07 | |
| *** anilvenkata has quit IRC | 16:07 | |
| *** vnogin has quit IRC | 16:14 | |
| *** bnemec has joined #rdo | 16:15 | |
| *** trown is now known as trown|lunch | 16:16 | |
| *** rbartal__ has quit IRC | 16:17 | |
| *** rbartal_ has quit IRC | 16:17 | |
| *** rbartal has quit IRC | 16:17 | |
| *** panda|ruck is now known as panda|ruck|bbl | 16:18 | |
| *** jhershbe has quit IRC | 16:19 | |
| *** fragatina has joined #rdo | 16:20 | |
| *** fragatina has quit IRC | 16:21 | |
| *** fragatina has joined #rdo | 16:22 | |
| *** gkadam has joined #rdo | 16:22 | |
| *** shardy is now known as shardy_afk | 16:24 | |
| *** lebauce has quit IRC | 16:24 | |
| *** _bsurfer has quit IRC | 16:25 | |
| *** jackNemrod has quit IRC | 16:25 | |
| *** aufi has quit IRC | 16:25 | |
| *** brault has quit IRC | 16:26 | |
| *** brault has joined #rdo | 16:26 | |
| adarazs|rover | amoralej, panda|ruck|bbl: hmm, interesting the pike-upload job failed with | 16:26 |
| adarazs|rover | /home/jenkins/workspace/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset002-pike-upload/devstack-gate/functions.sh: line 180: declare: gate_hook: not found | 16:26 |
| adarazs|rover | https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset002-pike-upload/223/console | 16:27 |
| amoralej | let me check | 16:27 |
| adarazs|rover | something might be different. | 16:27 |
| adarazs|rover | when triggered by hand | 16:27 |
| amoralej | i thing gate_hook: not found is expected | 16:27 |
| *** mosulica has quit IRC | 16:27 | |
| *** brault has quit IRC | 16:27 | |
| *** brault has joined #rdo | 16:28 | |
| adarazs|rover | I don't see any other error that made the job abort. but I might be tired. | 16:28 |
| dmsimard | that gate_hook thing rings a bell | 16:28 |
| amoralej | adarazs|rover, https://logs.rdoproject.org/82/82ea679c63284336806d19f2c821177c0046e11b/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset002-pike-upload/master/devstack-gate-setup-host.txt | 16:29 |
| amoralej | PING mirror01.dfw.rax.openstack.org (104.130.132.190) 56(84) bytes of data. | 16:29 |
| adarazs|rover | damn. | 16:29 |
| amoralej | dmsimard, ^ | 16:29 |
| adarazs|rover | there's always some new error. | 16:29 |
| *** iranzo has quit IRC | 16:30 | |
| amoralej | is it happeingint in master? | 16:30 |
| dmsimard | amoralej: http://logs.openstack.org/89/507889/2/gate/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-puppet/a8829f3/job-output.txt.gz#_2017-09-28_19_06_33_406425 | 16:30 |
| amoralej | not so far | 16:30 |
| dmsimard | from http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2017-09-28.log.html#t2017-09-28T19:16:00 | 16:30 |
| amoralej | it always appear | 16:30 |
| dmsimard | let me check what was the issue | 16:30 |
| amoralej | it checks if it's defined | 16:30 |
| amoralej | dmsimard, the actual problem is in https://logs.rdoproject.org/82/82ea679c63284336806d19f2c821177c0046e11b/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset002-pike-upload/master/devstack-gate-setup-host.txt | 16:30 |
| dmsimard | oh I remember why I remember that now, I ran through this issue with mwahaha | 16:30 |
| dmsimard | mwhahaha: * | 16:30 |
| amoralej | PING mirror01.dfw.rax.openstack.org (104.130.132.190) 56(84) bytes of data. failed | 16:31 |
| dmsimard | amoralej: that pings in both v4 and v6 for me, odd | 16:33 |
| amoralej | dmsimard, the "gate_hook: not found" is not an actual issue, it appears in jobs passing https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset002-pike-upload/222/consoleFull | 16:33 |
| mwhahaha | gate_hook not found is a symptom of host not setting up correctly | 16:33 |
| amoralej | dmsimard, ipv6? | 16:33 |
| mwhahaha | it's usually not the actual problem | 16:33 |
| mwhahaha | except in the v2/v3 switch where we needed to adjust things | 16:33 |
| mwhahaha | but it usually shows up when the host fails to configure | 16:33 |
| dmsimard | mwhahaha: yeah we discussed that issue in late august | 16:34 |
| dmsimard | september* | 16:34 |
| *** anilvenkata has joined #rdo | 16:34 | |
| amoralej | i only see ping to ipv4, right? | 16:34 |
| dmsimard | amoralej: just saying that "mirror01.dfw.rax.openstack.org" pings for me in v6 (i.e, from home) and in v4 (from a rdo cloud instance) | 16:34 |
| amoralej | oh, ok | 16:34 |
| mwhahaha | might be transient | 16:35 |
| adarazs|rover | probably it was, the other jobs worked and they use the same mirror | 16:35 |
| amoralej | it seems some jobs in the periodic pipeline have been restarted | 16:35 |
| *** rbowen has quit IRC | 16:36 | |
| amoralej | there may be network errors | 16:36 |
| amoralej | look at https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset002-master-upload/220/console | 16:36 |
| *** _bsurfer has joined #rdo | 16:38 | |
| *** ragiman has quit IRC | 16:38 | |
| sfbender | Jakub Ružička proposed rdopkg master: pkgenv: show patches base and base git ref information https://softwarefactory-project.io/r/10029 | 16:39 |
| *** oidgar has joined #rdo | 16:39 | |
| *** catintheroof has joined #rdo | 16:40 | |
| *** anilvenkata has quit IRC | 16:44 | |
| *** catintheroof has quit IRC | 16:45 | |
| *** iranzo has joined #rdo | 16:49 | |
| *** iranzo has quit IRC | 16:49 | |
| *** iranzo has joined #rdo | 16:49 | |
| *** mcornea has quit IRC | 16:52 | |
| *** rbowen has joined #rdo | 16:54 | |
| *** ChanServ sets mode: +o rbowen | 16:54 | |
| *** gkadam has quit IRC | 16:54 | |
| *** iranzo has quit IRC | 16:54 | |
| *** vnogin has joined #rdo | 16:57 | |
| *** bnemec has quit IRC | 16:57 | |
| amoralej | apevec, i'm adding openstack-macros to the new deps repo | 17:00 |
| *** rbowen has quit IRC | 17:00 | |
| amoralej | we need it until we have a new promotion | 17:00 |
| amoralej | it's not in current-passed-ci | 17:00 |
| *** emalikov has quit IRC | 17:00 | |
| *** Son_Goku has joined #rdo | 17:00 | |
| *** scorcoran_mtg is now known as scorcoran_biab | 17:01 | |
| *** vnogin has quit IRC | 17:01 | |
| *** jcoufal has quit IRC | 17:01 | |
| *** jcoufal has joined #rdo | 17:03 | |
| *** jhershbe has joined #rdo | 17:03 | |
| *** mvk has joined #rdo | 17:06 | |
| *** vnogin has joined #rdo | 17:07 | |
| *** _bsurfer has quit IRC | 17:07 | |
| *** paragan has quit IRC | 17:08 | |
| *** vnogin has quit IRC | 17:10 | |
| *** scorcoran_biab is now known as scorcoran | 17:11 | |
| *** tosky has quit IRC | 17:11 | |
| *** lucasagomes is now known as lucas-afk | 17:12 | |
| *** catintheroof has joined #rdo | 17:12 | |
| *** bnemec has joined #rdo | 17:12 | |
| *** jprovazn has joined #rdo | 17:15 | |
| *** fragatina has quit IRC | 17:24 | |
| *** fragatina has joined #rdo | 17:25 | |
| EmilienM | amoralej: what are remaining blockers at this point? | 17:29 |
| *** jpich has quit IRC | 17:29 | |
| amoralej | EmilienM, hopefully none | 17:29 |
| EmilienM | ok | 17:29 |
| amoralej | jobs are running | 17:29 |
| *** pcaruana has quit IRC | 17:29 | |
| *** fragatina has quit IRC | 17:29 | |
| *** trown|lunch is now known as trown | 17:30 | |
| *** tesseract has quit IRC | 17:36 | |
| rdogerrit | trown proposed rdo-infra/ci-config master: Compress undercloud image before upload https://review.rdoproject.org/r/10356 | 17:36 |
| *** d0ugal has quit IRC | 17:43 | |
| *** Son_Goku has quit IRC | 17:47 | |
| *** vnogin has joined #rdo | 17:49 | |
| *** fragatina has joined #rdo | 17:50 | |
| *** vnogin has quit IRC | 17:50 | |
| *** ccamacho has quit IRC | 17:53 | |
| *** sbulage has joined #rdo | 17:53 | |
| *** mcornea has joined #rdo | 17:53 | |
| *** jhershbe has quit IRC | 17:53 | |
| rdogerrit | Javier Peña created openstack/openstackclient-distgit rpm-master: Build python3 bits before compiling the catalog https://review.rdoproject.org/r/10362 | 18:01 |
| *** fzdarsky has quit IRC | 18:02 | |
| *** panda|ruck|bbl is now known as panda|ruck | 18:02 | |
| *** jackNemrod has joined #rdo | 18:02 | |
| *** d0ugal has joined #rdo | 18:03 | |
| rdogerrit | Javier Peña created openstack/congressclient-distgit rpm-master: Fix python3 build https://review.rdoproject.org/r/10363 | 18:07 |
| *** jpena is now known as jpena|off | 18:10 | |
| *** vnogin has joined #rdo | 18:11 | |
| *** MrAbaddon has quit IRC | 18:15 | |
| *** agurenko_ has quit IRC | 18:16 | |
| *** oidgar has quit IRC | 18:17 | |
| EmilienM | apevec: http://status.openstack.org/elastic-recheck/#1708832 - not sure what we can do to avoid this situation next time | 18:17 |
| *** jlibosva has quit IRC | 18:18 | |
| apevec | EmilienM, need to look at those failures, is it all the same root cause? | 18:18 |
| apevec | "A generic bug used to track DLRN build failures in the gate" | 18:19 |
| apevec | ^ is not useful | 18:19 |
| EmilienM | apevec: well, dlrn api was down this morning iiuc | 18:19 |
| EmilienM | apevec: it's useful for us when we need to know where comes frm the issue | 18:19 |
| EmilienM | apevec: not useful for you maybe, missing more context | 18:19 |
| apevec | lemme look what is es query actually matching | 18:20 |
| apevec | lemme look what is es query actually matching | 18:20 |
| apevec | oops | 18:20 |
| mwhahaha | it was due to openstack-macros missing | 18:20 |
| mwhahaha | i talked to amoralej about it this morning, something about repos being changed | 18:20 |
| apevec | you say dlrnapi but LP is about dlrn build ? | 18:20 |
| apevec | amoralej fixed openstack-macros | 18:21 |
| EmilienM | I think the query is too broad | 18:21 |
| *** racedo_ has quit IRC | 18:21 | |
| EmilienM | apevec: how did openstack-macros break? | 18:21 |
| *** pchavva has quit IRC | 18:21 | |
| apevec | it was missing in the last current-passed-ci | 18:21 |
| amoralej | apevec, i added openstack-macros to deps repo | 18:21 |
| *** d0ugal_ has joined #rdo | 18:21 | |
| apevec | we moved it to trunk, to follow upstream rpm-packaging | 18:22 |
| amoralej | as it's not in current-passed-ci | 18:22 |
| apevec | but current-passed-ci was way too old | 18:22 |
| apevec | yeah, so amoralej did workaround by putting it into deps repo | 18:22 |
| apevec | we had it (older version) in the <= pike deps | 18:22 |
| amoralej | apevec, we have to be careful when removing it | 18:22 |
| amoralej | to not break jobs with cached metadata | 18:22 |
| apevec | yeah, definitely need more coverage in rdoinfo buildsys ci | 18:22 |
| EmilienM | do we have actions on that? | 18:23 |
| EmilienM | like adding coverage, etc | 18:23 |
| mwhahaha | so i guess my question around this was where was the notificaiton that this was happening/testing around this change? | 18:23 |
| *** d0ugal has quit IRC | 18:23 | |
| amoralej | apevec, any idea about how to increase coverage with reasonable resources requirements? | 18:24 |
| apevec | mwhahaha, it was deps repo change https://review.openstack.org/508870 | 18:24 |
| mwhahaha | so how did that not get caught in that ci run | 18:25 |
| mwhahaha | are we not actually testing the config changes | 18:25 |
| apevec | it didn't build any package which has BR openstack-macros | 18:25 |
| apevec | this has indeed hit various issues in ci, as you can see from review history | 18:26 |
| amoralej | mmmm | 18:26 |
| apevec | but 100% it is not | 18:26 |
| *** hewbrocca is now known as hewbrocca_afk | 18:26 | |
| apevec | amoralej, I guess we could trigger few dlrn-build jobs on representative projects? | 18:26 |
| amoralej | apevec, we could have catch this with repoclosure? | 18:26 |
| *** anshul has joined #rdo | 18:27 | |
| apevec | nope, it's BR | 18:27 |
| mwhahaha | seems like we should always be building the tripleo projects or something | 18:27 |
| mwhahaha | cause it failed a bunch on tripleo-common | 18:27 |
| amoralej | repoclosure doesn't check BRs? | 18:27 |
| apevec | no, only Requires: | 18:27 |
| amoralej | it has -b | 18:28 |
| amoralej | -b, --builddeps | 18:28 |
| *** vnogin has quit IRC | 18:28 | |
| apevec | ok, but we might not actually ship ALL BRs | 18:28 |
| apevec | for some deps | 18:28 |
| apevec | amoralej, also this is BR of openstack-* which are not in deps repo... | 18:29 |
| amoralej | apevec, i run repoclosure with delorean + deps enabled | 18:29 |
| EmilienM | do we know why periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset002-pike-upload failed? | 18:29 |
| mwhahaha | this is really just fallout from the lack of promotion | 18:29 |
| amoralej | iirc | 18:29 |
| * EmilienM clicks on logs | 18:29 | |
| EmilienM | 16:23:42 + echo 'REPORTING FAILURE TO DLRN API' | 18:29 |
| EmilienM | on https://review.rdoproject.org/jenkins/job/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset002-pike-upload/223/console | 18:29 |
| EmilienM | oops? | 18:30 |
| *** chlong has joined #rdo | 18:30 | |
| amoralej | EmilienM, that looks like infra issue | 18:30 |
| amoralej | there was networking issues while running | 18:30 |
| *** julim_ has quit IRC | 18:31 | |
| EmilienM | where do you see it? | 18:31 |
| EmilienM | do we have actions to resolve / stabilize it? | 18:31 |
| EmilienM | is it a mirror issue? | 18:31 |
| *** julim has joined #rdo | 18:31 | |
| amoralej | we discussed that issue with adarazs|rover earlier | 18:31 |
| amoralej | let me look for it | 18:31 |
| amoralej | EmilienM, http://eavesdrop.openstack.org/irclogs/%23rdo/latest.log.html#t2017-10-31T16:29:22 | 18:32 |
| EmilienM | ok | 18:32 |
| amoralej | some other jobs lost connectivity with jenkins slaves and were restarted | 18:32 |
| amoralej | but that one didn't | 18:33 |
| *** toure is now known as toure_biab | 18:37 | |
| *** bnemec has quit IRC | 18:38 | |
| *** newmember has joined #rdo | 18:42 | |
| *** oidgar has joined #rdo | 18:44 | |
| *** abishop has joined #rdo | 18:44 | |
| abishop | hi folks, I have a question about the process for building RPMs on stable branches like pike | 18:46 |
| abishop | the openstack-glance spec file was updated a couple weeks back, but I don't see a new rpm | 18:47 |
| abishop | should an update to the spec file trigger building a new rpm? | 18:47 |
| *** dprince has quit IRC | 18:49 | |
| amoralej | abishop | 18:49 |
| amoralej | depends | 18:49 |
| EmilienM | abishop: we didn't have a promotion since 52 days | 18:49 |
| amoralej | where is the patch? | 18:49 |
| amoralej | EmilienM, he's talking about pike, not master | 18:49 |
| EmilienM | oops | 18:50 |
| * EmilienM hides | 18:50 | |
| abishop | https://review.rdoproject.org/r/10169 | 18:50 |
| amoralej | :) | 18:50 |
| amoralej | abishop, that patch makes that a new rpm is built in RDO Trunk | 18:50 |
| amoralej | but not in official cloudsig repos | 18:50 |
| amoralej | for it you need to update Release | 18:50 |
| amoralej | send a patch bumping the Release please | 18:51 |
| amoralej | abishop, change https://github.com/rdo-packages/glance-distgit/blob/pike-rdo/openstack-glance.spec#L22 to 2%{?dist} | 18:51 |
| abishop | sorry, I'm confused, this patch is on pike-rdo, so not sure what you mean by RDO trunk | 18:51 |
| amoralej | and add an entry to changelog | 18:52 |
| amoralej | abishop, we have two different sets of repos "RDO Trunk" and "RDO CloudSIG" | 18:52 |
| amoralej | https://www.rdoproject.org/blog/2016/05/new-in-rdo-repos-one-size-doesn-t-fit-all/ | 18:52 |
| amoralej | official repos are CloudSIG ones | 18:52 |
| * abishop goes to read blog | 18:53 | |
| amoralej | abishop, in the meanwhile you can send the review to bump Release | 18:54 |
| *** Pharaoh_Atem has joined #rdo | 18:57 | |
| *** Pharaoh_Atem has quit IRC | 18:57 | |
| *** Pharaoh_Atem has joined #rdo | 18:57 | |
| *** rbowen has joined #rdo | 19:01 | |
| *** ChanServ sets mode: +o rbowen | 19:01 | |
| *** oidgar has quit IRC | 19:01 | |
| amoralej | EmilienM, adarazs|rover all ovn and non-container multinode jobs passed | 19:01 |
| amoralej | let's see the containers ones | 19:02 |
| EmilienM | nice | 19:06 |
| abishop | amoralej: does changelog need to say anything more than "Update to 15.0.0-2" | 19:09 |
| amoralej | abishop, yes, in this case you are not updating to a new glance version | 19:09 |
| amoralej | so something describing the patch you sent would be more appropiate | 19:10 |
| amoralej | something like "Fixed links for glance_store rottwrap filters" | 19:10 |
| amoralej | abishop, was that the fix? | 19:11 |
| abishop | amoralej: got it | 19:11 |
| abishop | amoralej: yes | 19:11 |
| *** catintheroof has quit IRC | 19:14 | |
| *** pcaruana has joined #rdo | 19:15 | |
| *** dprince has joined #rdo | 19:15 | |
| rdogerrit | Alan Bishop created openstack/glance-distgit pike-rdo: Update to 15.0.0-2 https://review.rdoproject.org/r/10364 | 19:15 |
| abishop | amoralej: https://review.rdoproject.org/r/10364 | 19:15 |
| *** rbowen has quit IRC | 19:15 | |
| *** oidgar has joined #rdo | 19:18 | |
| panda|ruck | containers build jobs are taking 3 hours | 19:21 |
| *** newmember has quit IRC | 19:23 | |
| *** pkovar has quit IRC | 19:25 | |
| *** cylopez has joined #rdo | 19:26 | |
| *** eharney has quit IRC | 19:29 | |
| *** eharney has joined #rdo | 19:30 | |
| *** mcornea has quit IRC | 19:32 | |
| *** cylopez has quit IRC | 19:34 | |
| *** cylopez has joined #rdo | 19:35 | |
| amoralej | abishop, after merging that, we need some time to get it promoted to production repo | 19:35 |
| *** MrAbaddon has joined #rdo | 19:37 | |
| abishop | amoralej: thx for your attention and guidance on this! what sort of units is "some time?" | 19:37 |
| amoralej | abishop, days probably | 19:37 |
| amoralej | we have some bots automatically proposing updates | 19:38 |
| amoralej | we can try to speed it up if it's urgent | 19:38 |
| *** myoung is now known as myoung|afk | 19:39 | |
| abishop | amoralej: anything that expedites would be appreciated | 19:39 |
| *** hrw has quit IRC | 19:39 | |
| *** hrw has joined #rdo | 19:41 | |
| *** oidgar has quit IRC | 19:42 | |
| rdogerrit | Merged openstack/glance-distgit pike-rdo: Update to 15.0.0-2 https://review.rdoproject.org/r/10364 | 19:44 |
| rdogerrit | Alfredo Moralejo created rdoinfo master: Update openstack-glance https://review.rdoproject.org/r/10365 | 19:45 |
| amoralej | abishop, ^ | 19:46 |
| abishop | amoralej: is there a cloudsig repo status page similar to https://trunk.rdoproject.org ? | 19:46 |
| amoralej | that promotes it to testing repo and then we need to push it to -release | 19:46 |
| amoralej | abishop, nop | 19:46 |
| amoralej | repos for cloudsig are in | 19:46 |
| amoralej | abishop, http://mirror.centos.org/centos-7/7/cloud/x86_64/openstack-pike/ | 19:47 |
| abishop | amoralej: that works, bookmarking it | 19:48 |
| *** amoralej is now known as amoralej|off | 19:48 | |
| *** pcaruana has quit IRC | 19:49 | |
| *** rbowen has joined #rdo | 19:51 | |
| *** ChanServ sets mode: +o rbowen | 19:51 | |
| *** catintheroof has joined #rdo | 19:55 | |
| *** rbowen has quit IRC | 19:56 | |
| *** rbowen has joined #rdo | 19:56 | |
| *** ChanServ sets mode: +o rbowen | 19:56 | |
| *** cylopez has quit IRC | 20:05 | |
| *** gszasz has quit IRC | 20:06 | |
| *** ccamacho has joined #rdo | 20:10 | |
| *** gfidente is now known as gfidente|afk | 20:12 | |
| *** jtomasek has quit IRC | 20:17 | |
| *** mrunge has quit IRC | 20:18 | |
| *** kgiusti has left #rdo | 20:20 | |
| *** mrunge has joined #rdo | 20:22 | |
| *** jmelvin has quit IRC | 20:23 | |
| *** sbulage has quit IRC | 20:28 | |
| *** vnogin has joined #rdo | 20:29 | |
| *** vnogin has quit IRC | 20:33 | |
| *** julim has quit IRC | 20:35 | |
| *** jayg is now known as jayg|g0n3 | 20:35 | |
| *** jackNemrod has quit IRC | 20:38 | |
| *** sshnaidm is now known as sshnaidm|afk | 20:39 | |
| *** jackNemrod has joined #rdo | 20:40 | |
| *** trown is now known as trown|outtypewww | 20:45 | |
| *** jcoufal_ has joined #rdo | 21:02 | |
| *** jcoufal has quit IRC | 21:05 | |
| *** toure_biab is now known as toure | 21:05 | |
| *** toure is now known as toure_biab | 21:05 | |
| *** eharney has quit IRC | 21:07 | |
| *** catintheroof has quit IRC | 21:09 | |
| *** rhallisey has quit IRC | 21:10 | |
| *** dustins has quit IRC | 21:13 | |
| *** rbowen has quit IRC | 21:13 | |
| *** julim has joined #rdo | 21:25 | |
| *** julim has quit IRC | 21:26 | |
| *** jmelvin has joined #rdo | 21:27 | |
| *** DV has joined #rdo | 21:31 | |
| *** mrunge has quit IRC | 21:31 | |
| *** rcernin has joined #rdo | 21:32 | |
| *** anshul has quit IRC | 21:34 | |
| *** amoralej|off is now known as amoralej | 21:34 | |
| *** threestrands has joined #rdo | 21:36 | |
| *** DV has quit IRC | 21:38 | |
| *** mrunge has joined #rdo | 21:39 | |
| *** jprovazn has quit IRC | 21:45 | |
| panda|ruck | plase never retrigger manually with that zuul command again. | 21:45 |
| *** dprince has quit IRC | 21:46 | |
| *** dmarlin has quit IRC | 21:48 | |
| *** jcoufal has joined #rdo | 21:49 | |
| rdogerrit | rdo-trunk created openstack/kolla-distgit rpm-master: openstack-kolla: failed to build 8d024ff https://review.rdoproject.org/r/10366 | 21:49 |
| *** scorcora` has joined #rdo | 21:49 | |
| *** jcoufal_ has quit IRC | 21:50 | |
| dmsimard | panda|ruck: why ? | 21:52 |
| rdobot | [sensu] NEW: master.monitoring.rdoproject.org - check-delorean-master-current @ http://tinyurl.com/y8hxewm2 |#| Build failure on centos7-master/current: kolla: http://trunk.rdoproject.org/centos7-master/report.html | 21:52 |
| dmsimard | panda|ruck: ok, nevermind I see in #tripleo | 21:52 |
| panda|ruck | dmsimard: because it doesn't stop a second trigger, and running two periodic at the same time made a mess | 21:52 |
| *** imcleod has quit IRC | 21:53 | |
| *** scorcoran is now known as scorcoran_afk | 21:53 | |
| *** scorcoran_afk has quit IRC | 21:53 | |
| *** bnemec has joined #rdo | 21:55 | |
| *** julim has joined #rdo | 21:58 | |
| *** paramite has quit IRC | 21:59 | |
| *** julim has quit IRC | 22:00 | |
| *** jcoufal has quit IRC | 22:02 | |
| rdogerrit | Merged openstack/tripleoclient-distgit rpm-master: Add openstack-selinux as requirement https://review.rdoproject.org/r/10360 | 22:02 |
| rdogerrit | Merged puppet/puppet-pacemaker-distgit pike-rdo: Set upstream_name to openstack-pacemaker if built under DLRN https://review.rdoproject.org/r/10358 | 22:02 |
| rdogerrit | Merged puppet/puppet-qdr-distgit pike-rdo: Set upstream_name to openstack-qdr if built under DLRN https://review.rdoproject.org/r/10357 | 22:02 |
| *** jmelvin has quit IRC | 22:05 | |
| *** amoralej is now known as amoralej|off | 22:05 | |
| *** rpioso is now known as rpioso|afk | 22:13 | |
| *** scorcoran has joined #rdo | 22:13 | |
| *** chlong has quit IRC | 22:14 | |
| rdogerrit | Merged rdoinfo master: Update openstack-glance https://review.rdoproject.org/r/10365 | 22:19 |
| *** gfidente|afk has quit IRC | 22:19 | |
| apevec | panda|ruck, did we just have a Halloween release? https://trunk.rdoproject.org/centos7-master/current-tripleo/ timestamp 2017-10-31 17:46 ! | 22:23 |
| panda|ruck | apevec: yes, halloween, it mean that is haunted | 22:24 |
| panda|ruck | apevec: it's broken | 22:24 |
| panda|ruck | I'm trying to revert it, if it's possible | 22:24 |
| apevec | in dlrnapi not sure but we can change symlink on trunk.rdo manually | 22:24 |
| apevec | so when manually triggering, we need to stop the cron | 22:25 |
| apevec | dmsimard, what is --newrev 82ea679c63284336806d19f2c821177c0046e11b in the enqueue-ref | 22:26 |
| panda|ruck | apevec: I knew this was a lot more difficult than it lloked | 22:26 |
| apevec | ? | 22:26 |
| panda|ruck | apevec: he's out | 22:26 |
| apevec | it's not that difficult, we just need to remember what's going on | 22:26 |
| apevec | panda|ruck, ok, so you're saying let's move current-tripleo back to whatever it was before? | 22:26 |
| panda|ruck | apevec: another thing is that I don't know what's the distro hash of the previous promotion | 22:27 |
| apevec | do you have a record which hash was it? | 22:27 |
| apevec | heh | 22:27 |
| panda|ruck | apevec: yes | 22:27 |
| apevec | lemme check in log | 22:27 |
| panda|ruck | f442a3aa35981c3d6d7e312599dde2a1b1d202c9_0468cca4 | 22:27 |
| panda|ruck | this is the full hash of the previous promotion | 22:27 |
| panda|ruck | I'll launch a dlrnapi repo-promote with those hashes and current-tripleo | 22:28 |
| panda|ruck | will it work ? | 22:28 |
| apevec | panda|ruck, you just said you don't know it? :) | 22:28 |
| panda|ruck | apevec: I don't know the distro hash | 22:28 |
| panda|ruck | apevec: the full ditro hash | 22:28 |
| apevec | but that above is the full distro hash? | 22:28 |
| panda|ruck | apevec: I have the commit hash and the first 8 digits of the distro hash | 22:28 |
| apevec | not sure what do you mean by "distro hash" ? | 22:28 |
| apevec | ah | 22:28 |
| apevec | short should do, lemme find out from gerrit... | 22:29 |
| panda|ruck | apevec: the sha1 of the distgit repo associeted with the commit hash ? | 22:29 |
| apevec | that's distgit then | 22:29 |
| apevec | distro confused me | 22:29 |
| panda|ruck | usage: dlrnapi repo-promote [-h] --commit-hash COMMIT_HASH --distro-hash | 22:29 |
| panda|ruck | DISTRO_HASH --promote-name PROMOTE_NAME | 22:30 |
| panda|ruck | you should not have used it in dlrnapi then :) | 22:30 |
| panda|ruck | and the full is needed, 8 digits are not enough | 22:30 |
| apevec | I think that old promotion was before we started using dlrnapi... | 22:30 |
| panda|ruck | maybe | 22:30 |
| apevec | ok, searching in gerrit finds this https://review.rdoproject.org/r/8516 | 22:30 |
| panda|ruck | it was 52 days old | 22:30 |
| apevec | so full distgit hash is 0468cca4637df0d7ac7158d6eb9b558bb6aeb9cf | 22:31 |
| apevec | panda|ruck, 6 | 22:31 |
| apevec | ^ | 22:31 |
| panda|ruck | apevec: thanks, reverting | 22:31 |
| apevec | alright, let's see how that travel back in time works | 22:32 |
| *** Son_Goku has joined #rdo | 22:32 | |
| apevec | panda|ruck, it should work, hash is in dlrnapi db https://trunk.rdoproject.org/api-centos-master-uc/api/report.html?package=python-networking-odl | 22:33 |
| apevec | it's just Sep 9 ... | 22:34 |
| *** scorcoran is now known as scorcoran_afk | 22:35 | |
| panda|ruck | apevec: done | 22:35 |
| rdogerrit | Alan Pevec created openstack/kolla-distgit rpm-master: Revert "Switch to python-cffi for rpm based distros" https://review.rdoproject.org/r/10367 | 22:36 |
| panda|ruck | reverting images now | 22:36 |
| *** Son_Goku has quit IRC | 22:36 | |
| apevec | ok, https://trunk.rdoproject.org/centos7-master/current-tripleo/ is back to Sep 9 | 22:36 |
| panda|ruck | images reverted too | 22:37 |
| panda|ruck | container images were not touched | 22:37 |
| apevec | https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-master-current-tripleo/209/ picked up new hash, I'll abort it | 22:38 |
| panda|ruck | yes | 22:38 |
| apevec | panda|ruck, alright, now when is the next run? | 22:38 |
| panda|ruck | apevec: 12AM UTC | 22:39 |
| panda|ruck | apevec: but as long as we don't trigger manually the pipeline, we will test a single hash | 22:39 |
| panda|ruck | and the promotion will regularly pass | 22:39 |
| panda|ruck | apevec: I added some explanation here https://trello.com/c/abKfxbUB/338-cixlp1728070tripleoci-promotions-continuously-tackled-in-the-last-yards-multiple-bugs | 22:40 |
| apevec | what do you mean? | 22:40 |
| apevec | yeah, I got that | 22:40 |
| apevec | it's either/or | 22:41 |
| panda|ruck | the problem is tripleo-ci-testing link | 22:41 |
| panda|ruck | we change it to the latest consistent at the start of the pipeline | 22:42 |
| apevec | ah that's single big lock then | 22:42 |
| panda|ruck | if we change it during a periodic run, half of the jobs will test a different hash | 22:42 |
| apevec | but we should be able to run in parallel, just need to come up with a better sync point | 22:42 |
| apevec | anyway, let's now wait for the next cron trigger | 22:43 |
| panda|ruck | hope the revert worked fully | 22:43 |
| apevec | afaict it did | 22:44 |
| apevec | symlink was updated and you checked images | 22:44 |
| panda|ruck | and nothing else is trying to pick up the broken hah | 22:44 |
| *** wolverineav has quit IRC | 22:44 | |
| apevec | I've also disabled https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-master-current-tripleo/ so it doesn't retrigger on reverted hash | 22:44 |
| panda|ruck | yep but I bet some gate in maste already picked the broken hash | 22:44 |
| panda|ruck | if it testes containers it will not pass | 22:44 |
| *** Son_Goku has joined #rdo | 22:45 | |
| panda|ruck | hopefully | 22:45 |
| apevec | I guess we could check with ES query | 22:45 |
| *** cylopez has joined #rdo | 22:45 | |
| apevec | but too late for me to think | 22:45 |
| panda|ruck | yep | 22:46 |
| panda|ruck | same here | 22:46 |
| panda|ruck | Maybe it's better to write an email on openstack-dev | 22:46 |
| *** cylopez has quit IRC | 22:46 | |
| panda|ruck | warning about the broken promotion | 22:46 |
| *** rlandy has quit IRC | 22:49 | |
| rdogerrit | Merged openstack/kolla-distgit rpm-master: Revert "Switch to python-cffi for rpm based distros" https://review.rdoproject.org/r/10367 | 22:51 |
| *** Son_Goku has quit IRC | 22:52 | |
| *** Son_Goku has joined #rdo | 22:57 | |
| *** jlabarre has quit IRC | 22:57 | |
| rdobot | [sensu] RESOLVED: master.monitoring.rdoproject.org - check-delorean-master-current @ http://tinyurl.com/y8hxewm2 |#| No build failures detected: http://trunk.rdoproject.org/centos7-master/report.html | 22:59 |
| *** jackNemrod has quit IRC | 23:00 | |
| *** apevec has quit IRC | 23:13 | |
| *** brault_ has joined #rdo | 23:18 | |
| *** brault has quit IRC | 23:18 | |
| *** gildub has joined #rdo | 23:25 | |
| *** scorcoran_afk is now known as scorcoran | 23:26 | |
| *** scorcoran is now known as scorcoran_afk | 23:26 | |
| *** vnogin has joined #rdo | 23:29 | |
| *** vnogin has quit IRC | 23:33 | |
| *** daidv has quit IRC | 23:35 | |
| *** daidv has joined #rdo | 23:35 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
