Monday, 2017-02-06

*** pmannidi has joined #tripleo00:03
*** pmannidi has quit IRC00:05
*** ooolpbot has joined #tripleo00:11
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION00:11
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/166165600:11
*** ooolpbot has quit IRC00:11
openstackLaunchpad bug 1661656 in tripleo "CI: promoted overcloud-full image is missing from the mirror" [Critical,Triaged]00:11
*** pmannidi has joined #tripleo00:11
*** ansmith has quit IRC00:15
*** michapma_alt has joined #tripleo00:19
*** colonwq has quit IRC00:49
*** colonwq has joined #tripleo00:49
*** dmacpher has quit IRC00:53
*** jistr has quit IRC01:00
*** jistr has joined #tripleo01:00
*** colonwq has quit IRC01:03
*** colonwq has joined #tripleo01:03
*** ooolpbot has joined #tripleo01:11
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION01:11
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/166165601:11
*** ooolpbot has quit IRC01:11
openstackLaunchpad bug 1661656 in tripleo "CI: promoted overcloud-full image is missing from the mirror" [Critical,Triaged]01:11
*** limao has joined #tripleo01:19
openstackgerritRico Lin proposed openstack/os-collect-config master: Allow setting minimum polling interval for collect  https://review.openstack.org/42082301:22
*** dmacpher has joined #tripleo01:34
openstackgerritMerged openstack/instack-undercloud master: Clear any previous environment in stackrc  https://review.openstack.org/42882101:41
openstackgerritMerged openstack/instack-undercloud master: Increase sync timeout for nova db syncs  https://review.openstack.org/42884301:41
openstackgerritMerged openstack/puppet-tripleo master: Revert "Revert "set innodb_file_per_table to ON for MySQL / Galera""  https://review.openstack.org/42884601:41
*** ooolpbot has joined #tripleo02:11
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION02:11
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/166165602:11
*** ooolpbot has quit IRC02:11
openstackLaunchpad bug 1661656 in tripleo "CI: promoted overcloud-full image is missing from the mirror" [Critical,Triaged]02:11
*** ooolpbot has joined #tripleo03:11
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION03:11
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/166165603:11
*** ooolpbot has quit IRC03:11
openstackLaunchpad bug 1661656 in tripleo "CI: promoted overcloud-full image is missing from the mirror" [Critical,Triaged]03:11
*** dhill_ has quit IRC03:26
*** dhill_ has joined #tripleo03:28
*** dhill_ has quit IRC03:29
*** sudipto has joined #tripleo03:52
*** sudipto_ has joined #tripleo03:52
*** flepied has joined #tripleo03:52
*** psahoo has joined #tripleo03:56
openstackgerritzhangyanxian proposed openstack/tripleo-ui master: Fix typo in NodesActions.tests.js  https://review.openstack.org/42948803:57
*** rhallisey has joined #tripleo03:57
openstackgerritzhangyanxian proposed openstack/tripleo-ui master: Fix typo in NodesActions.tests.js  https://review.openstack.org/42948803:58
*** sdake has joined #tripleo04:00
*** rhallisey has quit IRC04:01
*** nmathew has joined #tripleo04:01
*** links has joined #tripleo04:09
*** ooolpbot has joined #tripleo04:11
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION04:11
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/166165604:11
*** ooolpbot has quit IRC04:11
openstackLaunchpad bug 1661656 in tripleo "CI: promoted overcloud-full image is missing from the mirror" [Critical,Triaged]04:11
*** sudipto_ has quit IRC04:14
*** sudipto has quit IRC04:14
*** flepied has quit IRC04:14
*** jokke__ has quit IRC04:14
*** prateek has joined #tripleo04:18
*** dmacpher has quit IRC04:26
openstackgerritAnh Tran proposed openstack/instack-undercloud master: Typo fix: occured => occurred  https://review.openstack.org/42949704:28
*** sudipto has joined #tripleo04:40
*** sudipto_ has joined #tripleo04:40
*** david-lyle has joined #tripleo04:46
*** david-lyle has quit IRC04:50
*** david-lyle_ has joined #tripleo04:50
*** david-lyle_ is now known as david-lyle04:52
*** gkadam has joined #tripleo04:56
*** ooolpbot has joined #tripleo05:11
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION05:11
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/166165605:11
*** ooolpbot has quit IRC05:11
openstackLaunchpad bug 1661656 in tripleo "CI: promoted overcloud-full image is missing from the mirror" [Critical,Triaged]05:11
*** nyechiel has joined #tripleo05:14
*** sdake has quit IRC05:18
*** prateek_ has joined #tripleo05:19
*** prateek_ has quit IRC05:20
*** prateek_ has joined #tripleo05:20
*** limao has quit IRC05:21
*** david-lyle has quit IRC05:21
*** dmacpher has joined #tripleo05:22
*** limao has joined #tripleo05:22
*** dmacpher has quit IRC05:30
*** dhill_ has joined #tripleo05:40
*** ramishra has quit IRC05:46
*** limao has quit IRC05:48
*** twiggy_ has joined #tripleo05:51
*** limao has joined #tripleo05:52
*** twiggy has quit IRC05:54
*** ramishra has joined #tripleo05:59
*** nyechiel has quit IRC05:59
*** pgadiya has joined #tripleo06:04
*** limao has quit IRC06:09
*** limao has joined #tripleo06:09
*** ooolpbot has joined #tripleo06:11
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION06:11
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/166165606:11
openstackLaunchpad bug 1661656 in tripleo "CI: promoted overcloud-full image is missing from the mirror" [Critical,Triaged]06:11
*** ooolpbot has quit IRC06:11
*** limao has quit IRC06:14
*** links has quit IRC06:19
*** limao has joined #tripleo06:32
*** links has joined #tripleo06:35
*** remix_tj has quit IRC06:39
*** yolanda has joined #tripleo06:43
*** remix_tj has joined #tripleo06:43
*** ealcaniz has joined #tripleo06:43
*** ealcaniz has quit IRC06:47
*** ealcaniz has joined #tripleo06:51
openstackgerritNam Nguyen Hoai proposed openstack/diskimage-builder master: Fix typo in README.rst  https://review.openstack.org/42953006:52
*** iranzo has joined #tripleo06:59
*** links has quit IRC07:05
*** agurenko has joined #tripleo07:12
*** tesseract has joined #tripleo07:12
*** limao has quit IRC07:16
*** links has joined #tripleo07:17
*** limao has joined #tripleo07:17
openstackgerritvinay kumar muddu proposed openstack/diskimage-builder master: Add support for building images capable of UEFI  https://review.openstack.org/28778407:19
*** saneax-_-|AFK is now known as saneax07:21
*** limao has quit IRC07:21
*** jprovazn has joined #tripleo07:24
*** florianf has joined #tripleo07:24
*** dsneddon_afk has joined #tripleo07:35
*** limao has joined #tripleo07:36
*** ramishra has quit IRC07:40
*** abehl has joined #tripleo07:40
*** rasca has joined #tripleo07:41
*** ramishra has joined #tripleo07:41
*** artom_ has joined #tripleo07:42
*** artom has quit IRC07:44
*** tremble has joined #tripleo07:44
*** tremble has joined #tripleo07:44
*** ebarrera has joined #tripleo07:47
*** ccamacho has joined #tripleo07:48
*** aufi has joined #tripleo07:50
*** pcaruana has joined #tripleo07:55
*** fzdarsky has joined #tripleo07:56
*** apetrich has quit IRC07:59
*** apetrich has joined #tripleo08:02
*** pcaruana has quit IRC08:07
*** pcaruana has joined #tripleo08:07
*** brault has joined #tripleo08:09
*** ooolpbot has joined #tripleo08:11
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION08:11
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/166165608:11
*** ooolpbot has quit IRC08:11
openstackLaunchpad bug 1661656 in tripleo "CI: promoted overcloud-full image is missing from the mirror" [Critical,Triaged]08:11
cmyster/6/1308:14
*** bvandenh has joined #tripleo08:18
*** yprokule has joined #tripleo08:18
*** jlinkes has joined #tripleo08:21
*** owalsh-afk has joined #tripleo08:22
*** nrado has joined #tripleo08:23
*** nrado has left #tripleo08:24
*** hjensas has joined #tripleo08:24
*** hjensas has joined #tripleo08:24
*** panda|Zz is now known as panda08:25
*** gkadam is now known as gkadam|afk08:25
pandasshnaidm: you know how to access the mirror server ?08:30
sshnaidmpanda, yep08:30
*** zoli is now known as zoli|wfh08:31
sshnaidmpanda, trying to understand what happened there08:32
pandasshnaidm: I think it's a permission problem, last promotion did not change the link. I remember Ben saying "I copyied the file as root, but it doesn't matter"08:34
pandamaybe it does08:34
sshnaidmpanda, I'll change it, but it seems like it really shouldn't matter08:34
zoli|wfhgood morning08:35
sshnaidmpanda, now it's apache08:36
sshnaidmpanda, but what is the problem? is it wrong link or not?08:36
sshnaidmpanda, doesn't it download?08:36
*** ebarrera has quit IRC08:37
ccamachomorning guys!08:38
arxcruzmorning08:38
pandasshnaidm: if you look here http://66.187.229.139/builds/?C=M;O=A08:38
pandasshnaidm: you should see that most of the promotion jobs worked, there are newer uploads than the one poited by current-tripleo08:39
pandabut the current-tripleo link is not updated to one of those successful image uploads08:39
sshnaidmpanda, it may be one worked, not both08:39
pandasshnaidm: for tdaoy for example I see trhee new uploads, and since a job uploads only whe it passes, I think they all passed08:40
sshnaidmpanda, yeah, I see it in status page..\08:41
sshnaidmpanda, wait, when bnemec di change the link?08:43
pandasshnaidm: I don't think he changed the link, he just copied an older imaege to the last position08:44
*** links has quit IRC08:45
*** links has joined #tripleo08:45
*** amoralej|off is now known as amoralej08:47
sshnaidmpanda, http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha/cf431e5/console.html#_2017-02-05_08_15_45_69412208:48
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates master: Delivers upgrade scripts where upgrade steps are disabled  https://review.openstack.org/41988608:48
sshnaidmpanda,  it's the same dlrn hash link that we have in /builds: http://66.187.229.139/builds/51/95/51953c10fff7c664c28c24d13e3f3bf3a1ee6251_5770f09c/08:49
sshnaidmpanda, it didn't change in last 3 days08:49
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates master: Delivers upgrade scripts where upgrade steps are disabled  https://review.openstack.org/41988608:50
mariosmatbu: can you check it when you get a chance please, i rebased onto your latest version  ^^^ and wasn't clean (finally i can try it today assuming no more escalations)08:50
*** limao has quit IRC08:51
*** limao has joined #tripleo08:52
*** jpich has joined #tripleo08:52
pandasshnaidm: so, wwe've been tested always the same hash, the scripts recognize that, and doesn't update the link to the promoted image ?08:54
*** lifeless has quit IRC08:54
pandahow's it possible that we are testing the same hash ?08:54
*** karimb has joined #tripleo08:55
sshnaidmpanda, if it doesn't change, that's possible08:55
sshnaidmpanda, but worth to ask on #rdo08:55
*** lifeless has joined #tripleo08:55
*** b00tcat has joined #tripleo08:56
pandasshnaidm:  I see the ironic inspector is failing to build since the Feb 308:56
pandasshnaidm: that's why consistent is not updated08:56
sshnaidmpanda, yeah, the last link is from 3 Feb08:56
*** limao has quit IRC08:57
pandasshnaidm: so, the best thing to do is make the current-tripleo point to http://66.187.229.139/builds/34/54/3454a21219eefdb526deb621f5ec42587d1b7b19_c3272afb/ ?08:57
sshnaidmpanda, why?08:59
*** zzzeek has quit IRC09:00
*** zzzeek has joined #tripleo09:00
sshnaidmpanda, it's periodic-tripleo-ci-centos-7-ovb-ha-ocata job09:00
*** mcornea has joined #tripleo09:02
*** pgadiya has quit IRC09:02
*** paramite has joined #tripleo09:03
*** pgadiya has joined #tripleo09:04
pandawhy is ocata uploading ?09:06
sshnaidmpanda, every periodic job uploads09:07
sshnaidmpanda, every successful09:07
sshnaidmpanda, we need to promote for all branches09:07
*** ooolpbot has joined #tripleo09:11
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION09:11
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/166165609:11
*** ooolpbot has quit IRC09:11
openstackLaunchpad bug 1661656 in tripleo "CI: promoted overcloud-full image is missing from the mirror" [Critical,Triaged]09:11
b00tcatHi, in T-H-T, is there any docs about the syntax for the firewall rules? I would need to know if there is any way to allow traffic on specific interfaces, and so on09:11
*** suuuper has joined #tripleo09:24
pandasshnaidm: ok, there's something really big missing here09:24
pandaand I'm not sure how many poeople know how this work. I didn't09:24
*** lucasagomes has joined #tripleo09:25
mariosb00tcat: hey, i just had a quick poke as haven't looked for a while... i am not aware of docs or at least couldn't quickly find at http://docs.openstack.org/developer/tripleo-docs/09:25
mariosb00tcat: but poking at puppet-tripleo it seems there is a https://github.com/openstack/puppet-tripleo/blob/master/manifests/firewall/rule.pp#L74 interface param to the rule class09:26
sshnaidmpanda, I don't know why EmilienM wrote it's mitaka image09:26
mariosb00tcat: the rules themselves are distributed to the service manifests, as one example see https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/neutron-api.yaml#L14709:26
*** sshnaidm is now known as sshnaidm|afk09:27
pandasshnaidm|afk: maybe because he lokked at the packages inside09:27
mariosb00tcat: so you should be able to pass the iniface to any rule you're declaring, anyway this is just from poking so your mileage may vary, but FYI ;)09:27
pandasshnaidm|afk: anyway, the canusecache function states that we are not storing artifacts for stable images09:27
pandasshnaidm|afk: and there is no trigger to automatically promote dlrn hashes for stable branches09:28
*** yprokule_ has joined #tripleo09:28
pandain fact, I think that nothing is even aware that we are uploading images from the periodic stable branches jobs09:29
b00tcatmarios: good, thanks! i didn't really know what was being invoked when declaring *.firewall_rules in the t-h-t yaml files09:29
*** yprokule has quit IRC09:29
*** yprokule_ is now known as yprokule09:29
*** karimb has quit IRC09:31
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart master: Fix and improve flake8 test.  https://review.openstack.org/42868909:31
openstackgerritFlorian Fuchs proposed openstack/tripleo-ui master: Add i18n to Notifications  https://review.openstack.org/41946909:32
*** bogdando has joined #tripleo09:34
*** links has quit IRC09:35
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates master: Add monitoring and logging agents deployment to CI  https://review.openstack.org/41021009:41
*** gkadam|afk is now known as gkadam09:45
*** cylopez has joined #tripleo09:46
mariosb00tcat: cool hope it helps as a start anyway09:46
openstackgerritFlorian Fuchs proposed openstack/tripleo-ui master: Adds language selector to NavBar  https://review.openstack.org/42467609:46
openstackgerritmathieu bultel proposed openstack/tripleo-heat-templates master: Add nova service support for composable upgrades  https://review.openstack.org/40524109:47
*** derekh has joined #tripleo09:50
*** karimb has joined #tripleo09:52
*** gfidente has joined #tripleo09:52
*** gfidente has quit IRC09:52
*** gfidente has joined #tripleo09:52
*** sshnaidm|afk is now known as sshnaidm09:53
*** yolanda has quit IRC09:56
*** yolanda has joined #tripleo09:57
*** yprokule has quit IRC09:59
openstackgerritJulie Pichon proposed openstack/tripleo-specs master: Add i18n bug tag  https://review.openstack.org/42961310:00
*** yprokule has joined #tripleo10:03
*** salmankhan has joined #tripleo10:05
*** ooolpbot has joined #tripleo10:11
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION10:11
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/166165610:11
*** ooolpbot has quit IRC10:11
openstackLaunchpad bug 1661656 in tripleo "CI: promoted overcloud-full image is missing from the mirror" [Critical,Triaged]10:11
*** dhill_ has quit IRC10:13
*** dtantsur|afk is now known as dtantsur10:14
*** pblaho has joined #tripleo10:15
*** owalsh-afk is now known as owalsh10:16
*** jokke_ has joined #tripleo10:16
*** dhill_ has joined #tripleo10:17
sshnaidmpanda, then what does "mitaka packaging" mean at https://dashboards.rdoproject.org/rdo-dev ?10:17
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci master: Enable pacemaker for upgrade job  https://review.openstack.org/42962710:18
*** akrivoka has joined #tripleo10:19
pandasshnaidm: better ask #rdo10:21
openstackgerritmathieu bultel proposed openstack/tripleo-heat-templates master: Add pacemaker services for composable upgrade scenario  https://review.openstack.org/42963010:22
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci master: Enable pacemaker for upgrade job  https://review.openstack.org/42962710:24
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/mitaka: Ensure working compute node during the whole migration.  https://review.openstack.org/42809310:24
*** yolanda has quit IRC10:26
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates master: Delivers upgrade scripts where upgrade steps are disabled  https://review.openstack.org/41988610:28
*** yolanda_ has joined #tripleo10:29
*** pgadiya has quit IRC10:32
*** salmankhan has quit IRC10:39
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/mitaka: Ensure working compute node during the whole migration.  https://review.openstack.org/42809310:43
*** pgadiya has joined #tripleo10:46
*** salmankhan has joined #tripleo10:52
*** nrado has joined #tripleo10:53
*** dbecker has joined #tripleo10:56
*** ccamacho is now known as ccamacho|lunch10:58
*** nyechiel has joined #tripleo11:00
*** karimb has quit IRC11:07
nradoHi,11:08
nradowhat is the "local" collector I found in my os-collect-config.conf:11:08
nrado[DEFAULT]11:08
nradocommand = os-refresh-config11:08
nradocollectors = ec211:08
nradocollectors = heat11:08
nradocollectors = local11:08
*** ooolpbot has joined #tripleo11:11
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION11:11
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/166165611:11
*** ooolpbot has quit IRC11:11
openstackLaunchpad bug 1661656 in tripleo "CI: promoted overcloud-full image is missing from the mirror" [Critical,Triaged]11:11
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates master: Add monitoring and logging agents deployment to CI  https://review.openstack.org/41021011:15
*** dsariel has quit IRC11:17
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Remove legacy major upgrade scripts for Ceph and BlockStorage  https://review.openstack.org/42691911:17
*** ealcaniz has quit IRC11:17
*** nmathew has quit IRC11:20
*** links has joined #tripleo11:21
*** mburned is now known as mburned_out11:23
*** nyechiel has quit IRC11:24
*** links has quit IRC11:25
*** sudipto_ has quit IRC11:25
*** sudipto has quit IRC11:25
*** zoli|wfh is now known as zoli|lunch11:26
openstackgerritMerged openstack-infra/tripleo-ci master: Fix floating IP tenant stack template path  https://review.openstack.org/42772511:26
mcorneamatbu: marios hey,do we move nova api under httpd during undercloud upgrade?11:28
*** nyechiel has joined #tripleo11:28
mariosmcornea: i was not aware of that change... vaguely remember jaosorior landing something relevant would have to check. is that what you noticed?11:29
mariosmcornea: /me just re-running undercloud ugprade again on fresh OSPvanila11:29
marios(osp10)11:29
mariosmcornea: this looks relevant https://review.openstack.org/#/c/376267/11:30
mariosmcornea: so the answer is probably yes11:30
mcorneamarios: I noticed Listen 192.168.0.1:8774 in /etc/httpd/conf/ports.conf and in one of my attempts nova-api failed to start because http was listening on that port11:31
mariosmcornea: cool so yeah we should at least document it11:33
mcorneamarios: hm I wonder why CI doesn't hit this issue11:34
*** karimb has joined #tripleo11:35
*** jkilpatr has quit IRC11:40
pandasshnaidm: if there is not eligible image to put there, maybe the best thing to do right now it to remove the image11:41
*** flepied has joined #tripleo11:42
pandathe job will take longer, but  they will not use a wrong image11:42
sshnaidmpanda, but what is the problem with image?11:42
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: List nova instances after overcloud deploy  https://review.openstack.org/42818711:43
*** dtantsur is now known as dtantsur|lunch11:48
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/mitaka: Ensure working compute node during the whole migration.  https://review.openstack.org/42809311:48
openstackgerritArx Cruz proposed openstack/tripleo-specs master: Tool to send tempest report by mail  https://review.openstack.org/42087811:49
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/mitaka: Ensure working compute node during the whole migration.  https://review.openstack.org/42809311:52
*** lmasiero has joined #tripleo11:54
*** artom_ is now known as artom11:55
*** pkovar has joined #tripleo11:58
lmasieromorning guys.. does anyone knows if simples tripleo deployments (1 compute, 1 controller) using default templates @ /usr/share/openstack-tripleo-heat-templates is supported?12:01
lmasieroi'm having some issues when running the heat stack: There are not enough hosts available., Code: 50012:01
*** dsariel has joined #tripleo12:02
*** mburned_out is now known as mburned12:03
*** prateek_ has quit IRC12:03
matbumcornea: marios idk, to be honest, i didn't pay much attention of the UC upgrade those days12:03
*** dmacpher has joined #tripleo12:03
matbutrying ot figure out a full upgrade with all services12:03
pandasshnaidm: look at this gate http://logs.openstack.org/02/427802/10/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/9c14a5c/logs/compute-0-tripleo-ci-a-test/var/log/host_info.txt.gz12:04
pandasshnaidm: this is supposed to test master12:04
pandasshnaidm: but the packages are from mitaka12:04
pandasshnaidm: ben copied a mitaka image on the link12:04
*** ansmith has joined #tripleo12:05
*** nyechiel has quit IRC12:09
sshnaidmpanda, I think we update images https://github.com/openstack-infra/tripleo-ci/blob/f11aa57461cb3db1f7011105aa5dad382a33440a/scripts/common_functions.sh#L76-L7612:09
*** prateek has quit IRC12:10
*** dtantsur|lunch is now known as dtantsur12:10
sshnaidmpanda, copying the repos from /etc/yum.repos.d to image and running update there: sudo chroot $MOUNTDIR /bin/yum update -y12:10
sshnaidmpanda, how do you see it's mitaka packages?12:11
*** ooolpbot has joined #tripleo12:11
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION12:11
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/166165612:11
*** ooolpbot has quit IRC12:11
openstackLaunchpad bug 1661656 in tripleo "CI: promoted overcloud-full image is missing from the mirror" [Critical,Triaged]12:11
pandasshnaidm: https://releases.openstack.org/teams/nova.html12:11
pandasshnaidm: openstack-nova release 13.x12:11
*** ebarrera has joined #tripleo12:11
*** jkilpatr has joined #tripleo12:12
*** dobson has quit IRC12:12
sshnaidmpanda, ok.. the repos on undercloud are correct: http://logs.openstack.org/02/427802/10/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/9c14a5c/console.html#_2017-02-03_21_39_20_96416812:13
sshnaidmpanda, so problem is with updating..12:13
pandabecause undercloud is not taken from the image12:14
*** sudipto has joined #tripleo12:14
*** sudipto_ has joined #tripleo12:14
pandawe are using overcloud mitaka images12:14
*** ansmith has quit IRC12:14
*** dobson has joined #tripleo12:15
sshnaidmpanda, what are you talking about now? tripleo-quickstart job?12:16
sshnaidmpanda, I see updating worked, but nova wasn't updated: http://logs.openstack.org/02/427802/10/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/9c14a5c/console.html#_2017-02-03_21_42_03_81970012:16
*** flepied has quit IRC12:17
pandasshnaidm: let me recap what happened friday. There was a promotion, the image upload failed because there was no space left to store it, we remained sevelar hourse without overcloud image, then ben freed some space, and copied an old image to the current-tripleo link. Wrong image, because it's a mitaka image, probably because ben did not know that we are uploading mitaka images too. We have now an overcloud12:17
pandamitaka image in the mirror server, where a master image it's supposed to be.12:17
pandaseveral hours*12:17
EmilienMhello12:17
*** mburned is now known as mburned_out12:17
*** ebarrera has quit IRC12:17
*** yolanda_ has quit IRC12:18
sshnaidmpanda, I try to understand why it should cause problem if we anyway copy our repos there and update, it shouldn't matter which image we have as I understand it now, or I miss something12:18
EmilienMsshnaidm: I said it's Mitaka because if you look at ovb/overcloud CI jobs, you'll see mitaka versions of openstack12:18
*** paramite has quit IRC12:18
sshnaidmEmilienM, yeah, I saw this, so we don't update them well, I try to understand why. We replace repo files on each image and run yum update there, it should update packages to master12:19
sshnaidmbut it seems like doesn't12:20
*** bogdando has quit IRC12:21
sshnaidmI see only puppet modules are updated12:21
*** ccamacho|lunch is now known as ccamacho12:21
*** bogdando has joined #tripleo12:21
sshnaidmhttp://logs.openstack.org/02/427802/10/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/9c14a5c/console.html#_2017-02-03_21_41_38_51048412:21
sshnaidmEmilienM, panda seems like all previous master images were deleted on mirror server, so as workaround I can just delete/rename the current image as panda proposed, so each job will build them, and run the promote job now12:23
*** yolanda_ has joined #tripleo12:23
sshnaidmuntil I understand why it matters which image to take..12:23
pandasshnaidm: the manual promotion will not work either, the consistent hash is not changed yet12:23
sshnaidmpanda, I'll delete all this directory then12:23
sshnaidmpanda, then it'll create it from scratch12:24
EmilienMsshnaidm: ok12:25
*** dougbtv has joined #tripleo12:27
*** yolanda_ has quit IRC12:27
*** yolanda_ has joined #tripleo12:28
pandasshnaidm:  in any case, I think a package updated from mitaka to master would take so long that the jobs would timeout too12:28
pandafull package update*12:28
sshnaidmpanda, they don't update to master unfortunately12:28
pandasshnaidm: yes, I don't know why, but even if they could, they'd have to update everything, taking a lot of time12:29
sshnaidmpanda, only openstack packages, and anyway less time then image building12:29
*** paramite has joined #tripleo12:30
*** saneax is now known as saneax-_-|AFK12:34
*** fultonj has joined #tripleo12:35
sshnaidmpanda, EmilienM image packages are not updated because copying repo files to it fails (no idea why), controller and computes don't have any delorean repo files on them12:37
*** owalsh is now known as owalsh-afk12:37
EmilienMsshnaidm: I have zero clue at this time12:38
sshnaidmEmilienM, it's a bug, I'll start reproducing it..12:38
pandasshnaidm: wait12:40
pandasshnaidm: the repos are copied successfully, and ther are deleted12:40
pandasshnaidm: we don't want overcloud to have access to any repo12:41
pandasshnaidm: that's why you are not finding them12:41
sshnaidmpanda, oh, you're right12:41
sshnaidmpanda, ok, will try to find out why it doesn't update to master..12:42
sshnaidmEmilienM, panda I've renamed file in current-tripleo folder, so each job should try to build image now12:42
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs  https://review.openstack.org/35921512:43
sshnaidmand let's try to promote12:43
EmilienMok12:43
openstackgerritM V P Nitesh proposed openstack/tripleo-common master: Use oslo.log library instead of system logging module  https://review.openstack.org/42967712:44
*** karimb has quit IRC12:45
derekhpanda: just reading back, so you've kicked off a new promote job?12:46
pandaderekh: sshnaidm did12:47
sshnaidmderekh, yes12:48
sshnaidmderekh, but I wonder why mitaka image with repos of master don't update its packages to master when running yum update12:48
*** lucasagomes is now known as lucas-hungry12:49
derekhsshnaidm: ya, wondering that too12:49
*** limao has joined #tripleo12:52
pandaNot sure how much an image create with mitaka tools would work with a master deployment, even after updating12:53
derekhpanda: ya, your probably correct but it would be nice to understand why12:54
derekhlogs arn't very helpfull12:55
derekh2017-02-06 04:39:34.538784 | No packages marked for update12:55
*** zoli|lunch is now known as zoli|wfh12:55
EmilienMsshnaidm, panda: in my knowledge, promotion should work, as we still pin nova12:55
sshnaidmderekh, it updates puppet modules only: http://logs.openstack.org/02/427802/10/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/9c14a5c/console.html#_2017-02-03_21_41_42_85352212:58
openstackgerritBrad P. Crochet proposed openstack/tripleo-common master: Implement check ironic boot configuration as a custom action  https://review.openstack.org/42822512:58
openstackgerritBrad P. Crochet proposed openstack/tripleo-common master: Implement check flavors as a custom action  https://review.openstack.org/42817812:58
openstackgerritBrad P. Crochet proposed openstack/tripleo-common master: Implement verify profiles as a custom action  https://review.openstack.org/42968412:58
*** flepied has joined #tripleo13:00
*** jayg|g0n3 is now known as jayg13:01
*** rbrady has joined #tripleo13:02
*** pgadiya has quit IRC13:07
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test ci  https://review.openstack.org/42968713:07
*** rhallisey has joined #tripleo13:08
*** mburned_out is now known as mburned13:10
*** ooolpbot has joined #tripleo13:11
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION13:11
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/166165613:11
*** ooolpbot has quit IRC13:11
openstackLaunchpad bug 1661656 in tripleo "CI: promoted overcloud-full image is missing from the mirror" [Critical,Triaged]13:11
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/mitaka: Ensure working compute node during the whole migration.  https://review.openstack.org/42809313:12
*** dprince has joined #tripleo13:15
*** salmankhan has quit IRC13:15
*** salmankhan has joined #tripleo13:15
dtantsurhey folks, may I get a 2nd +2 on https://review.openstack.org/#/c/428052/ please? this is quite important to get in13:16
openstackgerritBrad P. Crochet proposed openstack/tripleo-common master: Implement verify profiles as a custom action  https://review.openstack.org/42968413:17
openstackgerritBrad P. Crochet proposed openstack/tripleo-common master: Implement check ironic boot configuration as a custom action  https://review.openstack.org/42822513:17
openstackgerritBrad P. Crochet proposed openstack/tripleo-common master: Implement check flavors as a custom action  https://review.openstack.org/42817813:17
openstackgerritBrad P. Crochet proposed openstack/tripleo-common master: Implement check boot images as a custom action  https://review.openstack.org/42817713:17
EmilienMdtantsur: done13:19
dtantsurthnx13:19
EmilienMsshnaidm: can you kickoff promotion job again? ovb-ha failed on the random ssh error13:21
EmilienMhttp://logs.openstack.org/15/359215/49/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/38bca4b/logs/undercloud/var/log/undercloud_install.txt.gz#_2017-02-06_13_05_18_00013:21
EmilienMpanda: do we have news on the ssh issue?13:21
*** salmankhan has quit IRC13:24
*** amoralej is now known as amoralej|lunch13:26
*** karimb has joined #tripleo13:26
*** jcoufal has joined #tripleo13:27
*** gkadam has quit IRC13:27
*** cdearborn has joined #tripleo13:27
pandaEmilienM: the problem is that we don't have an acceptable workaround. The issue has high attention, but until the problem is fixed, there is not much we can do. After it is fixed, we'll try to push it everywhere as fast as we can do.13:28
openstackgerritMarius Cornea proposed openstack/instack-undercloud master: Remove backslash from awk command in stackrc  https://review.openstack.org/42969913:29
EmilienMpanda: can we build an rpm with the fix ourselves and deploy it in CI?13:29
*** jpena is now known as jpena|lunch13:31
*** jeckersb_gone is now known as eck`13:34
EmilienMsshnaidm: recheck on https://review.openstack.org/#/c/359215/ won't work13:35
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart master: Adds a Bashate target to tox.ini.  https://review.openstack.org/42970313:35
EmilienMsshnaidm: because jobs are still runnin13:35
EmilienMsshnaidm: you need to add something in code or commit message13:35
sshnaidmEmilienM, oh, right, will do now13:35
jpichflorianf: Yaaay the navbar is merging \o/13:36
*** sdake has joined #tripleo13:37
florianfjpich: woohooo!13:37
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs  https://review.openstack.org/35921513:38
*** pradk has joined #tripleo13:39
*** ansmith has joined #tripleo13:39
*** dcritch has joined #tripleo13:39
*** rlandy has joined #tripleo13:40
*** sdake_ has joined #tripleo13:40
*** rlandy has joined #tripleo13:41
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test OOOQ jobs  https://review.openstack.org/42970613:42
*** ealcaniz has joined #tripleo13:42
*** nyechiel has joined #tripleo13:43
pradkplease review this trivial change -  https://review.openstack.org/#/c/428420/13:43
*** sdake has quit IRC13:44
*** jmelvin has joined #tripleo13:45
*** fultonj has quit IRC13:52
*** fultonj has joined #tripleo13:53
*** limao has quit IRC13:54
*** limao has joined #tripleo13:55
*** tosky has joined #tripleo13:56
*** tzumainn has joined #tripleo13:58
*** limao has quit IRC13:58
*** limao has joined #tripleo13:59
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix updating images  https://review.openstack.org/42971314:00
sshnaidmEmilienM, panda found why it didn't update ^^14:00
sshnaidmderekh, ^^14:00
*** psahoo has quit IRC14:01
*** lucas-hungry is now known as lucasagomes14:01
EmilienMsshnaidm: woot14:02
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart-extras master: Adds a Bashate target to tox.ini.  https://review.openstack.org/42971514:02
*** sasha2 has joined #tripleo14:02
openstackgerritmathieu bultel proposed openstack/tripleo-quickstart-extras master: Implement overcoud sanity checks role  https://review.openstack.org/42971614:03
*** nyechiel has quit IRC14:04
*** jprovazn has quit IRC14:05
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart-extras master: Fix and improve flake8 test.  https://review.openstack.org/42871014:05
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart-extras master: Adds a Bashate target to tox.ini.  https://review.openstack.org/42971514:05
*** dtantsur is now known as dtantsur|brb14:06
EmilienMderekh, dprince: can you look https://review.openstack.org/#/c/429713/ when you can please?14:06
*** saneax-_-|AFK is now known as saneax14:10
*** dsavineau has quit IRC14:10
*** dsavineau has joined #tripleo14:10
*** ooolpbot has joined #tripleo14:11
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION14:11
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/166165614:11
*** ooolpbot has quit IRC14:11
openstackLaunchpad bug 1661656 in tripleo "CI: promoted overcloud-full image is missing from the mirror" [Critical,Triaged]14:11
*** owalsh-afk has quit IRC14:14
*** nyechiel has joined #tripleo14:15
*** dmacpher has quit IRC14:15
*** owalsh-afk has joined #tripleo14:15
openstackgerritPetr Kovar proposed openstack/tripleo-docs master: Fix all sphinx errors & warnings  https://review.openstack.org/42903814:16
*** owalsh-afk is now known as owalsh14:16
openstackgerritMerged openstack/tripleo-heat-templates master: Provide a default value for Ironic cleaning_network configuration  https://review.openstack.org/42805214:16
jpichhonza, jtomasek, florianf: Do you guys use fancy scripts to review or the gerrit UI? Would it be a huge stretch to say people should still look at a Workflow-1 patch if the commit message says "New dependency"?14:17
*** morazi has joined #tripleo14:18
*** chlong has joined #tripleo14:18
jtomasekjpich: gerrit UI, and to pull the patch, git review -d <patchnumber>14:19
jpichhonza, jtomasek, florianf: Also, I like jtomasek's idea of re-explaining the reason for the -1 (+ linking to our deps change doc?) to give reasons :)14:19
jpichjtomasek: Likewise! I know some people have complicated workflows that automatically remove patches with -1s in reviews or workflows to be more efficient14:20
jpichIf most reviewers still have visibility into in-progress patches we should be ok14:20
honzaone of these days i'll write an emacs mode for gerrit but until then i basically use what jtomasek described14:21
*** yolanda_ has quit IRC14:21
*** yolanda_ has joined #tripleo14:21
*** salmankhan has joined #tripleo14:21
*** eggmaster has joined #tripleo14:23
florianfjpich: same here: review -d  and UI14:23
jpichCool, so everybody sees all patches...14:24
florianfjpich, jtomasek, honza: I guess -1 is still the most effective way to prevent a dependency patch from being merged before we have a review from packaging. True, it might collide with some people habit to ignore -1 patches, but unlike draft patches (review -D) -1s should still be visible to everyone.14:25
*** amoralej|lunch is now known as amoralej14:26
florianfBut I'd say usually new dependencies involves some poking-for-packaging-reviews anyway14:27
*** jprovazn has joined #tripleo14:27
dprinceEmilienM: +2 on the patch14:28
openstackgerritmathieu bultel proposed openstack/tripleo-quickstart-extras master: Implement overcoud sanity checks role  https://review.openstack.org/42971614:28
openstackgerritmathieu bultel proposed openstack/tripleo-quickstart-extras master: Implement overcloud sanity checks role  https://review.openstack.org/42971614:28
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: Add ci-script for the OVB upgrade gate jobs  https://review.openstack.org/42694714:28
jpichflorianf: Yes, agreed. Both poking packagers, and poking core reviewers once that's done :) So reviewers don't have to keep checking before packaging gave their thumbs-up14:28
*** liverpooler has joined #tripleo14:30
openstackgerritMerged openstack/tripleo-ui master: Adds language selector to NavBar  https://review.openstack.org/42467614:30
jpichhonza: Did you feel your patch was getting ignored because of -1? Since we changed a dependency completely before the other one merged I thought that was going well :)14:30
honzajpich: Not really --- I just thought there was some urgency to that patch given that it blocks the release notes patch so I didn't want it to get unnecesarily delayed.14:33
honzajpich: I don't follow the second sentence.14:33
jpichhonza: You need to remind reviewers of that context in review comments or on IRC, I don't think the deps patch is being ignored. Apologies for the awkward 2nd sentence, I meant: if we'd merge it quickly, it would have had to be backed out anyway because we replaced the dependency with something else14:35
*** jpena|lunch is now known as jpena14:35
jpichhonza: Also I don't think the deps patch will be reviewed until it has ack from packagers, as well?14:35
honzajpich: I think we should make sure we in fact want that change before involving rdo?14:36
jpichhonza: Ok, I agree! So... I actually don't understand what problem we are trying to fix, when you suggested we may have to rethink the approval process?14:37
openstackgerritMerged openstack/tripleo-ui master: Fix typo in NodesActions.tests.js  https://review.openstack.org/42948814:37
honzajpich: The problem is that we're possibly overloading the meaning of "Workflow: -1".  But it sounds like it's not a big deal.14:38
weshaymatbu++14:42
weshaythanks for starting the sanity check14:42
jpichhonza: I see! I think the answer might be "yes" to both, for now :-)14:44
*** salmankhan has quit IRC14:45
sasha2EmilienM: hi, not sure if you saw the update on https://bugs.launchpad.net/tripleo/+bug/166173614:47
openstackLaunchpad bug 1661736 in tripleo "Error: neutron-db-manage upgrade heads returned 1 instead of one of [0] during undercloud deployment" [Undecided,New]14:47
EmilienMsasha2: I saw it14:48
*** salmankhan has joined #tripleo14:48
matbuweshay: np, i didn't test yet, my deployments are all broken, but i'll14:49
*** sdake_ is now known as sdake14:49
sasha2EmilienM: ok, thanks.14:49
*** numans has joined #tripleo14:52
openstackgerritMerged openstack/tripleo-quickstart-extras master: Remove unused undercloud_post_install_script and its log  https://review.openstack.org/42696715:02
*** dtantsur|brb is now known as dtantsur|mtg15:03
*** milan has joined #tripleo15:03
jtomasekhonza: so I've asked dtantsur @ #openstack-ironic on whether capabilities without value are supported and answer was that such capabilities are not supported and https://github.com/openstack/ironic/blob/77bcccf98c99eecc22b97c73d14ff5d1bdce7e8d/ironic/drivers/utils.py#L149-L156 should be used as a canonical implementation15:04
honzajtomasek: awesome! thanks for following up15:05
jtomasekhonza: when parsing, it removes any incomplete capabilities15:05
jtomasekhonza: I think this also means that capabilities which have value set to undefined/null/empty string should be removed when turned into string15:07
jtomasekhonza: what do you thinkg?15:08
jtomasek*think15:08
openstackgerrityolanda.robla proposed openstack/tripleo-quickstart master: Do not check for nested params if virt is not enabled  https://review.openstack.org/42921215:08
openstackgerritmathieu bultel proposed openstack/tripleo-heat-templates master: Add nova service support for composable upgrades  https://review.openstack.org/40524115:08
jtomasekhonza: also this: https://github.com/openstack/ironic/blob/77bcccf98c99eecc22b97c73d14ff5d1bdce7e8d/ironic/drivers/utils.py#L232-L25415:10
*** bogdando has quit IRC15:10
*** ooolpbot has joined #tripleo15:11
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION15:11
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/166165615:11
*** ooolpbot has quit IRC15:11
openstackLaunchpad bug 1661656 in tripleo "CI: promoted overcloud-full image is missing from the mirror" [Critical,Triaged]15:11
openstackgerritmathieu bultel proposed openstack/tripleo-heat-templates master: Add nova service support for composable upgrades  https://review.openstack.org/40524115:14
honzajtomasek: yep, you're right --- do you want to comment on the patch?  i'll update my vote15:14
*** ramishra has quit IRC15:15
jtomasekhonza: ok, I'll add the comment to the patch15:15
openstackgerritMarius Cornea proposed openstack/instack-undercloud master: Remove backslash from awk command in stackrc  https://review.openstack.org/42969915:16
EmilienMowalsh: regarding https://github.com/openstack/nova/blob/f40467b0eb2b58a369d24a0e832df1ace6c400c3/nova/conf/scheduler.py#L138-L14315:16
owalshEmilienM: ack15:16
jtomasekhonza: thanks for a review, I'll still update the patch to have regexp with * instead of +15:16
EmilienMowalsh: I wonder if we should run "nova-manage cell_v2 discover_hosts" in a workflow when adding nodes in ironic15:16
EmilienMowalsh: so we don't have to use discover_hosts_in_cells_interval15:17
honzajtomasek: np15:17
owalshEmilienM: we do15:17
*** yamahata has joined #tripleo15:17
*** ramishra has joined #tripleo15:17
owalshEmilienM: https://review.openstack.org/42753415:17
EmilienMowalsh: yeah, but even when new does are added at day 2?15:17
EmilienMsounds like yes, indeed15:18
owalshEmilienM: the provide workflow does this, so I think that covers day 2 also15:18
EmilienMowalsh: ok thanks15:19
jpichflorianf: Hi! Can we mark https://blueprints.launchpad.net/tripleo/+spec/tripleo-ui-i18n-support-for-js as implemented now that the navbar is merged? And track notifications in a separate bug?15:19
openstackgerrityolanda.robla proposed openstack/tripleo-quickstart master: Do not check for nested params if virt is not enabled  https://review.openstack.org/42921215:19
florianfjpich: done!15:20
florianfjpich: high five!15:20
jpichflorianf: \o/ high five back :D15:20
EmilienMmwhahaha, owalsh: so the plan is to merge my patches that disable nova/wsgi in tripleo, and try again a patch in nova/master with check experimental, to see how it works now.15:20
EmilienMmwhahaha, owalsh: and hope nova/ironic works again15:21
*** saneax is now known as saneax-_-|AFK15:23
*** cwolferh has quit IRC15:24
owalshEmilienM: yes LGTM15:24
*** toure has joined #tripleo15:25
openstackgerrityolanda.robla proposed openstack/tripleo-quickstart master: Do not check for nested params if virt is not enabled  https://review.openstack.org/42921215:27
*** lblanchard has joined #tripleo15:29
openstackgerritMarius Cornea proposed openstack/tripleo-heat-templates master: Remove openstack-ceilometer-api pre upgrade check  https://review.openstack.org/42974915:30
*** toure has quit IRC15:31
openstackgerrityolanda.robla proposed openstack/tripleo-quickstart master: Do not check for nested params if virt is not enabled  https://review.openstack.org/42921215:33
*** toure has joined #tripleo15:33
*** limao has quit IRC15:33
*** cwolferh has joined #tripleo15:35
*** ramishra has quit IRC15:36
openstackgerritmathieu bultel proposed openstack/tripleo-heat-templates master: Add nova service support for composable upgrades  https://review.openstack.org/40524115:36
*** ramishra has joined #tripleo15:38
openstackgerrityolanda.robla proposed openstack/tripleo-quickstart master: Do not check for nested params if virt is not enabled  https://review.openstack.org/42921215:40
*** beekneemech is now known as bnemec15:40
bnemecEmilienM: panda sshnaidm: So did we figure out why the images weren't uploaded on Friday when the promotion jobs passed?15:40
openstackgerritmathieu bultel proposed openstack/tripleo-heat-templates master: Remove precheck on services which run on httpd for upgrade  https://review.openstack.org/42975215:41
sshnaidmbnemec, as I understood the dlrn hash didn't change15:41
pandabnemec: consistent repo is blocked by ironic FTBFS since friday15:41
*** milan has quit IRC15:42
pandabnemec: I didnt' look at the flow for the upload/promotion in detail though15:42
bnemecHmm.  It's really nasty that the promotion jobs can all be green and not actually get a promotion.15:43
*** jaosorior has joined #tripleo15:43
*** nijaba has quit IRC15:43
EmilienMbnemec: iiuc, you copied a mitaka image to the directory (it should have been a master image)15:45
openstackgerritJiri Tomasek proposed openstack/tripleo-ui master: Nodes Table tagging  https://review.openstack.org/42454715:46
EmilienMbnemec: have you seen http://paste.openstack.org/show/zgb3f6FBCPgv51OYK0Mu/ when you deploy the undercloud? i think it's related to https://github.com/openstack/instack-undercloud/commit/03d5cd9a17525ea44dd2c69d2d95b75c3b7f4c0d15:46
bnemecEmilienM: Yeah, there's a patch up to fix it already.15:47
bnemecEmilienM: https://review.openstack.org/42969915:47
EmilienMbnemec: ack thx15:48
openstackgerritMerged openstack/tripleo-quickstart master: Move libvirt_uri to common role  https://review.openstack.org/42431915:49
mariosmatbu: gfidente shall we just land this one wdyt? the problem is for anyone trying to upgrade newton -> ocata and not using the new hiera hook format in their templates15:51
mariosgfidente: matbu mcornea but we need it for n->o anyway otherwise we can't upgrade (we are all currently doing this in our env with the upgrade init)15:51
*** ealcaniz has quit IRC15:51
*** nijaba has joined #tripleo15:51
*** nijaba has quit IRC15:51
*** nijaba has joined #tripleo15:51
*** smalleni has joined #tripleo15:51
gfidentemarios which one?15:52
marioshttps://review.openstack.org/#/c/424715/15:52
mariosgfidente: mcornea matbu  ^^^ heh :)15:52
mariosthanks gfidente i had half of my irssi screen covered with ongoing undercloud upgrade and couldn't see all that i wrote :)15:52
openstackgerritMerged openstack/tripleo-specs master: Add i18n bug tag  https://review.openstack.org/42961315:53
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: add quickstart release files for ocata  https://review.openstack.org/42836215:54
mariosgfidente: see discussion in that review for link to the mailing list and related review for those templates that would be affected if they tried to upgrade15:54
*** tremble has quit IRC15:55
matbumarios: yep lets land that too15:55
*** nrado has quit IRC15:55
matbumarios: there is too much thing every where15:55
matbu:)15:55
matbuit's hard to have an env with all reviews15:55
mariosmatbu: yeah almost there though i mean the basic workflow at least15:56
mariosmatbu: then the real bugs start :)15:56
EmilienMjpich: i18n tag has been created in Launchpad15:56
jpichEmilienM: Awesome, thank you!15:57
matbumarios: yep exactly :)15:57
matbumarios: like what im facing with heat15:57
EmilienMjpich: no, thank *you*15:57
gfidentemarios can we pass that via UpgradeInitCommand ?15:57
gfidente*could15:58
mariosgfidente: yeah i mean thats what we are all doing in our envs15:58
jpich:)15:58
*** twiggy_ has quit IRC15:58
*** twiggy has joined #tripleo15:59
mariosgfidente: but the we'd have to document it and it would be upto the operator to add it into the UpgradeInitCommand. this just adds it by default as part of the UpgradeInitCommand, actually just after it like https://review.openstack.org/#/c/424715/2/puppet/compute-role.yaml15:59
gfidenteah so we could but we don't want to, right15:59
sshnaidmbnemec, derekh do you know why we upload stable branch images in promotion jobs like periodic-tripleo-ci-centos-7-ovb-ha-newton ?15:59
jaosoriorjistr: ping16:00
gfidente+2 from me, I could merge it but it's red?16:00
bnemecsshnaidm: Nope.  I wasn't even aware that we did, hence the screwup on Friday.16:00
bnemecAlthough since we're doing it anyway, it would be nice to make use of those images.16:00
openstackgerritBrad P. Crochet proposed openstack/tripleo-common master: Implement check hypervisor stats as a custom action  https://review.openstack.org/42975716:01
derekhsshnaidm: iirc we don't use them in CI but we uploaded them so users could download/use them if they wanted, ya using them would be nice16:01
sshnaidmbnemec, there is patch to promote ocata: https://review.openstack.org/#/c/427385/ of weshay16:01
sshnaidmbnemec, derekh ok, but I think we need at least to separate them in different folders16:02
derekhsshnaidm: ok16:02
sshnaidmand not to have everything mixed in builds/16:02
mariosgfidente: matbu: ok lets try get green ci and land this then https://review.openstack.org/#/c/424715/216:03
EmilienMmcornea, marios: quick FYI on https://bugs.launchpad.net/tripleo/+bug/166216716:05
openstackLaunchpad bug 1662167 in tripleo "nova-api fails to start during undercloud upgrade because httpd is already listening on port 8774" [Critical,Triaged]16:05
*** pblaho has quit IRC16:05
EmilienMmcornea, marios: I'm removing nova/wsgi from apache, and we'll switch back to eventlet16:05
EmilienMsee https://launchpad.net/bugs/166136016:05
openstackLaunchpad bug 1661360 in OpenStack Compute (nova) "tempest test fails with "Instance not found" error" [Undecided,In progress] - Assigned to Dan Smith (danms)16:05
mcorneaEmilienM: ack16:05
mariosthanks EmilienM mcornea brought this up on a call a little while ago (and i saw your post to mailing list after)16:06
EmilienMmcornea, marios: in other words, you shouldn't have the bug anymore once my patches are landed: https://review.openstack.org/#/c/428778/16:06
*** lmasiero has quit IRC16:06
*** dtantsur|mtg is now known as dtantsur16:07
EmilienMsshnaidm, panda: promotion seems to pass16:08
dtantsurslagle, hi! mind removing your +A and adding it back on https://review.openstack.org/#/c/428066/ please? otherwise it won't merge (dependencies ftw)16:08
sshnaidmEmilienM, panda I see new images in the directory \o/16:08
sshnaidmhttp://66.187.229.139/builds/current-tripleo/16:09
EmilienMsshnaidm: what are the next steps now?16:09
EmilienMsshnaidm: can we do recheck already on other patches?16:09
sshnaidmEmilienM, yep16:09
sshnaidmEmilienM, I'll delete renamed one and everything should work16:09
*** ealcaniz has joined #tripleo16:09
EmilienMok16:10
EmilienMsshnaidm: thx16:10
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo master: Stop deploying Nova API in WSGI with Apache  https://review.openstack.org/42878316:10
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Stop deploying Nova API in WSGI with Apache  https://review.openstack.org/42877816:10
*** ealcaniz has quit IRC16:10
*** ooolpbot has joined #tripleo16:11
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION16:11
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/166165616:11
*** ooolpbot has quit IRC16:11
openstackLaunchpad bug 1661656 in tripleo "CI: promoted overcloud-full image is missing from the mirror" [Critical,Triaged]16:11
EmilienMsshnaidm: closing https://bugs.launchpad.net/tripleo/+bug/1661656 again16:12
*** rlandy is now known as rlandy|brb16:12
sshnaidmEmilienM, ok16:12
pandayay16:12
openstackgerritMerged openstack/tripleo-heat-templates master: Remove old host param  https://review.openstack.org/42842016:17
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy novajoin in the TLS-everywhere job  https://review.openstack.org/42748116:17
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job (novajoin)  https://review.openstack.org/42748216:17
EmilienMowalsh: what is the plan in the case nova/trunk is still failing for us in TripleO, even when deploying nova in eventlet?16:17
*** florianf has quit IRC16:18
*** nmathew has joined #tripleo16:19
*** cylopez has quit IRC16:20
*** florianf has joined #tripleo16:21
* EmilienM afk until tonight16:22
*** jcoufal_ has joined #tripleo16:25
*** jcoufal has quit IRC16:27
*** jprovazn is now known as jprovazn_bbl16:30
*** rlandy|brb is now known as rlandy16:32
*** owalsh is now known as owalsh-afk16:33
*** limao has joined #tripleo16:34
*** panda is now known as panda|bbl16:35
*** nmathew has quit IRC16:35
openstackgerritMerged openstack/tripleo-docs master: Update bare metal cleaning documentation with latest changes  https://review.openstack.org/42806616:37
*** pcaruana has quit IRC16:38
*** remix_tj has quit IRC16:39
*** limao has quit IRC16:39
*** mburned is now known as mburned_out16:39
*** mburned_out is now known as mburned16:39
karimbhello folks. is it possible to used a bonded interface for provisioning network? to put vlan ( and native vlan) on top this bond ?16:40
karimbuse case for machines with only 2 nics16:40
*** remix_tj has joined #tripleo16:41
*** cwolferh has quit IRC16:45
*** abehl has quit IRC16:48
*** dmarlin is now known as dmarlin_16:52
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Save images according to their branch  https://review.openstack.org/42978216:52
*** yprokule has quit IRC16:55
*** jmelvin has quit IRC16:57
*** iranzo has quit IRC16:57
*** [1]cdearborn has joined #tripleo17:02
openstackgerritAna Krivokapic proposed openstack/tripleo-heat-templates master: Add plan-environment.yaml  https://review.openstack.org/41738517:03
bnemeckarimb: I think you could only do that if the machine is capable of PXE booting off a bonded interface.17:03
*** nyechiel has quit IRC17:03
bnemeckarimb: But you might be able to do the initial PXE deploy on one of the nics, then bond them afterward.17:04
*** cwolferh has joined #tripleo17:04
bnemeckarimb: Note that I've never done that, but I think it's possible in theory.17:04
karimbbnemec is that a "possibly", but no one ever tried ?17:04
karimb:) bnemec++ thanks17:05
bnemeckarimb: I'm not personally aware of anyone doing it, but dsneddon_afk is probably the one to talk to about it.17:05
bnemecHe deals with a lot more networking configurations than I do.17:05
*** milan has joined #tripleo17:08
*** charliejllewelly has joined #tripleo17:08
*** [2]cdearborn has joined #tripleo17:09
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo master: nova/libvirt: switch vnc server binding  https://review.openstack.org/42733917:10
openstackgerritAna Krivokapic proposed openstack/tripleo-heat-templates master: Add plan-environment.yaml  https://review.openstack.org/41738517:11
*** paramite has quit IRC17:12
*** tesseract has quit IRC17:18
*** jlinkes has quit IRC17:18
*** cdearborn has quit IRC17:18
*** [2]cdearborn is now known as cdearborn17:18
mcorneamatbu: I got an upgrade failure - http://paste.openstack.org/show/597866/17:19
mcorneamatbu: does the pacemaker module support this parameter - unsupported parameter for module: check_mode?17:19
*** [1]cdearborn has quit IRC17:21
*** suuuper has quit IRC17:21
*** jmelvin has joined #tripleo17:22
*** rasca has quit IRC17:22
matbumcornea: ha it's because the ansible module is not up to date17:22
matbumcornea: which version do you have ?17:23
mcorneamatbu: ansible-pacemaker-0.0.1-0.20170119114253.9ebe948.el7.centos.noarch17:23
matbumcornea: k17:23
matbumcornea: i'll check why the latest change is not built17:25
matbubut you can either remove the checkÃ_ode in the ansible command or just git clone the repo17:25
mcorneamatbu: ok17:26
openstackgerritMichael Bayer proposed openstack/tripleo-heat-templates master: add configurable timeouts for DB sync  https://review.openstack.org/41629817:26
*** florianf has quit IRC17:26
*** florianf has joined #tripleo17:26
mcorneamatbu: now I'm hitting  specified reference "BlockStorageUpgradeBatch_Step2" (in CephStorageUpgradeConfig_Step0) is incorrect. Did you find a workaround for it?17:26
matbumcornea: when you do heat resource-list or whatever ?17:27
matbumcornea: i think there is 2 bugs here, one in heat and one in tht17:27
mcorneamatbu: yeah17:27
*** dsneddon_afk is now known as dsneddon17:28
matbumcornea: but the most critical is in heat, because it let the user create dependencies on resource which doesn't exist17:28
matbuand in tht we create dependencies with unexiting resources17:29
mcorneamatbu: can you check this https://review.openstack.org/#/c/428349/ not sure if it's related17:29
matbumcornea: i'll try to solve the issue at in the heat side17:29
dsneddon karimb, Some switches support some sort of LACP fallback, which will use a single NIC out of the bond if an LACP bond cannot be established. This is how I PXE boot my systems, is by using "lacp force-up" on one of the interfaces, so it will boot off that interface, then after booting the interface will join the bond.17:30
matbumcornea: hm i don't think, the issue came with the recent patch merged on friday17:30
mcorneamatbu: ok, please let me know when you have something ready so I can test it17:30
matbumcornea: disable puppet for none controller17:30
matbumcornea: k17:31
matbumcornea: it's the end of the day for you ?17:31
mcorneamatbu: yep but I'll be around for 30 minutes or so17:31
dsneddonkarimb, In my case, "lacp force-up" is a Juniper JunOS command17:33
*** jpich has quit IRC17:34
*** karimb has quit IRC17:36
*** abehl has joined #tripleo17:37
*** agurenko has quit IRC17:37
*** jmelvin_ has joined #tripleo17:40
*** aufi has quit IRC17:47
*** jmelvin has quit IRC17:48
*** derekh has quit IRC17:49
jaosoriordprince: ping17:50
dprincejaosorior: hi17:51
jaosoriordprince: hey dude, regarding the containerization work17:51
jaosoriordprince: are the puppet steps ran at any point of a containerized service dpeloyment?17:51
jaosoriordprince: I ask, since we do use puppet to request the service certificates for the TLS-everywhere work, and would like to figure out how that would work out with containerized services.17:52
dprincejaosorior: the undercloud installer supports (currently) baremetal and containers17:54
dprincejaosorior: I had it fully working before Christmas when I started all the containers stuff17:54
dprincejaosorior: we have the ability to use puppet for some initialization tasks17:55
dprincejaosorior: and generate configs, but the stepwise stuff plays a bit different with containers17:55
jaosoriordprince: so, currently we don't run any of the steps for a containerized servce, right?17:56
dprincejaosorior: I'm not sure what you are doing with TLS within puppet is the best fit yet. Havent gotten there17:56
jaosoriordprince: currently what I'm doing is that on the first step, I use certmonger (exposed via a provider in puppet) to request those certs17:56
*** yamahata has quit IRC17:56
dprincejaosorior: we generally just generate configs. But for example Mysql, Keystone, and Mongo we also run manifests to create Mysql users, keystone endpoints, etc.17:56
dprincejaosorior: I think we can support what you are doing, we've got options17:57
*** mcornea has quit IRC17:57
dprincejaosorior: see the bottom here https://review.openstack.org/#/c/416605/31/docker/services/keystone.yaml17:57
dprincejaosorior: if what you require is per-server though it might be easier to just wire in a docker-cmd at a specific step. That would probably be the easiest to do to distribute the requests17:59
*** sudipto_ has quit IRC18:00
*** sudipto has quit IRC18:00
jaosoriordprince: well, most services use httpd to do the TLS serving or termination18:00
jaosoriordprince: so the amount of certs are not that many and are only dependant on the network18:00
jaosoriordprince: would it be possible to run a generic set of steps per-node, before the containers are up?18:01
*** lucasagomes is now known as lucas-afk18:01
dprincejaosorior: oh yes. I think I just need to see the commands you require18:01
jaosoriordprince: in that case, we could just expose those files (the cert and the key) to the container18:01
dprincejaosorior: if you want to try this I think perhaps pair programming is the fastest way to attack it18:01
jaosoriordprince: that would be excellent18:01
dprincejaosorior: exactly. I think we'd probably end up running your cert requests in an "init container"18:02
dprincejaosorior: those would get saved off to a docker volume (or directory on the host). Then we'd be able to bind mount them in wherever we want them18:02
jaosoriordprince: so, for all the services running over httpd18:03
jaosoriordprince: via t-h-t I pass a hash containing the specs for the certs18:03
jaosoriordprince: so, for instance, in keystone, I do this https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/keystone.pp#L12318:03
dprincejaosorior: with containers each container runs its own httpd BTW. They could share certs... but do note these end up being separate processes18:03
jaosoriordprince: and the cert-request is basically this https://github.com/openstack/puppet-tripleo/blob/master/manifests/certmonger/httpd.pp#L4818:04
dprincejaosorior: I need to see what certmonger_certificate does before I form an oppinion I think.18:05
jaosoriordprince: all it does it a call to getcert18:06
jaosoriorwhich is certmonger's CLI interface18:06
dprincejaosorior: it might be easier to just short circut puppet out of this and use docker-cmd along w/ an init container for these things18:07
dprincejaosorior: especially if they are idempotent on their own18:07
dprincejaosorior: we can use puppet for some things though. Its just playing less of an orchestrative role... and more of a utility role18:07
jaosoriordprince: are you using the kolla dockerfiles?18:07
dprincejaosorior: we are18:08
dprincejaosorior: but the commands you require likely wouldn't go there either. We could keep them in tripleo-heat-templates18:08
dprincejaosorior: see the exec for the bootstrap in that keystone file as an example18:08
dprincejaosorior: or any of the db sync steps in other docker services18:09
*** charliejllewelly has quit IRC18:09
*** dtantsur is now known as dtantsur|afk18:09
jaosoriordprince: the thing is, that the getcert call needs to have permissions to talk to the CA18:10
jaosoriordprince: and I guess we wouldn't have proper authentication credentials if we do that within the container, would we?18:10
jaosoriordprince: or can we do initialization tasks for the container within the node?18:10
dprincejaosorior: we can give it --privileged --net=host --user=root or whatever you require I think18:10
dprincejaosorior: arguably a misuse of containers to do that, but some other openstack services require just this18:11
openstackgerritJustin Kilpatrick proposed openstack/tripleo-quickstart-extras master: Add introspection with retries option  https://review.openstack.org/40367718:12
jaosoriordprince: gotta think about it, cause there are two approaches here18:12
jaosoriorwe could generate private keys and certs per containerized service18:12
jaosorioror, we could just generate a set private certs and keys, and get the containers to just read those files18:13
*** salmankhan has quit IRC18:14
*** dmarlin has joined #tripleo18:14
jaosoriorI'm kind of leaning more to the second option, since the certificate renovation would rely on the containers' hosts certmonger, and the contianers wouldn't need any privileges from the CA18:14
dprincejaosorior: almost all of our containers run --net host due to network requirements of this architecture18:15
jaosoriordprince: are the services in individual docker containers? or do we have the concept of pods like in kubernetes?18:15
dprincejaosorior: right now individual18:15
jaosoriordprince: some services can't run over httpd, so I'm using mod_proxy to terminate TLS for those.18:15
jaosoriordprince: with Pods, we could easily run mod proxy in another container, and terminate TLS there. However, I'm not sure if it's possible with the current container approach18:16
dprincejaosorior: unrelated, but I *love* that Nova now calls httpd experimental. When we'd been using it for like a year I think18:16
dprincejaosorior: our heat teampltes can act like a "pod" of sorts. I don't see any reason why we can't jiggle in mod_proxy if we need it18:19
jaosoriordprince: that would be great18:21
jaosoriordprince: yeah... I'm not too happy about the nova thing18:22
*** gbarros has joined #tripleo18:22
jaosoriordprince: so, do we run any configuration bits besides the network parts on the node itself? we could run the certmonger bits on the node; and the containers would then just need to read the private keys and certs.18:24
dprincejaosorior: the more we do in containers the better. But we can use a common init container to generate things once (per node). And then re-use those certs elsewhere18:25
jaosoriordprince: I'm not sure if the contianers should have credentials to the CA, that's the thing. Which is why I was proposing for the containers to just read the certs and keys18:27
dprincejaosorior: letting one container have access to the CA creds is fine I think18:27
dprincejaosorior: but they all don't need this access18:28
jaosoriordprince: I'm not sure if I'm getting the whole picture, would you have some time to do a call so we could talk more about this?18:28
dprincejaosorior: sure, would tomorrow work though?18:28
jaosoriordprince: sure, at what time are you free?18:29
dprincejaosorior: I'll check your call and send you an invite. My morning perhaps18:29
dprincejaosorior: cal18:29
jaosoriordprince: where are you based at?18:29
dprincejaosorior: EST18:30
jaosoriordprince: currently I'm in Mexico, so I'm one hour behind yu18:30
jaosorior12:30 here18:30
*** nyechiel has joined #tripleo18:36
*** pcaruana has joined #tripleo18:36
dprincejaosorior: invite sent18:36
jaosoriordprince: works for me18:37
*** zoli|wfh is now known as zoli|gone18:37
dprincejaosorior: we can go a bit longer if needed as well18:37
*** tosky has quit IRC18:38
jaosoriordprince: split stack is not being assumed in this scheme, right? We still spawn nova/ironic instances, right?18:39
dprincejaosorior: I use deployed-server for the undercloud installer. But the containers arch should work regardless I think18:40
*** amoralej is now known as amoralej|off18:41
*** salmankhan has joined #tripleo18:42
jaosoriordprince: so, the common init container is being used already, right?18:43
dprincejaosorior: we typically use the services container themselves for their own initialization18:44
dprincejaosorior: and with the new arch I eliminated the old heat-agents container18:44
dprincejaosorior: that said if what you are getting at is the need for something in the base image we can do that18:44
dprincejaosorior: https://github.com/dprince/undercloud_containers/blob/master/build_kolla.sh18:45
*** yamahata has joined #tripleo18:45
dprincejaosorior: unpolished. But the story develops there18:45
jaosoriordprince: where and at what point in the deployment is that run?18:46
*** abehl has quit IRC18:47
dprincejaosorior: that is run before any deployment. To create kolla docker containers18:47
jaosoriordprince: uhm... does that create the base image from where the containers are created?18:48
jaosoriordprince: so that is not part of the undercloud/overcloud deploy, right? that is a step before that18:49
dprincejaosorior: this isn't required. It is only if you build your own containers18:49
jaosoriorooh, ok18:49
dprincejaosorior: we push containers to dockerhub18:50
jaosoriorok, ok, that makes sense18:50
dprincejaosorior: but I thought you wanted to see how we built them so I linked you to it18:50
*** pkovar has quit IRC18:51
jaosoriordprince: is the common init container left running? or is that ran once to generate some bits and then killed?18:52
*** chlong has quit IRC18:52
dprincejaosorior: killed18:53
jaosoriordamn, ok18:53
dprincejaosorior: generally, I mean. we have options man18:53
dprincejaosorior: Try it man18:53
dprincejaosorior: I think 1-2 hours of playing with it would answer more questions at this point18:53
jaosoriordprince: what od you mean, the containerized undercloud?18:54
dprincejaosorior: yes18:54
jaosoriordprince: ok, will do. got any blog post or something I could follow?18:54
dprincejaosorior: I documented the docker-puppet thing here https://dprince.github.io/docker-puppet.html18:55
dprincejaosorior: I may post a containerized undercloud one soon. But first I gotta actually get that working. All the nova breakages threw me a curve ball here18:55
jaosoriordprince: was kolla also deploying nova over httpd18:56
jaosorior?18:56
dprincejaosorior: then we get CI on it...18:56
dprincejaosorior: no. Kolla wasn't18:56
dprincejaosorior: their container supports it though18:56
dprincejaosorior: kolla-ansible doesn't use that "feature"18:56
*** nyechiel has quit IRC18:58
jaosoriordprince: mind if a colleague joins tomorrow's call?18:58
*** dsneddon is now known as dsneddon_afk18:59
*** dsneddon has joined #tripleo19:00
rookwith custom roles, if I create a new controller role, ie r930Controller, do I need to set the ControlCount:0 ?19:00
dprincejaosorior: not at all, anybody interested can come I think19:01
*** bkopilov has quit IRC19:01
*** bkopilov_ has quit IRC19:01
*** gfidente is now known as gfidente|afk19:01
fultonjrook: you'd set ControlCount:0 and R930ControlCount:N for howmever many r930 controlers you'd want19:02
rookThat didn't work fultonj19:02
rookI still had a rogue controller launch19:02
*** salmankhan has quit IRC19:02
rookI am going to try to set   CountDefault: 019:03
rookfor Controller19:03
fultonjrook: here's an example that worked for me https://github.com/RHsyseng/hci/blob/master/other-scenarios/mixed-nodes/layout-mixed.yaml#L34-L3519:03
rookparameter_defaults:19:04
rook  ControlCount: 019:04
rook  R930ControllerCount: 119:04
rook  R930ComputeCount: 119:04
rookis what i tried19:04
rookstill had a rogue controller deploy19:04
fultonjrook: ControllerCount != ControlCount19:04
rookI had one R930Controller deploy which is what I expect19:04
rookah ha19:04
rookok19:04
*** salmankhan has joined #tripleo19:05
*** chlong has joined #tripleo19:08
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci master: Make graphite metrics more granular  https://review.openstack.org/42983219:09
*** pcaruana has quit IRC19:11
*** rhallisey has quit IRC19:12
*** salmankhan has quit IRC19:12
*** prateek has joined #tripleo19:12
*** nyechiel has joined #tripleo19:13
*** lmasiero has joined #tripleo19:16
*** skarthik has joined #tripleo19:21
*** kjw3 has joined #tripleo19:23
*** numans has quit IRC19:24
*** karts has quit IRC19:25
*** numans has joined #tripleo19:25
*** gbarros has quit IRC19:25
*** nyechiel has quit IRC19:28
*** owalsh-afk has quit IRC19:31
*** gbarros has joined #tripleo19:32
*** owalsh-afk has joined #tripleo19:33
*** jmiu has quit IRC19:37
*** kberger has joined #tripleo19:37
*** kberger has quit IRC19:37
*** kberger has joined #tripleo19:38
*** dsariel has quit IRC19:39
*** nrado has joined #tripleo19:39
*** jmiu has joined #tripleo19:39
jidaris there anything in tripleO to do ELK style logging?19:42
jidaror any projects out there that come out of tripleo to do centralized logging19:42
bkeroTehre is the collect-logs publisher that is part of tripleo-quickstart-extras19:48
bkeroThat's an ansible role that can be used for aggregating and uploading logs. The upstream CI worker uploads them to swift and publishes a link to the logs19:48
jidarbkero: thanks for this! Upon reviewing it, this seems good for stuff like monitoring the deployment itself and the CI status, but I wonder if it's the right tool for ongoing monitoring of a deployed overcloud?19:53
*** suuuper has joined #tripleo19:58
slaglejidar: https://specs.openstack.org/openstack/tripleo-specs/specs/newton/tripleo-opstools-centralized-logging.html19:59
slaglethe blueprint says implemented19:59
jidarOoooo19:59
slaglei see composable services for fluentd, sensu, and there are some patches  pending for collectd20:00
slaglei don't really see any docs about how to use it all though20:00
slagleif you have specific questions, larsks could possibly help :)20:01
*** salmankhan has joined #tripleo20:01
* larsks is all ears.20:01
*** hjensas has quit IRC20:04
jidaryea... I'm just trying to form some thoughts on how to implement this and maybe give a talk on it...20:04
*** lmasiero has quit IRC20:06
openstackgerritBen Kero proposed openstack/tripleo-quickstart master: Add optional undercloud-setup role to quickstart-extras playbook  https://review.openstack.org/42494720:06
*** fzdarsky has quit IRC20:15
*** milan has quit IRC20:15
openstackgerritBen Kero proposed openstack/tripleo-quickstart master: Add optional undercloud-setup role to quickstart-extras playbook  https://review.openstack.org/42494720:16
*** dsariel has joined #tripleo20:17
*** nrado has quit IRC20:19
*** florianf has quit IRC20:28
*** twiggy is now known as crenguta20:31
*** owalsh-afk is now known as owalsh20:31
*** crenguta is now known as twi99y20:34
*** prateek has quit IRC20:36
*** salmankhan has quit IRC20:40
rookfultonj: I am seeing this :20:41
rook2017-02-06 20:41:23Z [overcloud.R930Controller.0.NetworkConfig]: CREATE_FAILED  resources.NetworkConfig: Property error: resources.OsNetConfigImpl.properties.config: The Parameter (StorageMgmtNetworkVlanID) was not provided.20:41
rookHowever, i am passing the StorageMgmtNetworkVlandID20:42
fultonjrook: is StorageMgmtNetworkVlandID in hiera on the overcloud node where that is failing?20:43
rookfultonj: is there a way to look on the uc?20:43
rookthe overcloud is fubar atm20:43
fultonjjust want to point out one some all thing: StorageMgmtNetworkVlandID20:44
fultonj                                                                      ^20:44
fultonjmissed it20:44
*** twi99y is now known as crenguta20:44
fultonjbasically is it "Vlan d ID"20:44
fultonjor just VlanID20:44
rookvlan'd"20:44
rookah20:44
rookhm20:44
rooklooking20:44
fultonjpossible idea20:44
rookwell20:45
rooki miss typed that20:45
fultonjrook: the heat env vars are shipped over to the overcloud nodes and then made available in hiera20:45
fultonjif puppet is failing because it doesn't see them, then hiera on the overcloud would be a place to check if they made it to see where the info go lost on the way20:45
*** dmarlin_ has quit IRC20:46
rookAHH20:46
rooki see the problem20:46
fultonjrook: i take it you cannot ssh or console into your overcloud nodes20:46
fultonjrook: ?20:46
*** salmankhan has joined #tripleo20:47
*** akrivoka has quit IRC20:48
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Deploy novajoin in the TLS-everywhere job  https://review.openstack.org/42748120:48
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE - testing TLS everywhere job (novajoin)  https://review.openstack.org/42748220:48
rookit was missing from my parameters: fultonj20:48
fultonjrook: ok, cool. i guess you're back on your way20:49
rookfultonj: i hope20:49
rook:D20:49
rookthanks20:49
*** chlong has quit IRC20:49
fultonjnp20:49
*** jobewan has joined #tripleo20:52
*** crenguta has quit IRC20:53
*** twi99y has joined #tripleo20:55
jidardon't be afraid to post up your network-env yaml (with removed IPs) if you want a second pair of eyeballs on it20:55
jidaralso creating a password for your overcloud image to help you debug this helps20:56
*** numans has quit IRC20:56
jidarget everything right and then re-upload without a password20:56
*** skarthik has quit IRC20:57
*** lmasiero has joined #tripleo20:59
*** dmarlin_ has joined #tripleo21:00
*** rajinir has joined #tripleo21:00
*** numans has joined #tripleo21:03
jidarso larsks this seems like some really great work, re: centralized logging. Has anybody documented it though? At this point I just see a bunch of code but no idea on how to glue it all together. For instance, we should expect a baremetal node for the ES server, but I don't see how that would come out of a overcloud deploy (among other things)21:04
jidarI get it depends on compostible nodes21:04
larsksjidar: this is *only* the client side agents.  Server side (e.g., your ES server) are left as an exercise to the reader.  For a variety for reasons: you may want those services to have a lifetime different from your overcloud, or to support multiple overclouds,  etc.21:05
*** rhallisey has joined #tripleo21:05
larsksjidar: we have some documentation, but I'm not sure if it's made it anywhere useful yet.  I will try to track that down.  I have to take off right now.21:05
jidarthanks larsks21:06
openstackgerritMerged openstack/diskimage-builder master: Fix typo in README.rst  https://review.openstack.org/42953021:06
*** skarthik has joined #tripleo21:09
*** ooolpbot has joined #tripleo21:11
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION21:11
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/166232121:11
*** ooolpbot has quit IRC21:11
openstackLaunchpad bug 1662321 in tripleo "CI jobs failing due to XStatic dependencies" [Critical,Triaged]21:11
*** hjensas has joined #tripleo21:14
*** hjensas has quit IRC21:14
*** hjensas has joined #tripleo21:14
*** rhallisey has quit IRC21:14
rookfultonj ok21:14
rookso I think i am still hitting a snag wrt to the composable roles here21:14
rookos-net-config is throwing up21:14
rooknetaddr.core.AddrFormatError: invalid IPNetwork /2421:15
dsneddonrook: That sounds like one of the addresses doesn't exist, probably because the network ports aren't being instantiated for the role.21:15
dsneddonrook: What does your network-environment.yaml look like (or wherever you define your role network ports)?21:16
rookhttps://gist.github.com/jtaleric/b1d3b90e4a35b0c814e9116cd6f83a65 dsneddon21:16
rook^ Role port info21:16
dsneddonrook: Hmm, from pool. OK, did you customize that file?21:17
rookfor storage, no21:18
rooki suppose I could just pass storage21:18
rookand storage_mgmt21:18
dsneddonrook: If you use the from_pool templates, you also have to do something like in environments/ips-from-pool-all.yaml21:18
dsneddonrook: But customized for your roles.21:18
dsneddonrook: It'd actually be kid of cool if you tested that, but you could just pass storage.yaml, etc. if you want to use the AllocationPools.21:19
*** jprovazn_bbl has quit IRC21:19
fultonjrook: an example is here https://github.com/RHsyseng/hci/blob/master/other-scenarios/mixed-nodes/layout-mixed.yaml#L96-L10421:19
dsneddonfultonj: Exactly! Thanks.21:20
*** jayg is now known as jayg|g0n321:20
rookah ha, nice21:20
rookI am just going to go w/o pool for now21:20
*** gbarros has quit IRC21:23
fultonjok, cool21:24
dprinceEmilienM: very important. Do you have a link to the new foundation proposed Tripleo mascot21:32
Slowerif it's not good I might have to quit my job21:32
dprinceyeah, totally man21:33
dprincelike motivation21:33
*** salmankhan has quit IRC21:39
*** salmankhan has joined #tripleo21:40
openstackgerritDimitri Savineau proposed openstack/tripleo-quickstart-extras master: Tempest: ensure heat_stack_owner is present  https://review.openstack.org/42089621:40
bnemecdprince: The draft version was here: http://lists.openstack.org/pipermail/openstack-dev/attachments/20161025/8fa4802d/attachment.jpeg21:41
dprincebnemec: thanks :)21:42
SlowerWHAAT!!21:43
Slowerwhere's the three O's in that ow.21:43
Slowerowl21:43
*** lmasiero has quit IRC21:43
bnemecI'm not sure whether any changes were made before the final.  The response was kind of "not as good" overall to the draft.21:43
Slowerdraft of the own or the whole idea?21:43
dprincebnemec: sounds like we're gonna be burning some t-shirts in Atlanta21:44
bnemecI hear there may be some "Falcons Super Bowl Champions" shirts that they can't give away. ;-)21:44
Slowerhahaha21:45
*** lmasiero has joined #tripleo21:45
* bnemec wonders if any of the projects that previously had logos actually like the new ones21:45
*** gfidente|afk has quit IRC21:50
*** dprince has quit IRC21:55
*** ansmith has quit IRC21:56
slaglei'm kind of jealous we didnt get Suprise Bear21:57
jaosoriorslagle: surprise bear?21:58
slagleum...21:58
slaglesee openstack-dev, that's all i can say21:58
*** lmasiero has quit IRC22:00
openstackgerritMerged openstack/tripleo-heat-templates master: Remove precheck on services which run on httpd for upgrade  https://review.openstack.org/42975222:01
openstackgerritDimitri Savineau proposed openstack/tripleo-quickstart-extras master: Tempest: install dependencies when it's required  https://review.openstack.org/42082422:03
jidarhahah22:05
*** gbarros has joined #tripleo22:08
*** dbecker has quit IRC22:08
smallenifultonj: seeing this when deploying with a custom R730Controller and R730Compute role "The specified reference "Controller" (in AllNodesValidationConfig.Properties.PingTestIps.list_join[1][0]) is incorrect.',"22:09
smalleniany idea where i should be looking?22:09
*** ooolpbot has joined #tripleo22:11
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION22:11
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/166232122:11
*** ooolpbot has quit IRC22:11
openstackLaunchpad bug 1662321 in tripleo "CI jobs failing due to XStatic dependencies" [Critical,Triaged]22:11
*** fultonj has quit IRC22:14
*** rbrady is now known as rbrady-afk22:15
openstackgerritMerged openstack/puppet-tripleo master: Stop deploying Nova API in WSGI with Apache  https://review.openstack.org/42878322:16
*** gbarros has quit IRC22:25
*** rhallisey has joined #tripleo22:34
*** rhallisey has quit IRC22:34
*** rhallisey has joined #tripleo22:35
openstackgerritBen Nemec proposed openstack/python-tripleoclient master: Make overcloud image upload honor --image-path  https://review.openstack.org/40726122:35
openstackgerritBen Nemec proposed openstack/python-tripleoclient master: Mock time.sleep in unit tests  https://review.openstack.org/42990922:35
*** karimb has joined #tripleo22:40
karimbdsneddon thank you ;)22:41
karimbso pretty much pxebooting over bond is switch dependent22:42
dsneddonkarimb: Yeah, although another approach is to configure two identical interfaces with the same trunked VLANs, then use one to PXE boot and bond them with OVS bonds and "balance-slb", but that mode only efficiently load balances VLAN mode tenant networks.22:43
dsneddonkarimb: In that case, you would not configure any kind of bonding on the switch.22:43
*** eggmaster has quit IRC22:44
dsneddonkarimb: OVS will only send/receive traffic for a given VLAN over one link at any time22:44
*** mburned is now known as mburned_out22:44
dsneddonkarimb: And then spread the VLANs over the bond slaves, configurable with or without automatic rebalancing at configurable intervals.22:45
karimbdsneddon so this is the answer to my other question which was whether  i could put both vlan and native vlan on this same bond22:45
karimbfor 2-nics nodes22:45
dsneddonkarimb: In the future, I want to have switch hardware configuration automation, so Ironic can automatically put a port into boot mode or bond mode as needed.22:45
karimbdsneddon that would be nice to have yeah :)22:46
dsneddonkarimb: For all but a few switches, you need to have the boot VLAN on native in both the boot and bond case.22:46
dsneddonkarimb: But a couple of switches (some Cisco, Cumulus, for instance), you can configure a native VLAN that then gets trunked when LACP forms the bond.22:46
karimbdsneddon but does that mean that once you get vlan on this bond, the native vlan traffic stops to work ?22:47
dsneddonkarimb: OVS is smart enough that it can put the native VLAN on one link or the other, and move it back and forth in the case of link failure or rebalancing.22:48
openstackgerritMerged openstack/tripleo-heat-templates master: Remove openstack-ceilometer-api pre upgrade check  https://review.openstack.org/42974922:49
openstackgerritMerged openstack/puppet-tripleo master: nova/libvirt: switch vnc server binding  https://review.openstack.org/42733922:49
dsneddonkarimb: Here are the docs on that. Note that to disable rebalancing, you set the rebalance interval to zero. Some switches detected the rebalancing as link flapping, and would drop packets. http://docs.openstack.org/developer/tripleo-docs/advanced_deployment/network_isolation.html22:50
dsneddonkarimb: Also, in production we found that OVS can cause packet loss in LACP mode, but a future version will fix that.22:50
karimbdsneddon yeah i do have such setups deployed22:50
karimbbut generally i will dedicated a single nic for provisioning22:51
karimband then the other two as bonded and vlan on otp22:51
karimbtop22:51
dsneddonkarimb: It's good to have options, especially since Redis still runs over the provisioning interface, so there is some reason to make it redundant.22:51
karimbi mean, i ve never used only 2 nics, always had at least 322:51
dsneddonkarimb: Understood. It's still easiest to deploy with a dedicated provisioning interface, partly because the TripleO templates were originally developed around that model originally.22:52
openstackgerritBen Kero proposed openstack-infra/tripleo-ci master: WIP: Add scenario testing for multinode-oooq jobs  https://review.openstack.org/42746822:53
*** pradk has quit IRC22:54
smallenianybody here for a composable roles question?22:58
*** rhallisey has quit IRC22:59
*** jmelvin_ has quit IRC23:00
*** rhallisey has joined #tripleo23:00
karimbdsneddon do you also know how can i troubleshoot the ceph part of a director deployment?23:00
*** tbonds has joined #tripleo23:01
karimbnetwork isolation did work as expected, ceph nodes do have prepared vdb and vdc GPT partitions but i dont see OSD nodes23:01
*** jaosorior has quit IRC23:04
*** jkilpatr has quit IRC23:16
*** salmankhan has quit IRC23:20
*** morazi has quit IRC23:23
*** igorbelikov_ has joined #tripleo23:27
*** rajinir_ has joined #tripleo23:27
*** hrybacki_ has joined #tripleo23:27
*** lblanchard has quit IRC23:28
*** ipsecguy_ has joined #tripleo23:28
*** Nakato_ has joined #tripleo23:28
*** matbu_ has joined #tripleo23:29
*** markmc` has joined #tripleo23:32
*** anton- has joined #tripleo23:32
*** hewbrocc` has joined #tripleo23:32
*** adarazs_ has joined #tripleo23:32
*** rajinir has quit IRC23:34
*** twi99y has quit IRC23:34
*** cdearborn has quit IRC23:34
*** sasha2 has quit IRC23:34
*** Nakato has quit IRC23:34
*** matbu has quit IRC23:34
*** ipsecguy has quit IRC23:34
*** anton has quit IRC23:34
*** larstobi has quit IRC23:34
*** markmc has quit IRC23:34
*** adarazs has quit IRC23:34
*** hewbrocca_afk has quit IRC23:34
*** igorbelikov has quit IRC23:34
*** hrybacki has quit IRC23:34
*** adarazs_ is now known as adarazs23:34
*** rajinir_ is now known as rajinir23:34
*** igorbelikov_ is now known as igorbelikov23:35
*** twi99y has joined #tripleo23:35
*** jkilpatr has joined #tripleo23:35
*** twi99y is now known as Guest8713523:35
*** hrybacki_ is now known as hrybacki23:37
*** larstobi has joined #tripleo23:39
*** sdake has quit IRC23:39
*** karimb has quit IRC23:40
*** sasha2 has joined #tripleo23:43
*** ansmith has joined #tripleo23:50

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!