Monday, 2016-09-05

*** chlong has joined #tripleo00:45
*** limao has joined #tripleo01:08
*** StevenK has quit IRC01:14
*** TSCHAK_ has joined #tripleo01:15
*** TSCHAK has quit IRC01:18
*** StevenK has joined #tripleo01:22
*** penick has quit IRC01:29
*** TSCHAK has joined #tripleo01:30
*** TSCHAK_ has quit IRC01:32
*** StevenK has quit IRC01:55
*** StevenK has joined #tripleo01:56
*** michchap has joined #tripleo02:35
*** chlong has quit IRC02:36
*** chlong has joined #tripleo02:52
*** colonwq has quit IRC03:11
*** Goneri has joined #tripleo03:57
*** Goneri has quit IRC04:24
*** oshvartz has quit IRC04:30
*** links has joined #tripleo04:49
*** bandini has joined #tripleo04:53
*** florianf has joined #tripleo04:54
*** bkopilov has joined #tripleo05:06
*** bkopilov has quit IRC05:06
*** bkopilov has joined #tripleo05:07
*** fragatina has quit IRC05:07
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone  https://review.openstack.org/32702905:10
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Add profile to write overcloud VIPs into /etc/hosts  https://review.openstack.org/35776205:10
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone  https://review.openstack.org/32702905:12
*** jaosorior has joined #tripleo05:13
*** colonwq has joined #tripleo05:21
*** aqkhan__ has joined #tripleo05:33
*** aqkhan_ has quit IRC05:34
*** aqkhan has joined #tripleo05:35
*** aqkhan__ has quit IRC05:38
*** padkrish has joined #tripleo05:38
*** padkrish has quit IRC05:40
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone  https://review.openstack.org/32702905:52
*** oshvartz has joined #tripleo05:58
*** dsariel has joined #tripleo05:58
*** leanderthal|afk is now known as leanderthal06:10
openstackgerritAdriano Petrich proposed openstack/tripleo-heat-templates: GATE TEST, please ignore  https://review.openstack.org/36544906:14
*** pcaruana has joined #tripleo06:20
*** apetrich has quit IRC06:21
*** liverpooler has joined #tripleo06:21
openstackgerritJiri Tomasek proposed openstack/tripleo-heat-templates: Update capabilities-map.yaml  https://review.openstack.org/36484206:22
*** aufi has joined #tripleo06:23
*** apetrich has joined #tripleo06:25
*** jprovazn has joined #tripleo06:26
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone  https://review.openstack.org/32702906:28
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for aodh  https://review.openstack.org/36545706:28
*** shadower has joined #tripleo06:39
*** nyechiel has joined #tripleo06:45
*** rasca has joined #tripleo06:48
*** jbadiapa has joined #tripleo06:52
*** chlong has quit IRC06:53
*** tobias-fiberdata has joined #tripleo06:58
*** hjensas has joined #tripleo07:00
*** tobias_fiberdata has quit IRC07:01
*** chlong has joined #tripleo07:06
*** abregman has joined #tripleo07:08
d0ugalAnyone got a moment to help me figure out why this is failing? https://review.openstack.org/#/c/335460/07:13
d0ugalhttp://logs.openstack.org/60/335460/9/check/gate-tripleo-ci-centos-7-nonha-multinode/ad9e287/console.html#_2016-09-04_13_11_51_18728507:15
*** ifarkas_afk is now known as ifarkas07:16
*** jpena|off is now known as jpena07:17
*** mhenkel has joined #tripleo07:18
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add service networks to hieradata  https://review.openstack.org/36547507:26
*** ebarrera has joined #tripleo07:29
*** akuznetsov has joined #tripleo07:30
*** abregman_ has joined #tripleo07:30
*** zoli_gone-proxy is now known as zoliXXL07:32
*** abregman has quit IRC07:33
*** zoliXXL is now known as zoli|wfh07:33
*** tesseract- has joined #tripleo07:33
*** oshvartz has quit IRC07:34
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review  https://review.openstack.org/36485907:34
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review  https://review.openstack.org/36485907:42
*** zoli|wfh is now known as zoli_gone-proxy07:45
*** mcornea has joined #tripleo07:48
*** zoli_gone-proxy is now known as zoliXXL07:52
*** oshvartz has joined #tripleo07:53
*** akrivoka has joined #tripleo07:55
*** fzdarsky has joined #tripleo07:56
*** paramite has joined #tripleo07:56
*** akuznetsov has quit IRC07:58
*** shardy has joined #tripleo07:59
*** jpich has joined #tripleo08:02
bandiniany soul feels like reviewing this one: https://review.openstack.org/#/c/352484/ ? it'd be a security regression if we did not land it for newton08:02
*** dtantsur|afk is now known as dtantsur08:03
dtantsurmorning, Owlers, happy Monday08:06
shardybandini: done!08:06
shardy'morning all08:06
bandinishardy: you are made of awesome, sir! thanks ;)08:06
bandinimorning dtantsur08:06
marioshey guys! guys did you hear!???08:07
mariosgfidente is away this week08:07
marioswo/08:07
marioswo/08:07
marios\o/08:07
marioskeyboard fail08:07
marioswelcome back bandini08:07
bandinilol hey hey marios08:08
mariosmorning all08:08
socialmornin08:09
socialmarios: https://review.openstack.org/#/c/34598008:09
jaosoriorshardy: hey man, saw your comment on my keystone/endpoint hieradata patch08:11
jaosoriorshardy: the deal is that if we don't have that hieradata where keystone is deployed, we're gonna run into trouble.08:11
shardyjaosorior: Yeah, I get that, I just think we need to wire it in such that it's actually composable08:11
shardyvs going back to a monolithic approach where everything is hard-coded in one place08:12
jaosoriorshardy: so somehow we need to get those <service>::keystone::auth hieradata key-values pairs in a way that it gets to where keystone is08:12
jaosoriorshardy: sure, I get that, I have nothing against that. I just didn't come up with something better08:12
jaosoriorshardy: one thing would be to have a keystone_data output for the services. merge that, and deploy that, filtering with yaql, only in a node-role that contains the keystone service08:12
shardyjaosorior: let me give it some thought - I figure we either add an interface to the service templates specially for this, go for the global_settings thing you/gfidente have previously proposed (and accepte we put the endpoint hiera everwhere), or do something else I've not yet had enough coffee to think of ;)08:13
jaosoriorshardy: haha same situation here.08:13
openstackgerritMichele Baldessari proposed openstack/puppet-tripleo: WIP DO NOT MERGE Initial Composable HA  https://review.openstack.org/36238908:13
jaosoriorshardy: by the way, based on your work. I have one commit that gets the networkname for each service on the hieradata https://review.openstack.org/#/c/365475/08:14
jaosoriorshardy: so instead of having something like this https://review.openstack.org/#/c/327029/35/manifests/certmonger/httpd_internalapi.pp I will have a generic manifest that will dynamically create certificates based on the network08:15
*** ohamada has joined #tripleo08:16
shardyjaosorior: Ack, will check it out08:18
shardyso, basically it's just a filtered version of the ServiceNetMap?08:18
jaosoriorshardy: yeah, pretty much08:18
jaosoriorshardy: nothing special there. only info I need is the network name08:18
*** openstackgerrit has quit IRC08:18
shardyk, possibly we can simplify it a little bit but looks good08:18
*** openstackgerrit has joined #tripleo08:19
jaosoriorshardy: if you know how to simplify it I would greatly appreciate it08:19
jaosoriorshardy: I'm still new to this yaql business08:19
*** dbecker has joined #tripleo08:19
mariossocial: ack, looking08:28
*** lucas-dinner is now known as lucasagomes08:28
*** chlong has quit IRC08:31
*** abregman_ has quit IRC08:33
jaosoriorshardy: is the hiera key 'public_virtual_ip' only available in the controllers?08:35
mariossocial: wow, literally a one-line include https://review.openstack.org/#/c/345980/8/elements/puppet-stack-config/puppet-stack-config.pp08:36
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review  https://review.openstack.org/36485908:36
mariossocial: so everything is now puppet-ironic side (me pokes a bit more )08:36
shardyjaosorior: atm, yes08:37
*** tremble has joined #tripleo08:38
shardyjaosorior: VipConfig is only deployed on the controller right now - we may need to change that08:38
*** r-mibu has quit IRC08:38
shardyI was thinking about refactoring all of that stuff into allNodesConfig08:38
shardybut not yet got to actually doing it08:38
jaosoriorshardy: that would actually be useful, I think08:39
sshnaidmdtantsur, do you know maybe if I can tell ironic to dump console logs from deployment?08:40
*** derekh has joined #tripleo08:40
dtantsursshnaidm, yes you can, look into these options https://review.openstack.org/#/c/362026/6/manifests/drivers/agent.pp@3208:41
dtantsurbut this is newton-only btw08:41
*** apetrich has quit IRC08:42
socialmarios: yes08:42
sshnaidmdtantsur, seems like I need this patch already twice :) Does it need recheck?08:45
*** apetrich has joined #tripleo08:45
mariossocial: heh, well almost ... i mean it is not a copy/paste of what we had to do to setup ipxe ... so is a bit hard to review but given the instack-undercloud change is literally just including the puppet-ironic ironic::pxe and there is  +1 from ironic folks and other +2 i felt ok to add mine08:45
*** abregman_ has joined #tripleo08:46
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: Save console logs from all vms to files  https://review.openstack.org/36536908:46
dtantsursshnaidm, just did :) but ofc you can set these parameters manually08:47
sshnaidmdtantsur, ofc?08:47
dtantsurof course08:48
sshnaidmdtantsur, oh, you mean in iroic.conf08:48
dtantsuryep08:48
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review  https://review.openstack.org/36485908:50
d0ugalshardy: Hey, would you have time to look at: https://review.openstack.org/342261?08:50
shardyd0ugal: woohoo, it's passing CI! :)08:52
d0ugalshardy: :)08:52
shardyd0ugal: sure, I was planning to pull that and add the jinja patch from rbrady today anyway08:52
shardyd0ugal: related to that, I wondered if you had thoughts re the tripleoclient interface08:52
shardyI'm thinking overcloud deploy --templates -r optional_role_data.yaml08:53
shardyand have that update the plan before the jinja action gets run in the deployment workflow08:53
shardyanother option would be mandating a series of CLI calls, e.g create plan, update plan, deploy overcloud referencing plan08:53
shardyI suppose the latter may be possible regardless of the former looking at the overcloud plan deploy stuff08:54
openstackgerritMerged openstack/tripleo-ui: Refactor DeploymentPlan Component  https://review.openstack.org/35881408:54
openstackgerritMerged openstack/tripleo-ui: Mistral Actions error handling  https://review.openstack.org/35976408:54
openstackgerritMerged openstack/tripleo-ui: Remove Node registration username validation  https://review.openstack.org/36423808:55
socialI wonder what is asking for password during undercloud install in master08:55
*** atom_ has joined #tripleo08:56
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Move VIP-related data to AllNodesConfig  https://review.openstack.org/36552208:57
jaosoriorshardy: what about this? ^^08:57
socialmarios: it's follow up of https://review.openstack.org/#/c/354125/08:58
socialmarios: next step for undercloud what I'd love to do is to move the provisioning from element to puppet08:59
d0ugalshardy: I think we need to move away from overloading the deploy command.08:59
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review  https://review.openstack.org/36485908:59
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Move VIP-related data to AllNodesConfig  https://review.openstack.org/36552209:00
d0ugalshardy: we should think more along the lines of creating a plan, adding to that plan and deploying it09:00
d0ugalshardy: this will create a workflow that can then better line up with the GUI09:00
shardyd0ugal: Ok - I guess for now we can just create the plan, upload via swiftclient then deploy09:01
openstackgerritMerged openstack/puppet-tripleo: updates to fluentd support  https://review.openstack.org/35350709:01
d0ugalshardy: There is: openstack overcloud plan create :)09:01
shardyperhaps later we can add a tripleoclient interface that validates the roles_data somewhat09:01
shardyd0ugal: Yeah, it was the post-create updating of roles_data I was unsure of09:01
openstackgerritMerged openstack/instack-undercloud: galera and rabbit do not need to be accessible over the network  https://review.openstack.org/35248409:02
d0ugalshardy: right, I am not sure where that is best placed tbh09:03
shardyd0ugal: I added one question to https://review.openstack.org/#/c/342261 and I'll pull it locally now before hopefully approving09:03
*** openstackgerrit has quit IRC09:04
*** openstackgerrit has joined #tripleo09:04
d0ugalshardy: thanks - replied.09:06
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: Save console logs from all vms to files  https://review.openstack.org/36536909:07
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Create entries for overcloud VIPs in /etc/hosts  https://review.openstack.org/35776509:08
d0ugalAnyone got a moment to help me figure out why this is failing CI? https://review.openstack.org/#/c/335460/09:09
jaosoriorjistr: ok, so I made a patch that moves the vip data to all nodes config https://review.openstack.org/365522 and with that, we can now use hiera for the creation of the /etc/hosts entries https://review.openstack.org/35776509:09
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: Save console logs from all vms to files  https://review.openstack.org/36536909:10
*** tremble has quit IRC09:22
atom_hi everyone - I'm trying to install the undercloud but getting package conflicts when instack-undercloud starts...09:23
*** r-mibu has joined #tripleo09:23
atom_this is what I do to initially provision the server: https://paste.fedoraproject.org/422238/06735914/09:23
atom_and here the output of the command: https://paste.fedoraproject.org/422237/47306726/09:24
atom_and I doing something wrong?09:24
atom_I followed the docs as is09:24
atom_am I*09:27
*** athomas has joined #tripleo09:27
*** tremble has joined #tripleo09:28
*** tobias_fiberdata has joined #tripleo09:29
jaosorioratom_: I'm not sure if something wrong was done. But sure seems to be a package problem. Need to verify in #rdo09:30
jistrjaosorior: thanks, i reviewed the moving of vip data to all nodes config. IIUC it would still not work like this, but i think it's the right way. I suggested two alternatives on one of the issues there, wondering if you'd prefer one or the other.09:31
atom_jaosorior: ok I'll try there, thanks!09:31
jistrjaosorior: perhaps moving to all nodes config as you've done is better, but we just need to compute the values in the config rather than pass them as inputs?09:31
*** tobias-fiberdata has quit IRC09:32
jaosoriorjistr: lol, that's what I wanted to do in the first place09:32
jistrhehe :)09:32
jaosoriorjistr: I messed up and added stuff to controllerallnodesconfig instead09:32
jaosoriorjistr: I blame it on my lack of caffeine09:32
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Create entries for overcloud VIPs in /etc/hosts  https://review.openstack.org/35776509:40
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Move VIP-related data to AllNodesConfig  https://review.openstack.org/36552209:40
jaosoriorjistr: what about that?09:43
jaosorior^^09:43
jistrjaosorior: i like that better, but still -- why don't we just compute the values from VipMap and ServiceNetMap, when we are already passing both in? I think we could get rid of the *_virtual_ip params altogether. It's hardwired either way, so i don't think we need to parametrize them...09:45
*** dsariel has quit IRC09:47
jistrjaosorior: i think we could just move the `get_attr` inside the config, and rename it to `get_param` probably09:48
* jistr lunch, bbl09:48
*** dtantsur is now known as dtantsur|bbl09:50
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review  https://review.openstack.org/36485909:51
*** snecklifter has joined #tripleo09:52
snecklifterhi, waiting for a second docs code review on a trivial but important addition https://review.openstack.org/#/c/363760/09:55
*** limao has quit IRC09:58
snecklifterjaosorior: thanks09:59
jaosoriorjistr: damn good idea.10:00
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Create entries for overcloud VIPs in /etc/hosts  https://review.openstack.org/35776510:00
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Move VIP-related data to AllNodesConfig  https://review.openstack.org/36552210:00
openstackgerritMerged openstack/tripleo-docs: Fix missing tenant network item  https://review.openstack.org/36376010:05
lucasagomesderekh, re nodelock error, did the pyghmi fix helped with that ?10:06
*** dobson` has joined #tripleo10:07
*** dobson has quit IRC10:07
*** tremble has quit IRC10:08
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review  https://review.openstack.org/36485910:08
derekhlucasagomes: checking10:10
*** tremble has joined #tripleo10:10
*** tremble has joined #tripleo10:10
*** adarazs is now known as adarazs_lunch10:10
*** chem has joined #tripleo10:13
derekhlucasagomes: I think its helped some random failures but it doesn't look like it helped the "is locked by host " error10:19
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates: WIP Add a function to upgrade from full HA to NG HA  https://review.openstack.org/35862610:20
lucasagomesderekh, right :-/ will see if I can think of something to help with it10:22
*** tremble has quit IRC10:22
lucasagomesit's strange because it's fails at node registration phase10:22
*** tremble has joined #tripleo10:23
*** tobias-fiberdata has joined #tripleo10:26
derekhlucasagomes: ok10:27
*** mhenkel has quit IRC10:29
*** tobias_fiberdata has quit IRC10:29
*** ramishra has quit IRC10:30
*** ramishra has joined #tripleo10:32
*** hjensas has quit IRC10:37
*** pblaho has joined #tripleo10:40
*** mbound has joined #tripleo10:42
*** mbound has quit IRC10:47
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Create entries for overcloud VIPs in /etc/hosts  https://review.openstack.org/35776510:48
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Move VIP-related data to AllNodesConfig  https://review.openstack.org/36552210:48
shardyHey all, can I get a +A for https://review.openstack.org/#/c/364388/ please?10:50
*** hjensas has joined #tripleo10:50
*** hjensas has quit IRC10:50
*** hjensas has joined #tripleo10:50
shadowershardy: done10:51
shardyshadower: thanks!10:52
sshnaidmderekh, our template changes didn't help much indeed, seems like it's not the big impact on this problem10:55
sshnaidmderekh, this should solve: https://review.openstack.org/#/c/361915/10:56
derekhsshnaidm: ya, it probably helped some random ironic timeouts but it didn't have an impact on the locked node problem I was hoping it would10:56
shardyd0ugal: testing the deploy patch, works great, I think we can send it10:58
d0ugalshardy: yay!10:59
shardyd0ugal: one remaining question - what happens if the deploy workflow exceeds the time of the token timeout?10:59
shardyheat will keep running and reauthenticate, but I'm wondering what happens to the workflow10:59
shardyas IIRC mistral can't reauthenticate atm10:59
*** tremble has quit IRC11:00
*** rodrigods has quit IRC11:00
*** rodrigods has joined #tripleo11:00
d0ugalshardy: The workflow only triggers the deployment, it doesn't wait for it to finish for that reason11:00
shardyd0ugal: ack, OK sounds good11:01
openstackgerritMerged openstack/python-tripleoclient: Use heatclient poll_for_events in wait_for_stack_ready  https://review.openstack.org/36014111:03
*** tremble has joined #tripleo11:04
d0ugalshardy: yay, thanks!11:04
shardyAwesome, that poll_for_events fix will enable us to consume https://review.openstack.org/#/c/364667/11:06
* shardy tries it11:06
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for aodh  https://review.openstack.org/36545711:08
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for keystone  https://review.openstack.org/32702911:08
*** dtantsur|bbl is now known as dtantsur11:09
dtantsurfolks, could you please take a look at the ironic FFE series? https://review.openstack.org/#/q/status:open+topic:bp/ironic-integration11:11
dtantsursmall patches (ignore the docn one for now), all but one are green in CI, all tested locally11:11
*** lucasagomes is now known as lucas-hungry11:23
jaosoriorshardy: hey man, it seems that in this commit https://review.openstack.org/#/c/327029/ the yaql expression is not being evaluated :/11:23
jaosoriorshardy: any idea what I did wrong there? :/11:23
jaosoriorshardy: nevermind, think I found it11:24
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add service networks to hieradata  https://review.openstack.org/36547511:24
*** zoliXXL is now known as zoli|lunch11:27
openstackgerritDmitry Tantsur proposed openstack/tripleo-docs: Documentation for installing and using Ironic in overcloud  https://review.openstack.org/35401611:28
*** abregman_ is now known as abregman|mtg11:30
openstackgerritJiri Tomasek proposed openstack/tripleo-ui: Update Nodes listing  https://review.openstack.org/36558011:33
sshnaidmdid anybody see such errors? https://bugs.launchpad.net/tripleo/+bug/162026411:34
openstackLaunchpad bug 1620264 in tripleo "CI: periodic jobs fail because of "Data too long for column.."" [Undecided,New]11:34
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Make fqdn fact names match service net map  https://review.openstack.org/36558311:35
sshnaidmheat tries to insert such string: https://launchpadlibrarian.net/282768017/str11:35
*** rbrady has quit IRC11:35
*** akrivoka has quit IRC11:38
openstackgerritJiri Tomasek proposed openstack/tripleo-ui: Migrate Deploy action to Mistral  https://review.openstack.org/35712511:38
*** akrivoka has joined #tripleo11:38
*** tobias_fiberdata has joined #tripleo11:39
openstackgerritJiri Tomasek proposed openstack/tripleo-ui: Roles Listing using Mistral Action  https://review.openstack.org/36056911:39
*** tobias-fiberdata has quit IRC11:41
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Make apache-based services use network-dependent servername  https://review.openstack.org/36558511:41
openstackgerritJiri Tomasek proposed openstack/tripleo-ui: Migrate Deploy action to Mistral  https://review.openstack.org/35712511:45
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Map ctlplane network to management fqdn  https://review.openstack.org/36558811:46
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Make apache-based services use network-dependent servername  https://review.openstack.org/36558511:47
*** adarazs_lunch is now known as adarazs11:54
*** pkovar has joined #tripleo11:56
sshnaidmderekh, is it still relevant? https://bugs.launchpad.net/tripleo/+bug/159985811:58
openstackLaunchpad bug 1599858 in tripleo "CI jobs failing with "Message: No valid host was found. There are not enough hosts available., Code: 500"" [High,Confirmed] - Assigned to Derek Higgins (derekh)11:58
*** abregman|mtg has quit IRC12:00
*** dsariel has joined #tripleo12:01
derekhsshnaidm: probably, in order to work around it at the time we added an extra node to the env so failed node could retry12:01
derekhsshnaidm: thats done here http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/toci_gate_test.sh#n23512:01
derekhsshnaidm: $NODECOUNT+112:01
derekhsshnaidm: I then tried another change that I though would fix the problem https://review.openstack.org/#/c/338886/12:02
derekhsshnaidm: but that broke ssl jobs, so it got reverted12:02
derekhsshnaidm: I havn't revisited it sense then12:02
derekhsshnaidm: the easies way to find out would be to remove that +1 (probably better to wait until after the release) and see if the problem comes back again12:03
sshnaidmderekh, ok12:03
sshnaidmderekh, or maybe to search for failed deployments in ironic logs?12:04
derekhsshnaidm: yup, that would work too12:04
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review  https://review.openstack.org/36485912:04
dtantsursshnaidm, I saw it this morning12:04
dtantsuroops, I thought about a different error, sorry12:08
*** tremble has quit IRC12:08
* dtantsur can't wake up still12:08
*** fzdarsky has quit IRC12:09
*** tremble has joined #tripleo12:12
sshnaidmdtantsur, monday..12:12
dtantsurtotal Monday.. and I'm extremely decaffeinated12:12
*** abregman|mtg has joined #tripleo12:12
*** fzdarsky has joined #tripleo12:12
dtantsurI meant to say that we still struggle from "node locked" problem12:13
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Enable TLS in the internal network for aodh  https://review.openstack.org/36545712:19
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add service networks to hieradata  https://review.openstack.org/36547512:21
dtantsurgood news: stack update now works for me. bad news: I wonder why setting a few puppet variables takes 25 minutes...12:21
jaosoriordtantsur: did updating the undercloud work for you?12:22
*** jpena is now known as jpena|lunch12:23
*** lucas-hungry is now known as lucasagomes12:25
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: DO NOT MERGE - TESTING BARBICAN  https://review.openstack.org/35326212:26
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add Barbican to the overcloud - WIP  https://review.openstack.org/35245812:26
*** zoli|lunch is now known as zoli|wfh12:29
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Add barbican profile  https://review.openstack.org/35244712:32
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: Save console logs from all vms to files  https://review.openstack.org/36536912:34
*** jprovazn has quit IRC12:38
openstackgerritAna Krivokapic proposed openstack/tripleo-ui: Add app.conf to .gitignore  https://review.openstack.org/36561812:38
akrivokaanyone know how to access the invision links posted here? https://blueprints.launchpad.net/tripleo-ui/+spec/nodes-tagging12:39
dtantsurjaosorior, I ended up not trying it12:47
*** jaosorior has quit IRC12:47
*** jaosorior has joined #tripleo12:48
openstackgerritMerged openstack/tripleo-heat-templates: Add default for DeployIdentifier in nested templates  https://review.openstack.org/36438812:50
openstackgerritMerged openstack/python-tripleoclient: Update the 'overcloud deploy --templates' command to use Mistral  https://review.openstack.org/34226112:51
openstackgerritFlorian Fuchs proposed openstack/tripleo-ui: Update validations after Mistral executions listing is updated  https://review.openstack.org/36562312:53
*** apetrich has quit IRC12:54
openstackgerritDougal Matthews proposed openstack/tripleo-common: Set Deployment Parameters  https://review.openstack.org/36562512:54
*** shardy has quit IRC12:54
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Use a released version of tripleo-common  https://review.openstack.org/36442512:56
jaosoriormarios, shadower hey guys, can you review this commit? https://review.openstack.org/#/c/365522/ it moves the vip hieradata to all ndoes config.12:58
mariosjaosorior: ack queued12:58
*** pkovar has quit IRC13:04
openstackgerritFlorian Fuchs proposed openstack/tripleo-ui: Update validations after Mistral executions listing is updated  https://review.openstack.org/36562313:05
openstackgerrityolanda.robla proposed openstack/tripleo-quickstart: Customize undercloud and overcloud with virt-customize  https://review.openstack.org/36111213:05
openstackgerritDmitry Tantsur proposed openstack/tripleo-docs: Documentation for installing and using Ironic in overcloud  https://review.openstack.org/35401613:06
*** abregman|mtg has quit IRC13:09
jpichakrivoka: I think you'll need to ask lblanchard to post updated links (assuming she's the author), I've seen this in a few other places and it was because the "public" sharing URL wasn't used13:13
akrivokajpich: thanks13:13
jaosoriorjistr: can you check this one out? https://review.openstack.org/#/c/352713/ not a complicated one :D13:15
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review  https://review.openstack.org/36485913:16
*** zoli|wfh is now known as zoliXXL13:19
*** abregman|mtg has joined #tripleo13:22
jaosoriorjistr: ah, not a feature... just refactor. Should I abandon it then?13:22
jistrjaosorior: no i think it's fine, just wondering what the rules are for landing such stuff after feature freeze. Probably we should wait after we branch out stable/newton and after that we can land whatever we can.13:23
jistrjaosorior: maybe you could just mark it WIP for now13:23
jaosoriorbummer13:23
*** tobias-fiberdata has joined #tripleo13:24
jaosoriorproposed it about a month ago :/13:24
jaosoriorjistr: by the way, any idea about the FFE rules? sent a mail about the certmogner bp, requesteing a FFE but had no answer about it13:25
*** oshvartz has quit IRC13:25
*** ebarrera has quit IRC13:25
*** oshvartz has joined #tripleo13:25
*** ebarrera has joined #tripleo13:25
*** nyechiel has quit IRC13:25
*** tobias_fiberdata has quit IRC13:27
jistrhmm i don't know... might be good to ping shardy when he's around13:27
jistrin general i think TripleO is kinda swarmed with FFEs, i'd be very surprised if we were able to land all the proposed FFEs to be honest13:27
jistrstill might be worth at least trying it if the changes are incremental perhaps13:28
atom_hi, should instack-undercloud detect (and possibly warn) of packages that are already installed on the system that will block the installation?13:30
atom_just spent some time with that and wondered if I should submit a patch or let the sysadmin debug that himself13:30
*** jpena|lunch is now known as jpena13:31
dtantsurfolks, do we document how to configure several NICs? not for network isolation, just to be able to create several provider networks (flat in my case)?13:33
jaosorioratom_: I'm honestly not sure. I would think that's up to the sysadmin to check for the packages, but that's just me.13:33
*** jprovazn has joined #tripleo13:33
jaosoriorjistr: do you know if we enable the fact hash from puppet?13:34
jistrjaosorior: hmm fact hash? i'm not sure what you mean13:34
jistryou mean if we can use facts from `facter` in puppet?13:34
jistri think we can13:34
jaosoriorjistr: https://docs.puppet.com/puppet/latest/reference/lang_facts_and_builtin_vars.html#the-factsfactname-hash13:34
*** akuznetsov has joined #tripleo13:35
jistrjaosorior: hmm don't know, i think you'd have to try it in a deployment :) i just tore down mine to redeploy with swift...13:35
*** akuznetsov has quit IRC13:35
*** flepied has quit IRC13:35
sshnaidmjaosorior, hi, do you maybe have any ideas why this happens in periodic? https://bugs.launchpad.net/tripleo/+bug/162026413:36
openstackLaunchpad bug 1620264 in tripleo "CI: periodic jobs fail because of "Data too long for column.." in heat-engine" [Critical,Triaged]13:36
*** abregman|mtg is now known as abregman13:36
jaosoriorsshnaidm: that would usually be that heat messed up something in the database-related code.13:37
jaosoriorsshnaidm: most likely they didn't take into account for innodb limits13:37
jaosoriorlet me check13:37
jaosoriorO_O13:38
jaosoriorwell that is damn strange13:38
jaosoriorsshnaidm: nah dude, haven't seen that one and it wasn't what I had initially theorized13:38
*** links has quit IRC13:38
sshnaidmjaosorior, is this right string to be there? looks very strange13:38
jaosoriorsshnaidm: where is that error encountered?13:39
jaosoriorsshnaidm: is that in the installation or where?13:39
sshnaidmjaosorior, in overcloud deployment13:39
sshnaidmjaosorior, and in periodic jobs only13:40
jaosoriorsshnaidm: well, I do think it's a heat issue. But no idea why that would be.13:40
*** limao has joined #tripleo13:40
sshnaidmyeah, me too13:41
jaosoriorjistr: can you check this one out? https://review.openstack.org/#/c/365475/ at least in the multinode-nonha it does what it's meant to. I will use that for choosing certificates based on the network.13:42
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review  https://review.openstack.org/36485913:43
atom_so finally my _instack_ VM is running, but when trying to SSH into it using what the output says: `ssh root@192.168.122.39`, I get prompted for a password13:44
atom_what's this default pw?13:44
atom_I don't see anything in the docs13:44
jaosorioratom_: how did you do the deployment?13:45
atom_just ran `instack-virt-setup`13:45
jaosorioratom_: if you're following this http://tripleo.org/index.html then that's pretty strange, as it should have copied your key there and you should have passwordless login (as far as I remember)13:46
atom_the environment vars that I used can be found here: http://paste.fedoraproject.org/422238/6735914/ (but I think it's the default values)13:46
jaosorioratom_: but if you're gonna test out a virtual deploy ment. I really recommend just using tripleo-quickstart.13:46
jaosorioratom_: it hopefully will replace instack-virt-setup https://github.com/openstack/tripleo-quickstart13:47
openstackgerritMerged openstack/puppet-tripleo: Ironic: manage PXE environment for conductors  https://review.openstack.org/36110913:47
jaosoriorat some point13:47
atom_hm right - going to check the output of the setup, otherwise I'll use tripleo-quickstart13:48
*** shardy has joined #tripleo13:48
jaosoriorshardy: hey dude, I think it was last week, but I sent a mail applying for an FFE regarding the internal TLS work. Have you had time to check it out?13:50
jaosoriorshardy: for reference http://lists.openstack.org/pipermail/openstack-dev/2016-September/102702.html13:52
dtantsurfolks, can I please get a second +2 on a few patches for ironic FFE? https://review.openstack.org/#/q/status:open+topic:bp/ironic-integration (ignore the docs one)13:52
dtantsursorry for being annoying, just trying to get this blueprint out of the radar13:53
jaosoriordtantsur: not annoying at all, no worries13:53
*** pkovar has joined #tripleo13:56
dtantsurmerge party \o/ thanks jaosorior, jistr13:56
shardyjaosorior: Hey, sorry I thought I replied to that13:56
*** limao has quit IRC13:56
shardyI added it to RC1 last week, although it's a little unclear from your ML message exactly which patches remain13:56
shardyif you feel like sending a followup patch with a full list of patches that would help us focus review attention13:57
jaosoriorshardy: when is RC1 due?13:57
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates: Rework the pacemaker_common_functions for M..N upgrades  https://review.openstack.org/32102713:58
jaosoriorshardy: also, when will stable/newton be created?13:58
shardyjaosorior: It's next week https://releases.openstack.org/newton/schedule.html13:58
*** limao has joined #tripleo13:58
shardywith some possibility of another RC after that, but we really only have a couple of weeks to land everything that will end up in newton13:59
mariosjaosorior: should be Sep 12-16  from https://releases.openstack.org/newton/schedule.html - i think EmilienM said there had been newton cut for tripleoclient (or was it common or both, github can soon tell you) but not for tripleo heat templates,. at least friday13:59
marios(i mean at least this was the case last friday, jaosorior )13:59
shardyjaosorior, marios: Yeah some projects considered libraries have been released already, but we may be able to release them again before declaring the release final14:00
shardyjaosorior: I expect us to branch stable/newton for everything when we cut the first RC release14:00
jaosorioralright14:00
shardyjaosorior: do you still think the TLS stuff will land, or should we be looking at deferring to ocata at this point?14:00
*** jistr is now known as jistr|afk14:01
dtantsurso essentially only the final release will be "trailing", right? other milestones match other projects?14:01
mariosshardy: ack thanks14:01
jaosoriorshardy: I honestly think we gotta defer to ocata14:01
shardydtantsur: Yes, the trailing part is just in case we need some extra time due to last minute changes in other projects14:01
dtantsurgot it, makes sense14:02
shardydtantsur: all other releases align with the main schedule, and I'm hoping we won't need much of the cycle-trailing contingency14:02
shardyjaosorior: Ok, that's cool - I'll re-target the BP14:02
dtantsurdo you think non-core folks could help with something? I guess I will have a few spare cycles after this ironic madness merges :)14:02
shardydtantsur: absolutely, any help with reviews of stuff targetted to https://launchpad.net/tripleo/+milestone/newton-rc1 would be a great help14:04
openstackgerritMerged openstack/tripleo-common: Allow running validation against different plans  https://review.openstack.org/31819414:04
shardyyou can always chase for core +A's when stuff looks ready14:04
*** zoliXXL is now known as zoli|brb14:05
dtantsurokie. I'll try to test something locally, I'm interested in learning more things here.14:05
shardydtantsur: sounds good, thanks!14:05
dtantsurshardy, do you maybe know the answer to my early question " do we document how to configure several NICs? not for network isolation, just to be able to create several provider networks (flat in my case)?"14:07
shardydtantsur: I'd chat with dsneddon, he may have docs related to that14:07
dtantsursure, will do14:08
jaosoriorshardy: by the way, mind giving this one a check? https://review.openstack.org/#/c/365522/5 moves the VIP config/deployment to allNodesConfig14:08
*** pradk has joined #tripleo14:09
shardyjaosorior: will do but I'm mid-debugging atm14:09
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Fix aodh auth url to remove suffix  https://review.openstack.org/36511714:16
*** tremble has quit IRC14:20
*** tremble has joined #tripleo14:23
*** saneax-_-|AFK is now known as saneax14:24
d0ugaldtantsur: Hey, thanks for the review on: https://review.openstack.org/#/c/335460/ - how did you figure that out? I've been trying to find the error for a while.14:24
*** athomas has quit IRC14:25
*** zoli|brb is now known as zoli|wfh14:25
dtantsurd0ugal, dived into DLRN logs :)14:27
d0ugaldtantsur: aha, cool - they are something of a mystery to me.14:27
dtantsurd0ugal, yeah, there is a tarball, where in some huge amount of nested directories you can find the actual rpmbuild.log14:28
dtantsurbut yeah, I suspected something like that already14:28
d0ugalThat helps :)14:28
d0ugalThanks again!14:28
*** jistr|afk is now known as jistr14:29
dtantsurnp :)14:30
*** athomas has joined #tripleo14:30
matbumarios: meeting ?14:32
mariosmatbu: i have a conflict please go ahead (sorry mentioned it friday... i'll send out invites in a bit for the retrospective on wedensday14:33
mariosmatbu: thanks!14:33
*** chlong has joined #tripleo14:33
matbumarios: yep no worries :)14:33
*** pkovar has quit IRC14:36
openstackgerritMerged openstack/tripleo-heat-templates: Ironic: enable iPXE by default and make it configurable  https://review.openstack.org/36214814:42
*** nyechiel has joined #tripleo14:42
*** apetrich has joined #tripleo14:45
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Check for differences between introspected HW  https://review.openstack.org/35397814:47
*** saneax is now known as saneax-_-|AFK14:50
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Migrate to using osc-lib  https://review.openstack.org/33546014:51
dtantsurshadower, could you please point me where the information how to get introspection data you used comes from?14:56
dtantsurshadower, what you do was a hack back around Liberty...14:56
shadowerdtantsur: I'm not sure I got it from somewhere documented. And it was originally written somewhere around Liberty14:57
shadowerdtantsur: is there a better way to get this now?14:57
dtantsurshadower, yeah, we have API for that now. please see my comment on the patch14:58
dtantsurand lemme know if you need any help with it14:58
*** chlong has quit IRC14:58
shadowerdtantsur: excellent, thanks!14:59
openstackgerritLucas Alvares Gomes proposed openstack/tripleo-common: Ironic Client: Bump the max_retries and retry_interval  https://review.openstack.org/36569415:00
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: [WIP] Add an optional timeout when waiting for websocket messages  https://review.openstack.org/36425215:00
lucasagomesderekh, I think this will workaround the NodeLocked problem https://review.openstack.org/#/c/365694/15:01
lucasagomesderekh, looking at the logs, apparently when the nodes are registered in ironic via the --register-nodes command they are "powered on" but they are marked as "powered off" in the ironic db15:02
lucasagomesderekh, so the sync power state periodic task will turn them off15:02
lucasagomeshttp://logs.openstack.org/65/361165/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/171987a/logs/undercloud/var/log/ironic/ironic-conductor.txt.gz#_2016-08-29_15_57_25_11115:03
lucasagomesthis takes ~24s in ovb15:03
lucasagomesso I've bumped the retry timeout on that patch15:03
derekhlucasagomes: ok, ya, the node would be powered on when they are registered, I guess we could power them off pre registration but no harm in making sure the code can handle case.15:05
lucasagomesyeah15:08
lucasagomesI think I should open a bug for the NodeLocked error, lemme do it15:08
dtantsurI think we have a bug, don't remember it's # though15:09
lucasagomesdtantsur, a-ha, ok lemme find it15:09
dtantsurfolks, how do I know what is going on during a deployment? I suspect mine hanged on step 5 of controller configuration...15:09
lucasagomesdtantsur, good point on the fixme there in the code as well... the thing is, it's a chain of commands that comes from tripleo-ci -> ooo-client -> ooo-common... So we can work it around to get the CI going and fix it for good with time15:10
dtantsurright, I was not asking for a fix right now, just pointing out15:10
lucasagomesyeah i got it :-)15:10
dtantsurreally strange, it looks like the deployment is done, it just does nothing... has anyone seen it?15:12
dtantsurhmm, nevermind, seems like it just took longer than usual15:13
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: [WIP] Add an optional timeout when waiting for websocket messages  https://review.openstack.org/36425215:14
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Testing low timeout  https://review.openstack.org/36569815:14
*** ebarrera has quit IRC15:14
openstackgerritMerged openstack/tripleo-heat-templates: Ironic: configure disk erasing during cleaning  https://review.openstack.org/36430415:15
openstackgerritLucas Alvares Gomes proposed openstack/tripleo-common: Ironic Client: Bump the max_retries and retry_interval  https://review.openstack.org/36569415:15
openstackgerritMerged openstack/tripleo-heat-templates: Ironic: update default enabled drivers  https://review.openstack.org/36534915:15
openstackgerritMerged openstack/puppet-tripleo: Ironic: update included drivers  https://review.openstack.org/36534815:15
*** pkovar has joined #tripleo15:16
jtomasekmandre, shadower: are all validations patches merged now? I am getting this when trying to run validation on latest master http://paste.openstack.org/show/566879/15:16
jtomasekmandre: see the end of the output for error15:17
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: [WIP] Add an optional timeout when waiting for websocket messages  https://review.openstack.org/36425215:19
shadowerdtantsur: do you know off hand how to create the session needed for ironic_inspector_client.ClientV1?15:21
*** akuznetsov has joined #tripleo15:22
mandrejtomasek: all the patches should have merged, let me see15:23
sshnaidmderekh, about https://review.openstack.org/#/c/361915/ , is this just the same is in https://review.openstack.org/#/c/365694/1 ?15:24
sshnaidmlucasagomes, ^^15:24
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Add an optional timeout when waiting for websocket messages  https://review.openstack.org/36425215:24
sshnaidmderekh, I don't know if this hardcode is better than configuration in config file15:24
dtantsurshadower, like any keystone session: http://docs.openstack.org/developer/keystoneauth/using-sessions.html#sessions-for-users15:25
*** oshvartz has quit IRC15:25
dtantsurshadower, use regular users credentials, not service ones15:25
shadowerthanks15:25
lucasagomessshnaidm, lemme look15:25
openstackgerritJulie Pichon proposed openstack/tripleo-ui: Expand help message on config sample file  https://review.openstack.org/36570715:26
*** akuznetsov has quit IRC15:26
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Add an optional timeout when waiting for websocket messages  https://review.openstack.org/36425215:27
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Add an optional timeout when waiting for websocket messages  https://review.openstack.org/36425215:28
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Testing low timeout  https://review.openstack.org/36569815:28
lucasagomessshnaidm, hmm so "post_deploy_get_power_state_retries" increase the timeout of soft reboot by the agent15:29
lucasagomessshnaidm, this only affects the nodes being deployed15:29
mandrejtomasek: I see you've tried to run from a different plan than the default one (overcloud)15:30
jtomasekmandre: yes15:30
lucasagomesbasically we wait IPA to power them off gracefully with a soft reboot, to make sure all the changes to the disks are sync'ed etc (since only calling "sync" does not guarantee that the buffers are finilized)15:30
mandrejtomasek: the last patch that added this feature landed just a little more than an hour ago15:30
lucasagomessshnaidm, the other patch is increasing the retries when the Ironic api returns a NodeLocked error15:31
jtomasekmandre: yeah, I've reinstalled tripleo-common, to include it15:31
sshnaidmlucasagomes, it's actually what my patch should do15:31
lucasagomessshnaidm, right I don't think it works for that case then15:32
mandrejtomasek: so the output you sent was with https://review.openstack.org/#/c/318194/ applied?15:32
lucasagomessshnaidm, that config option you are changing will only affect the nodes being soft rebooted by the agent ramdisk15:32
jtomasekmandre: yes15:32
lucasagomessshnaidm, not when the conductor is syncing the power state of the node (there's no ramdisk involved, ironic talks to the bmc directly to do it)15:32
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Add an optional timeout when waiting for websocket messages  https://review.openstack.org/36425215:32
mandrejtomasek: hmm, lemme try to reproduce in my environment15:33
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Testing low timeout  https://review.openstack.org/36569815:33
*** rasca has quit IRC15:33
jtomasekmandre: the error seems to be somewhere in the template as it complains on yaml format "rror: Unknown function "#property#stderr""15:33
sshnaidmlucasagomes, http://paste.openstack.org/show/566881/15:33
jtomasekerr workflow15:33
sshnaidmlucasagomes, I think it's exactly the case15:34
jtomasekmandre: s/template/workflow15:34
sshnaidmlucasagomes, when conductor is syncing the node state - yes, it is15:34
lucasagomessshnaidm, it's not... this timeout is only called when the node is being deployed (perhaps the help doc there is a bit confusing)15:35
lucasagomessshnaidm, lemme find the code15:35
lucasagomessshnaidm, https://github.com/openstack/ironic/blob/9fc4ede1a23cc0c754a8fa482725e2f1a2370e3c/ironic/drivers/modules/agent_base_vendor.py#L53315:36
*** pcaruana has quit IRC15:36
lucasagomesthat bug is when registering the nodes and moving it to the available provision state, so there's no deployment going on at that time15:37
sshnaidmlucasagomes, I think it's exactly when deployment was going on, as I saw it in logs, it definitely wasn't in registering stage15:38
lucasagomessshnaidm, oh right, so maybe we have this exception happening in different parts15:39
lucasagomessshnaidm, cause here: http://logs.openstack.org/65/361165/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-nonha/171987a/console.html#_2016-08-29_15_57_55_11981015:39
sshnaidmlucasagomes, I posted some of them in https://bugs.launchpad.net/tripleo/+bug/161262215:39
openstackLaunchpad bug 1612622 in tripleo "CI: Exception registering nodes: Node 59ecc491-41b0-4950-8d00-ddce532ba0f4 is locked by host localhost.localdomain, please retry after the current operation is completed" [Critical,In progress] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)15:39
lucasagomes^ that was in the registering nodes stage15:39
lucasagomessshnaidm, ""CI: Exception registering nodes:"15:39
sshnaidmlucasagomes, let me look..15:40
* lucasagomes reads the comments in the bug ticket15:40
*** leanderthal is now known as leanderthal|afk15:43
sshnaidmlucasagomes, sorry, you're right, it is in registration stage, seems like I confused it with something else15:45
*** nyechiel_ has joined #tripleo15:46
*** igorbelikov_ has joined #tripleo15:46
lucasagomessshnaidm, no problem. So, that option you are changing on that patch is to make sure the data buffer has been commit the disk after the deployment15:47
sshnaidmlucasagomes, so it powering nodes on and off in registration stage?15:47
*** CaptTofu_ has joined #tripleo15:47
*** patrickeast_ has joined #tripleo15:47
*** bnemec has joined #tripleo15:47
lucasagomessshnaidm, well, not exactly... What is happening is that, the function that register the nodes in TripleO also changes it's provision state15:47
mandrejtomasek: I get the same error, debugging now15:47
*** spredzy_ has joined #tripleo15:47
lucasagomessshnaidm, nodes will start at "enroll" state and the function in triepleo tries to move then to "available"15:48
lucasagomesthat shouldn't be a problem, what is triggering this problem at the moment is that15:48
jtomasekmandre: maybe something slightly changed in mistral, or an error occured during rebases?15:48
*** social_ has joined #tripleo15:48
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Convert SwiftDevicesAndProxyConfig to composable format  https://review.openstack.org/36474815:48
lucasagomessshnaidm, tripleo is creating the nodes and they are already "powered on", ironic at the beggining assumes they are "powered off"15:48
mandrejtomasek: possibly, I'll investigate15:48
*** CaptTofu has quit IRC15:48
*** Ng has quit IRC15:48
*** igorbelikov has quit IRC15:48
*** beekneemech has quit IRC15:48
*** jaosorior has quit IRC15:48
*** tdasilva has quit IRC15:49
*** patrickeast has quit IRC15:49
*** apetrich has quit IRC15:49
*** nyechiel has quit IRC15:49
*** akrivoka has quit IRC15:49
*** bkopilov has quit IRC15:49
*** yolanda has quit IRC15:49
*** jistr has quit IRC15:49
*** cinerama has quit IRC15:49
*** markmc has quit IRC15:49
*** jpena has quit IRC15:49
*** fungi has quit IRC15:49
*** social has quit IRC15:49
*** spredzy has quit IRC15:49
*** spredzy_ is now known as spredzy15:49
*** jpena has joined #tripleo15:49
*** jistr has joined #tripleo15:49
*** cinerama has joined #tripleo15:49
*** markmc has joined #tripleo15:49
lucasagomessshnaidm, so, that triggers a perodic task in the ironic-conductor that will power all the nodes off, and it takes ~24s per node in OVB (according to the logs I've seen)15:49
lucasagomesthat's where the conflict is15:49
*** yolanda has joined #tripleo15:49
*** igorbelikov_ is now known as igorbelikov15:49
*** bkopilov has joined #tripleo15:49
sshnaidmlucasagomes, yeah, I exactly supposed to increase this timeout, but seems like chosen the wrong parameter15:50
*** jaosorior has joined #tripleo15:50
lucasagomessshnaidm, another fix for that problem would be to make sure that we always create the nodes already powered off. But increasing the timeout might be better because we cna handle both cases15:50
lucasagomessshnaidm, yeah15:50
*** saneax-_-|AFK is now known as saneax15:50
sshnaidmlucasagomes, so is it tripleo client that timeouts here?15:50
jaosoriorshardy: hey, is there a way to defer the evaluation of a get_param until after the execution of a repeat in heat?15:51
lucasagomessshnaidm, ironicclient15:51
*** fungi has joined #tripleo15:51
lucasagomessshnaidm, tho, which is called by the tripleoclient15:51
sshnaidmlucasagomes, I mean who is the end "user" which stops these tries15:51
*** ifarkas is now known as ifarkas_afk15:51
*** patrickeast_ is now known as patrickeast15:51
lucasagomessshnaidm, oh the ironicclient have the retry mechanism15:52
lucasagomes(at the http.py module I think)15:53
* lucasagomes checks15:53
shardyjaosorior: currently no, ref https://bugs.launchpad.net/heat/+bug/154668415:53
openstackLaunchpad bug 1546684 in heat "repeat can't reference attributes/parameters" [Medium,In progress] - Assigned to Steven Hardy (shardy)15:53
openstackgerritKeith Schincke proposed openstack/tripleo-heat-templates: Add support for deploying Ceph RGW role  https://review.openstack.org/28902715:53
lucasagomessshnaidm, there: https://github.com/openstack/python-ironicclient/blob/master/ironicclient/common/http.py#L182-L20215:54
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: Convert SwiftDevicesAndProxyConfig to composable format  https://review.openstack.org/36474815:54
sshnaidmlucasagomes,  DEFAULT_MAX_RETRIES = 5 - it's not even configurable as I see..15:57
lucasagomessshnaidm, it is15:57
lucasagomessshnaidm, that's the default15:57
sshnaidmlucasagomes, yeah, when you call the class15:57
sshnaidmI meant not configurable in some config file15:58
lucasagomessshnaidm, ah, it's a parameter in the command line15:58
lucasagomesthere's no .conf for the clients15:58
lucasagomessshnaidm, or when we are using the lib directly (like tripleoclient does) you can give these values at class instantiating time15:58
lucasagomesso it's configurable15:59
*** akrivoka has joined #tripleo15:59
sshnaidmlucasagomes, and can we have it configurable in our code anywhere?15:59
*** apetrich has joined #tripleo15:59
lucasagomessshnaidm, yeah, I left a FIXME on my last patch for tripleo-common15:59
lucasagomessshnaidm, python-tripleoclient IMHO should be the place16:00
openstackgerritLucas Alvares Gomes proposed openstack/tripleo-common: Fix pep8 errors when running with python3  https://review.openstack.org/36572416:02
*** dtantsur is now known as dtantsur|afk16:02
shardyd0ugal: hey, where is the tripleo.process_templates action supposed to be called?16:02
shardyI don't see it used anywhere and I need to have it run after creating a plan to populate the overcloud.yaml16:02
d0ugalshardy: It isn't directly, the deploy action (and I think a few others subclass it)16:02
d0ugalshardy: so it is called https://github.com/openstack/tripleo-common/blob/master/tripleo_common/actions/parameters.py#L42 and https://github.com/openstack/tripleo-common/blob/master/tripleo_common/actions/deployment.py#L13616:04
shardyd0ugal: Ok, I'll check out the deploy plan patch, as currently you can't run that action during the deployment because tripleoclient tries to parse overcloud.j2.yaml which isn't yet rendered to overcloud.yaml16:04
d0ugalah16:04
shardySo I can do an overcloud plan create, then run that action, then do a deploy16:04
*** Ng has joined #tripleo16:05
*** ChanServ sets mode: +v Ng16:05
d0ugalshardy: The deploy will call that action already, but I think you could call it manually too16:05
shardyd0ugal: Yeah, but it breaks before the deploy inside tripleoclient atm16:05
d0ugalright16:05
shardyI guess I can do it manually then figure out how to fix that up later16:06
d0ugalshardy: I can have a go at fixing if you send me steps to reproduce the problem16:06
sshnaidmlucasagomes, can you then please update the bug with your patch? I'll abandon the mine16:06
d0ugalshardy: i'm just going to walk the dog now tho'16:06
shardyd0ugal: Ok, np I'll drop you a mail with steps to reproduce, thanks!16:07
d0ugalshardy: no worries, always looking for wee tasks :)16:07
mandrejtomasek: good news (?) actually I can't reproduce your issue16:09
lucasagomessshnaidm, yes, I've tagged it in the commit message16:09
* lucasagomes checks if the bug is updated16:10
mandrejtomasek: I had forgot to deploy the ssh keys on my undercloud and that lead to the same type of issue you had16:10
lucasagomessshnaidm, done, apparently it wasn't updated. Ty for the reminder16:11
*** pkovar has quit IRC16:11
mandrejtomasek: which brings me to the bad news, the validations won't run until you've run the copy_ssh_key workflow16:12
*** oneswig has joined #tripleo16:12
jtomasekmandre: ok, I've re-run the workflow and it works now. somehow I must have forgotten to do that. Thanks16:12
openstackgerritSteven Hardy proposed openstack/python-tripleoclient: WIP allow overcloud.j2.yaml  https://review.openstack.org/36573516:13
jtomasekmandre: running that workflow should not be a problem since it can/should be part of undercloud installation16:13
*** pblaho has quit IRC16:14
openstackgerritSteven Hardy proposed openstack/python-tripleoclient: WIP allow overcloud.j2.yaml  https://review.openstack.org/36573516:14
mandrejtomasek: well... not exactly. It should really happen at the end of 'overcloud deploy', but it doesn't hurt running it during undercloud install16:14
mandreand as we've just seen, it's actually required16:15
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Move AllNodesValidationDeployments into jinja template loop  https://review.openstack.org/33758716:15
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Convert overcloud.yaml to support jinja2 templating  https://review.openstack.org/31567916:15
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Move AllNodesDeployments into jinja template loop  https://review.openstack.org/33726716:15
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Move per-role NetIpListMap's into jinja template loop  https://review.openstack.org/36474916:15
openstackgerritAna Krivokapic proposed openstack/tripleo-heat-templates: Add constraint for ControllerCount  https://review.openstack.org/34227216:17
jtomasekmandre: oh, I probably misunderstood the purpose of that workflow. do you know why is it actually required?16:18
*** jaosorior has quit IRC16:20
mandrejtomasek: because otherwise the run workflow will fail to find the identity file (private ssh key) it will pass to ansible and blow up16:20
*** pkovar has joined #tripleo16:20
mandrejtomasek: it's a bug16:20
jtomasekok16:21
mandrejtomasek: it shouldn't break, but on the other hand I don't know how ansible will react if it doesn't have an identity file16:22
mandrejtomasek: so, atm we need to run the copy_ssh_key during undercloud install to create the ssh key, and after the overcloud deploy to copy it to the overcloud nodes16:24
jtomasekyep16:24
*** abregman has quit IRC16:28
*** derekh has quit IRC16:31
*** ohamada has quit IRC16:31
*** aufi has quit IRC16:35
*** jpich has quit IRC16:36
*** jaosorior has joined #tripleo16:38
*** zoli|wfh is now known as zoli|gone16:40
*** zoli|gone is now known as zoli_gone-proxy16:41
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Save console logs from all vms to files  https://review.openstack.org/36536916:43
*** sshnaidm is now known as sshnaidm|afk16:44
*** tesseract- has quit IRC16:45
*** lucasagomes is now known as lucas-dinner16:50
*** oneswig has quit IRC16:54
*** oneswig has joined #tripleo17:01
*** saneax is now known as saneax-_-|AFK17:05
sshnaidm|afkshardy, can you please take a look in your time? maybe you have an idea what's wrong there https://bugs.launchpad.net/tripleo/+bug/162026417:13
openstackLaunchpad bug 1620264 in tripleo "CI: periodic jobs fail because of "Data too long for column.." in heat-engine" [Critical,Triaged]17:13
*** jbadiapa has quit IRC17:15
*** tdasilva has joined #tripleo17:17
*** yamahata has quit IRC17:22
*** dbecker has quit IRC17:23
*** athomas has quit IRC17:25
*** jaosorior has quit IRC17:25
*** mbound has joined #tripleo17:25
openstackgerritTomas Sedovic proposed openstack/tripleo-validations: Check for differences between introspected HW  https://review.openstack.org/35397817:25
*** fzdarsky is now known as fzdarsky|afk17:28
*** nyechiel_ has quit IRC17:30
*** pkovar has quit IRC17:32
*** dsariel has quit IRC17:32
*** pkovar has joined #tripleo17:35
*** pkovar has quit IRC17:36
*** pkovar has joined #tripleo17:39
*** jpena is now known as jpena|away17:39
*** jaosorior has joined #tripleo17:42
*** jaosorior has quit IRC17:44
*** jaosorior has joined #tripleo17:44
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates: Refactor upgrade checks.  https://review.openstack.org/35775017:48
*** pkovar has quit IRC17:50
*** mhenkel has joined #tripleo17:50
*** jpena|away is now known as jpena|off17:52
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Move role deployment steps into overcloud.yaml  https://review.openstack.org/36576317:57
*** jprovazn has quit IRC17:57
bkeroshardy: hey, are you around? I'm curious if you could help me understand multinode overcloud deployments. I'm unsure how the /etc/nodepool info gets used for deployed-server.17:59
*** mhenkel has quit IRC18:00
shardybkero: Hi, I'm not familiar with /etc/nodepool tbh, what data does it contain?18:00
bkeroshardy: for multinode deployments, it contains these files. http://docs.openstack.org/infra/nodepool/scripts.html#ready-script18:00
bkeroBasically subnode IPs, a key to use with them, etc.18:01
bkeroAIUI tripleo-ci multinode jobs somehow consume these for an 'openstack overcloud deploy' with deployed-server heat templates18:01
shardybkero: basically we configure one node as the undercloud, then configure the other one so it contains the agents needed to talk to the undercloud, at which point the deployment can proceed like a normal TripleO deployment (only we've not deployed any nodes)18:01
bkeroshardy: So there is no overcloud? O_o18:02
shardybkero: we "deploy" an overcloud onto one of the nodes18:03
shardyin an all-in-one configuration18:03
shardywe just use the existing nodepool node, instead of provisioning one via ironic18:03
bkeroRight18:03
bkeroI was wondering how that nodepool node is referenced18:03
*** openstackgerrit has quit IRC18:03
shardyhttps://github.com/openstack-infra/tripleo-ci/blob/master/toci_instack_osinfra.sh#L7218:04
*** openstackgerrit has joined #tripleo18:04
shardybkero: looks like it's there - basically SSH access to bootstrap the configuration on the "overcloud" node18:04
*** mcornea has quit IRC18:05
bkeroshardy: So this looks like it installs wget, activates the delorean repo, and runs deploy.env18:05
bkeroI haven't read deploy.env yet. Is that done in place of an 'openstack overcloud deploy'?18:06
bkeroor is that as the name implies, just a set of environment variables?18:07
shardybkero: I think we just need it to get access to the node, so we can run some scripts which prepare it to handle the overcloud deploy18:07
shardye.g https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/tripleo.sh#L83218:07
shardybkero: we still do openstack overcloud deploy, but things are configured such that one of the nodes just configures itself, there's not any actual deployment18:08
bkeroshardy: yeah, that function just sets up the vxlan tunnel, read that one :)18:08
bkeroshardy: what I'm having trouble finding is how the openstack overcloud deploy gets configured with the subnode18:08
shardybkero: for the gory details you may be best catching up with slagle when he's back in, but basically it's here I think:18:13
shardyhttps://github.com/openstack-infra/tripleo-ci/blob/master/scripts/deploy.sh#L14218:13
shardywe run get-occ-config.sh which configures os-collect-config as if the server had been deployed normally18:13
shardyat which point, we can do "openstack overcloud deploy", and it'll configure it self with the TripleO services18:14
bkeroYeah, I read get-occ-config.sh too. That just spins until the overcloud is deployed.18:14
bkeroFun, huh? http://paste.openstack.org/show/566898/18:15
shardyYeah, then when the overcloud deploy command gets run, that'll be able to collect the data needed, configure occ, and things will proceed as normal18:16
shardythat part of the bootstrapping is a bit hacky because there's a chicken/egg problem18:16
*** shardy is now known as shardy_afk18:20
bkeroIt seems like it. Really confusing to understand and trace. I figured that the subnodes-deploy glue might be in the OVERCLOUD_DEPLOY_ARGS here, but no such luck AFAICT: https://github.com/openstack-infra/tripleo-ci/blob/master/toci_gate_test.sh#L17718:21
*** mhenkel has joined #tripleo18:24
bkeroSearching for the subnode IP of '10.220.2.216' on http://logs.openstack.org/69/365369/8/check/gate-tripleo-ci-centos-7-nonha-multinode/211d974/console.html just seems like it sets up the repos on the subnode, creates the vxlan bridge, then leaves it alone.18:26
*** flepied has joined #tripleo18:28
openstackgerritDougal Matthews proposed openstack/python-tripleoclient: Add an optional timeout when waiting for websocket messages  https://review.openstack.org/36425218:29
*** pcaruana has joined #tripleo18:31
*** bana_k has joined #tripleo18:33
*** dhill_ has quit IRC18:34
*** mbound has quit IRC18:44
openstackgerritMichael  Henkel proposed openstack/tripleo-heat-templates: add composable services for Contrail  https://review.openstack.org/34679918:45
*** cwolferh has quit IRC18:46
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Do not merge - test Telemetry services (scenario001)  https://review.openstack.org/36439018:50
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: Fix aodh auth url to remove suffix  https://review.openstack.org/36511718:52
*** mhenkel has quit IRC18:53
*** hjensas has quit IRC18:57
*** hjensas has joined #tripleo18:57
*** hjensas has quit IRC18:57
*** hjensas has joined #tripleo18:57
*** akrivoka has quit IRC19:02
*** florianf has quit IRC19:10
*** oneswig has quit IRC19:15
*** dhill_ has joined #tripleo19:19
*** paramite has quit IRC19:22
*** jaosorior has quit IRC19:34
*** mhenkel has joined #tripleo19:40
*** fzdarsky|afk has quit IRC19:42
*** bana_k has quit IRC19:44
*** pcaruana has quit IRC19:52
*** shardy_afk has quit IRC19:56
*** yamahata has joined #tripleo19:57
*** shardy_afk has joined #tripleo19:57
openstackgerritSteven Hardy proposed openstack/tripleo-common: Change default deploy artifacts container name  https://review.openstack.org/36578320:04
*** Hazelesque_ has joined #tripleo20:07
*** HenryG_ has joined #tripleo20:08
*** StevenK has quit IRC20:10
*** onovy has quit IRC20:10
*** HenryG has quit IRC20:10
*** rwsu has quit IRC20:10
*** jtomasek has quit IRC20:10
*** larstobi has quit IRC20:10
*** Hazelesque has quit IRC20:10
*** StevenK has joined #tripleo20:11
*** onovy has joined #tripleo20:11
*** larstobi has joined #tripleo20:11
*** rwsu has joined #tripleo20:11
*** jtomasek has joined #tripleo20:12
*** rhallisey has joined #tripleo20:17
*** dbecker has joined #tripleo20:20
*** mhenkel has quit IRC20:20
*** mbound has joined #tripleo20:26
openstackgerritBrad P. Crochet proposed openstack/tripleo-heat-templates: Composable Mistral services  https://review.openstack.org/32343620:26
openstackgerritBrad P. Crochet proposed openstack/tripleo-heat-templates: Do not merge, checking composable mistral.  https://review.openstack.org/35294720:27
*** maeca1 has joined #tripleo20:28
openstackgerritDougal Matthews proposed openstack/tripleo-common: Set Deployment Parameters  https://review.openstack.org/36562520:28
*** snecklifter has quit IRC20:30
*** mbound has quit IRC20:30
*** snecklifter has joined #tripleo20:30
*** nyechiel_ has joined #tripleo20:35
*** chem has quit IRC20:41
*** akuznetsov has joined #tripleo20:52
*** chem has joined #tripleo21:06
*** phpcodemonkey has joined #tripleo21:07
*** phpcodemonkey has left #tripleo21:09
*** akuznetsov has quit IRC21:10
*** fragatina has joined #tripleo21:20
*** fragatina has quit IRC21:24
*** jeckersb_gone is now known as jeckersb21:24
*** fragatina has joined #tripleo21:24
*** jeckersb is now known as jeckersb_gone21:25
*** fragatina has quit IRC21:25
*** fragatina has joined #tripleo21:25
*** nyechiel_ has quit IRC21:41
*** dbecker has quit IRC21:43
*** akshai has joined #tripleo22:02
*** akshai has quit IRC22:03
*** fragatina has quit IRC22:04
*** akshai has joined #tripleo22:16
*** shardy has joined #tripleo22:17
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Move AllNodesValidationDeployments into jinja template loop  https://review.openstack.org/33758722:18
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Convert overcloud.yaml to support jinja2 templating  https://review.openstack.org/31567922:18
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Move AllNodesDeployments into jinja template loop  https://review.openstack.org/33726722:18
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Move per-role NetIpListMap's into jinja template loop  https://review.openstack.org/36474922:18
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Move RedisVirtualIP hiera into VipMap  https://review.openstack.org/36579122:18
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Remove *ExtraConfig parameters from overcloud.yaml  https://review.openstack.org/36579222:18
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Move role ResourceGroups inside the jinja2 loop  https://review.openstack.org/36579322:18
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Convert allNodesConfig properties to composable jinja2  https://review.openstack.org/36579422:18
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Convert VipConfig list to jinja2  https://review.openstack.org/36579522:18
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Convert deploy steps to jinja2 loop  https://review.openstack.org/36579622:18
*** maeca1 has quit IRC22:21
openstackgerritWaldemar Znoinski proposed openstack/diskimage-builder: fix systemd resource deadlock  https://review.openstack.org/36579722:24
openstackgerritWaldemar Znoinski proposed openstack/diskimage-builder: fix systemd resource deadlock  https://review.openstack.org/36579722:26
*** shardy has quit IRC22:47
*** shardy_afk has quit IRC22:51
*** akshai has quit IRC23:18
*** limao has quit IRC23:18
*** saneax-_-|AFK is now known as saneax23:19
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci: [DO NOT MERGE] Test a fake periodic job, newton 3  https://review.openstack.org/36580223:21
*** saneax is now known as saneax-_-|AFK23:21
stevebakerSlower: are you about?23:29
*** yamahata has quit IRC23:38
*** saneax-_-|AFK is now known as saneax23:39

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!