*** nyechiel_ has quit IRC | 00:11 | |
openstackgerrit | Merged openstack/tripleo-heat-templates: Convert SwiftDevicesAndProxyConfig to composable format https://review.openstack.org/364748 | 00:13 |
---|---|---|
*** saneax is now known as saneax-_-|AFK | 00:34 | |
*** dtrainor has joined #tripleo | 00:44 | |
*** dtrainor has quit IRC | 00:52 | |
*** dtrainor has joined #tripleo | 00:53 | |
*** rajinir has quit IRC | 00:55 | |
*** mlupton has joined #tripleo | 01:01 | |
*** mlupton has quit IRC | 01:03 | |
*** mlupton has joined #tripleo | 01:04 | |
*** bana_k has quit IRC | 01:23 | |
*** egafford has quit IRC | 01:31 | |
*** mlupton has quit IRC | 01:31 | |
*** yamahata has joined #tripleo | 01:38 | |
*** mlupton has joined #tripleo | 01:49 | |
*** mlupton has quit IRC | 01:54 | |
*** thrash is now known as thrash|g0ne | 01:57 | |
*** akshai has joined #tripleo | 02:04 | |
*** yamahata has quit IRC | 02:07 | |
*** chlong has joined #tripleo | 02:26 | |
*** mlupton has joined #tripleo | 02:32 | |
*** tzumainn has quit IRC | 02:44 | |
*** kjw3 has joined #tripleo | 02:47 | |
*** cwolferh has quit IRC | 02:49 | |
*** mlupton has quit IRC | 02:57 | |
*** fragatina has quit IRC | 03:03 | |
*** myoung is now known as myoung|afk | 03:05 | |
*** skramaja has joined #tripleo | 03:34 | |
*** rhallisey has joined #tripleo | 03:54 | |
*** absubram has joined #tripleo | 04:16 | |
*** absubram_ has joined #tripleo | 04:16 | |
*** rhallisey has quit IRC | 04:18 | |
*** absubram has quit IRC | 04:20 | |
*** absubram_ is now known as absubram | 04:20 | |
*** fragatina has joined #tripleo | 04:26 | |
*** fragatina has quit IRC | 04:30 | |
*** akshai has quit IRC | 04:32 | |
*** karthiks has joined #tripleo | 04:42 | |
*** coolsvap has joined #tripleo | 04:43 | |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Move elements & lib relative to diskimage_builder package https://review.openstack.org/344017 | 04:49 |
*** pgadiya has joined #tripleo | 04:52 | |
*** karthiks has quit IRC | 05:16 | |
*** jaosorior has joined #tripleo | 05:21 | |
*** chlong_ has joined #tripleo | 05:31 | |
*** karthiks has joined #tripleo | 05:32 | |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Fix value of ctlplane fqdn fact https://review.openstack.org/367754 | 05:43 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Generate VIP info for ctlplane VIP, not management https://review.openstack.org/367362 | 05:43 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Fix value of ctlplane fqdn fact https://review.openstack.org/367754 | 05:53 |
bandini | matbu: morning ;) Am attempting a bisect again today, hopefully we can narrow the upgrade list_join thing down | 05:56 |
jaosorior | bandini: what evil plans are you devising? | 05:58 |
*** aufi has joined #tripleo | 05:59 | |
*** saneax-_-|AFK is now known as saneax | 06:00 | |
matbu | bandini: hello :) | 06:00 |
bandini | jaosorior: ahah lol morning sir. trying to get to the bottom of https://bugs.launchpad.net/tripleo/+bug/1620696 | 06:00 |
openstack | Launchpad bug 1620696 in tripleo "M/N upgrades UPDATE_FAILED .enabled_services.list_join: Incorrect arguments to "list_join" should be: "list_join" : [ " ", [ "str1", "str2"]]" [Undecided,Confirmed] | 06:00 |
matbu | bandini: yeah, i tired yesterday something but it hanged my deployment for the night | 06:01 |
openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 06:04 |
*** aufi has quit IRC | 06:05 | |
*** aufi has joined #tripleo | 06:05 | |
saneax | EmilienM, shardy: today we complete all the patches for sriov blueprint https://blueprints.launchpad.net/tripleo/+spec/tripleo-sriov (FYI) | 06:05 |
*** yamahata has joined #tripleo | 06:09 | |
*** ccamacho has joined #tripleo | 06:13 | |
ccamacho | Good morning! | 06:14 |
jaosorior | ccamacho: hey dude! how was the vacation? | 06:18 |
jaosorior | bandini: got a system that's available with the same issue? I could try to help dig it out | 06:19 |
ccamacho | jaosorior hey man, really good actually 10 days without technology near to me.. | 06:20 |
openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: ModalPanel component https://review.openstack.org/366615 | 06:20 |
bandini | jaosorior: that'd be awesome yes. I will ping you in 30 mins or so as I am re-reproducing | 06:22 |
jaosorior | bandini: sure | 06:22 |
*** pcaruana has joined #tripleo | 06:23 | |
jaosorior | bandini: ok, I'll do some coffee meanwhile :P | 06:27 |
jaosorior | jtomasek: around yet? | 06:27 |
jtomasek | jaosorior: yep | 06:27 |
jaosorior | jtomasek: you're working in the UI, right? | 06:27 |
bandini | jaosorior: excellent idea ;) | 06:27 |
jaosorior | jtomasek: I saw a mail the other day about SSL support for the UI, any idea how that work is going? | 06:28 |
*** jprovazn has joined #tripleo | 06:29 | |
jtomasek | dtrainor is working on it, I think he did some progress | 06:29 |
jaosorior | jtomasek: does he have any CRs up? | 06:30 |
openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Roles Listing using Mistral Action https://review.openstack.org/360569 | 06:31 |
jtomasek | jaosorior: CRs? | 06:32 |
jaosorior | jtomasek: patches (CR = Change Request) | 06:33 |
* jtomasek is slow in the morning | 06:33 | |
jtomasek | jaosorior: I am not sure, this is the blueprint to track it https://blueprints.launchpad.net/tripleo-ui/+spec/ssl-tls | 06:34 |
jaosorior | jtomasek: he's in the US right? Gotta poke him about it later I guess | 06:35 |
*** fzdarsky has joined #tripleo | 06:35 | |
jtomasek | jaosorior: yeah, that would be best | 06:35 |
jaosorior | shadower, marios: if you guys have time can you check this commit? https://review.openstack.org/#/c/367176/1 | 06:36 |
marios | ack in a bit jaosorior | 06:38 |
openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: ModalPanel component https://review.openstack.org/366615 | 06:38 |
openstackgerrit | Merged openstack/tripleo-heat-templates: Add Ceph cluster health validation on upgrade https://review.openstack.org/359410 | 06:39 |
openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Update Nodes listing https://review.openstack.org/365580 | 06:43 |
*** bana_k has joined #tripleo | 06:47 | |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: DO NOT MERGE - TESTING BARBICAN https://review.openstack.org/353262 | 06:51 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates: Add Barbican to the overcloud - WIP https://review.openstack.org/352458 | 06:51 |
*** zoli_gone-proxy is now known as zoliXXL | 06:58 | |
*** zoliXXL is now known as zoli|wfh | 06:58 | |
*** florianf has joined #tripleo | 07:00 | |
*** social has joined #tripleo | 07:06 | |
*** tesseract- has joined #tripleo | 07:07 | |
*** flepied has quit IRC | 07:07 | |
*** zoli|wfh is now known as zoli_gone-proxy | 07:11 | |
*** mcornea has joined #tripleo | 07:12 | |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-quickstart: DO NOT MERGE: Testing workers https://review.openstack.org/366119 | 07:15 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-quickstart: Make undercloud service workers configurable https://review.openstack.org/366116 | 07:15 |
openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review https://review.openstack.org/364859 | 07:17 |
*** zoli_gone-proxy is now known as zoliXXL | 07:17 | |
*** jpena|off is now known as jpena | 07:18 | |
jaosorior | shadower: Hey man | 07:18 |
jaosorior | shadower: it's not entirely clear to me what's up here https://review.openstack.org/#/c/367422/3 | 07:18 |
jaosorior | shadower: seems to me that it'll be quite prone to confusion if the fact that the environments from mistral client have 'ssh_keys' tells whether validations are enabled or not | 07:19 |
*** chlong has quit IRC | 07:19 | |
jaosorior | shadower: isn't there another way to tell? | 07:19 |
jaosorior | shadower: else I think it would be better to add a comment to avoid confusion | 07:19 |
*** chlong_ has quit IRC | 07:19 | |
*** jlinkes has joined #tripleo | 07:22 | |
*** ebarrera has joined #tripleo | 07:24 | |
*** jpich has joined #tripleo | 07:29 | |
*** hjensas has quit IRC | 07:32 | |
*** bvandenh has joined #tripleo | 07:32 | |
shadower | jaosorior: yeah that's a little weird. Afaik the only other way to do it is to shell out to hiera, which I'm not fond of. | 07:33 |
shadower | jaosorior: but yeah adding a comment makes sense | 07:33 |
*** limao has joined #tripleo | 07:34 | |
jaosorior | shadower: well... I don't think shelling out hiera is that bad to be honest. | 07:34 |
jaosorior | shadower: if it's more explicit. | 07:34 |
*** limao_ has joined #tripleo | 07:37 | |
shadower | jaosorior: maybe. I fear it might be less robust and this is the first time a mistral action needs that. If there's more than sure | 07:37 |
shadower | though in that case, we'd probably just load hiera into a mistral env | 07:37 |
shadower | like, all of it | 07:37 |
d0ugal | Can we write mistral and not mc? :P | 07:39 |
*** limao has quit IRC | 07:39 | |
shadower | d0ugal: fine by me -- I was trying to be consistent with the existing code in that file | 07:40 |
d0ugal | ah | 07:40 |
d0ugal | fair enough | 07:40 |
jaosorior | shadower: I'll comment on the patch | 07:40 |
d0ugal | I try hard not to nit pick... but one or two letter variable names are a weak point of mine | 07:40 |
*** bana_k has quit IRC | 07:40 | |
shadower | -) | 07:41 |
d0ugal | shadower: What creates the ssh_keys env? | 07:41 |
*** chem|off is now known as chem | 07:42 | |
*** flepied has joined #tripleo | 07:42 | |
*** nyechiel_ has joined #tripleo | 07:43 | |
shadower | d0ugal: this does? https://review.openstack.org/#/c/322893/ | 07:45 |
shadower | (that was supposed to be a period) | 07:45 |
d0ugal | lol, I was confused :) | 07:46 |
d0ugal | shadower: thanks | 07:46 |
openstackgerrit | Merged openstack/tripleo-heat-templates: Add service networks to hieradata https://review.openstack.org/365475 | 07:46 |
shadower | d0ugal: it's not merged yet, but since my patch won' t break without it, I didn't use a depends-on | 07:46 |
d0ugal | Sure | 07:46 |
d0ugal | I probably would anyway for context, but it's fine. | 07:46 |
*** b00tcat has joined #tripleo | 07:47 | |
*** ohamada has joined #tripleo | 07:47 | |
mandre | d0ugal: it's the copy_ssh_key workflow that generates ssh key pair and store it in the ssh_keys mistral environment | 07:47 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud: Make HAProxy terminate OpenStack internal/admin endpoints https://review.openstack.org/360366 | 07:48 |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud: Always deploy HAProxy https://review.openstack.org/360368 | 07:48 |
d0ugal | mandre: Thanks | 07:48 |
mandre | so if you have a ssh_keys mistral environment you can safely assume you have enabled validations in your undercloud | 07:49 |
jaosorior | bandini: no environment yet? :O | 07:51 |
*** mbound has joined #tripleo | 07:54 | |
*** hjensas has joined #tripleo | 07:57 | |
*** mbound has quit IRC | 07:59 | |
bandini | jaosorior: meh one deployment failed | 08:01 |
jaosorior | :( | 08:02 |
*** derekh has joined #tripleo | 08:05 | |
marios | alert... a gfidente has been spotted in the wild rechecking things https://review.openstack.org/#/c/336542/5 | 08:05 |
jaosorior | lol | 08:05 |
*** dsneddon has quit IRC | 08:06 | |
*** athomas has joined #tripleo | 08:06 | |
*** lmiccini_ has joined #tripleo | 08:09 | |
*** mhenkel has joined #tripleo | 08:09 | |
bandini | ahahahah | 08:09 |
*** lmiccini has quit IRC | 08:10 | |
*** snecklifter has joined #tripleo | 08:19 | |
openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 08:21 |
*** lmiccini_ is now known as lmiccini | 08:23 | |
*** nyechiel_ has quit IRC | 08:25 | |
jaosorior | marios: yeah... those failures in the patch you reviewed seem pretty random. The change has nothing to do with instack-undercloud. So it just seems to me that CI is flaky | 08:28 |
jaosorior | marios: a change that depends on that patch is all green though https://review.openstack.org/#/c/367208/1 | 08:29 |
*** yolanda has quit IRC | 08:31 | |
*** yolanda has joined #tripleo | 08:37 | |
*** mhenkel has quit IRC | 08:38 | |
openstackgerrit | Merged openstack/tripleo-validations: Validate the open-files-limit value in mysql https://review.openstack.org/353961 | 08:39 |
openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 08:39 |
*** jaosorior is now known as jaosorior_lunch | 08:40 | |
openstackgerrit | Merged openstack/tripleo-validations: Validate ntp is running and synchronised https://review.openstack.org/353967 | 08:40 |
openstackgerrit | Merged openstack/tripleo-validations: Validate pacemaker status https://review.openstack.org/353969 | 08:40 |
openstackgerrit | Merged openstack/tripleo-validations: Validate rabbitmq limits https://review.openstack.org/353970 | 08:40 |
openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Migrate to using osc-lib https://review.openstack.org/335460 | 08:42 |
*** mhenkel has joined #tripleo | 08:42 | |
openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove "type: direct" from workflows as it is the default https://review.openstack.org/341617 | 08:43 |
openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Update the Mistral action names https://review.openstack.org/366519 | 08:43 |
openstackgerrit | Merged openstack-infra/tripleo-ci: scenario002: deploy Swift https://review.openstack.org/367401 | 08:43 |
*** jschlueter has quit IRC | 08:45 | |
*** myoung|afk has quit IRC | 08:45 | |
*** mhenkel has quit IRC | 08:49 | |
openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Use a released version of tripleo-common https://review.openstack.org/364425 | 08:52 |
openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove the unused service_host arg from node registration https://review.openstack.org/326036 | 08:52 |
openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Add an optional timeout when waiting for websocket messages https://review.openstack.org/364252 | 08:52 |
openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Remove the old, deprecated Mistral action names. https://review.openstack.org/366529 | 08:52 |
bandini | marios, jistr: if you have a minute to check out the last comments on https://review.openstack.org/#/c/358626 that would be awesome ;) | 08:54 |
*** tosky has joined #tripleo | 08:55 | |
*** akrivoka has joined #tripleo | 08:57 | |
*** cwolferh has joined #tripleo | 08:58 | |
*** kbyrne has quit IRC | 09:04 | |
*** abehl has joined #tripleo | 09:04 | |
openstackgerrit | Waldemar Znoinski proposed openstack/diskimage-builder: don't configure 'lo' for dhcp https://review.openstack.org/367527 | 09:04 |
*** nijaba has quit IRC | 09:05 | |
openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 09:06 |
*** nijaba has joined #tripleo | 09:06 | |
*** nijaba has quit IRC | 09:06 | |
*** nijaba has joined #tripleo | 09:06 | |
openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 09:07 |
*** zoliXXL is now known as zoli|wfh | 09:09 | |
*** kbyrne has joined #tripleo | 09:10 | |
*** shardy has joined #tripleo | 09:11 | |
*** limao_ has quit IRC | 09:12 | |
*** limao has joined #tripleo | 09:12 | |
marios | bandini: will do | 09:12 |
marios | bandini: i missed that thanks for pointing at it | 09:13 |
d0ugal | ovb gates are down? | 09:14 |
*** mbound has joined #tripleo | 09:14 | |
bandini | marios: merci! | 09:15 |
jistr | bandini, marios: added option C), curious what you'd think about it | 09:19 |
*** bvandenh_ has joined #tripleo | 09:20 | |
openstackgerrit | Merged openstack/tripleo-heat-templates: Add global_config_settings to services' output https://review.openstack.org/336542 | 09:20 |
*** limao has quit IRC | 09:20 | |
marios | jistr: i was thinking 'why not also start at the end of the update but then got thinking about the controlled way we do restart after converge | 09:20 |
marios | jistr: i mean after i posted the reply | 09:20 |
*** bvandenh__ has joined #tripleo | 09:23 | |
shardy | https://review.openstack.org/#/c/348905 https://review.openstack.org/#/c/346799 needs reviews please | 09:23 |
shardy | contrail integration, looks ready to land | 09:23 |
jistr | marios: i'm not sure i follow your 'ahem' :D -- what are you saying? | 09:24 |
*** bvandenh has quit IRC | 09:24 | |
marios | jistr: i mean we have the common functions that can handle sthing like 'restart_service' and the functions check if bootstrap & pcmk and does the restart, and if systemd it happens on all controllers | 09:24 |
*** bvandenh_ has quit IRC | 09:25 | |
bandini | marios: yes but they don't do "systemctl start openstack-foo" on *all* controllers right? We'd have to create another step for that no? | 09:26 |
bandini | but in any case agreed, the sooner we land your function rework the better | 09:27 |
marios | bandini: for example the restart_service either calls manage_systemd_service or manage_pacemaker_service. For systemd it runs on all controllers, for pacemaker only if it is bootstrap | 09:29 |
*** bvandenh has joined #tripleo | 09:30 | |
ccamacho | hey shardy, Im coming back from PTO, checking again the contrail submissions | 09:30 |
marios | bandini: so yes, depending also on where (if from a bootstrap only clause in another script for example it changes things) if you call restart_service openstack-foo it will "systemctl $action $service" on all controllers | 09:31 |
jistr | marios, bandini: we shouldn't need another step. _2.sh runs on all controllers too, and all controllers have there a wait check until the pcs cluster is up and running AFAIK, so we can put the systemd starts after that and they will run on all controllers | 09:31 |
*** bvandenh__ has quit IRC | 09:31 | |
bandini | marios, jsutr: ah righto, somehow I forgot that _[1,2].sh *are* run on all controllers. So we will just need to add a bit of logic to implement C) | 09:32 |
marios | jistr: right that would work fine we could do restart_service foo and it will be systemd on all or pcmkd on bootstrap | 09:32 |
marios | jistr: i mean with the rework on those common scripts | 09:32 |
jistr | marios: i think it should be systemd on all | 09:32 |
jistr | marios: they'd already be removed from pcmk at that point | 09:32 |
bandini | yep | 09:33 |
marios | jistr: yeah sorry forgot these were the services we're migrating | 09:33 |
*** r-mibu has quit IRC | 09:33 | |
*** r-mibu has joined #tripleo | 09:33 | |
bandini | marios: so we're just missing CI for your rework, yes? | 09:34 |
shardy | ccamacho: thanks, I got confirmation from mhenkel that they've been tested and they look OK to me | 09:34 |
shardy | ccamacho: if you feel like pulling them and testing too, that would be good | 09:34 |
marios | jistr: bandini well ok, then we'd be able to use function manage_systemd_service { in https://review.openstack.org/#/c/321027/23/extraconfig/tasks/pacemaker_common_functions.sh :D in the very least | 09:34 |
shardy | ccamacho: I think we can probably go ahead and land them now though, as first revision | 09:35 |
ccamacho | yeahp Im merging my last changes to redeploy all just to double check it. | 09:35 |
bandini | marios: yes if we can land your review, I will then depend-on it with mine and implement C) as suggested by jistr | 09:35 |
marios | bandini: yeah. ideally i'd want to include them in a run to sanity check again it has been a while. but interupts. speaking of which i have to go deal with something now | 09:35 |
*** nyechiel_ has joined #tripleo | 09:38 | |
marios | jistr: you forgot to capture the discussion though :) https://review.openstack.org/#/c/358626/ | 09:41 |
marios | jistr: oh, nm i thought you forgot to copy/paste (I was thinking of added the chat to explain the 'ahem' since it wasn't clear) | 09:41 |
jistr | marios: i though i summarized it :D but feel free to add anything more, or link or copy/paste the irc | 09:45 |
*** nyechiel_ has quit IRC | 09:46 | |
*** oshvartz has quit IRC | 09:48 | |
openstackgerrit | Tomas Sedovic proposed openstack/tripleo-common: Run copy_ssh_keys after overcloud finishes https://review.openstack.org/367422 | 09:49 |
*** nyechiel_ has joined #tripleo | 09:53 | |
*** akrivoka has quit IRC | 09:54 | |
*** skramaja has quit IRC | 09:55 | |
*** akrivoka has joined #tripleo | 09:56 | |
openstackgerrit | Steven Hardy proposed openstack/tripleo-common: Wire in jinja templating for custom roles https://review.openstack.org/362465 | 09:58 |
*** yamahata has quit IRC | 09:59 | |
*** nyechiel has joined #tripleo | 10:04 | |
*** nyechiel_ has quit IRC | 10:07 | |
*** jprovazn has quit IRC | 10:11 | |
openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Convert UpdateWorkflow to support composable roles https://review.openstack.org/367282 | 10:12 |
openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move AllNodesValidationDeployments into jinja template loop https://review.openstack.org/337587 | 10:12 |
openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Convert overcloud.yaml to support jinja2 templating https://review.openstack.org/315679 | 10:12 |
openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Convert allNodesConfig properties to composable jinja2 https://review.openstack.org/365794 | 10:12 |
openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move AllNodesDeployments into jinja template loop https://review.openstack.org/337267 | 10:12 |
openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move role ResourceGroups inside the jinja2 loop https://review.openstack.org/365793 | 10:12 |
openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move role deployment steps into puppet/post.yaml https://review.openstack.org/365763 | 10:12 |
openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Remove *ExtraConfig parameters from overcloud.yaml https://review.openstack.org/365792 | 10:12 |
openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Move per-role NetIpListMap's into jinja template loop https://review.openstack.org/364749 | 10:12 |
openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Convert deploy steps to jinja2 loop https://review.openstack.org/365796 | 10:12 |
openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates: Convert AllNodesExtraConfig to support composable roles https://review.openstack.org/367295 | 10:16 |
dtantsur | wow, that's a dump :) | 10:17 |
shardy | That BP has been > 60 patches, hopefully those are the final remaining ones | 10:19 |
shardy | I'm on PTO today so dropping offline for a while, reviews appreciated, will be around again this evening :) | 10:20 |
* derekh install rh3 to deal with shardy's latest dump of patches | 10:20 | |
shardy | lol :) | 10:20 |
shardy | sorry | 10:20 |
openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Add an action and workflow to update a plan https://review.openstack.org/367921 | 10:20 |
*** shardy has quit IRC | 10:21 | |
openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Add an action and workflow to update a plan https://review.openstack.org/367921 | 10:22 |
*** mhenkel has joined #tripleo | 10:22 | |
*** karthiks has quit IRC | 10:26 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements https://review.openstack.org/361501 | 10:26 |
*** sshnaidm|afk is now known as sshnaidm|pto | 10:28 | |
akrivoka | jtomasek: ping regarding https://review.openstack.org/#/c/367562/ | 10:28 |
openstackgerrit | Merged openstack/tripleo-docs: Documentation for installing and using Ironic in overcloud https://review.openstack.org/354016 | 10:29 |
akrivoka | jtomasek: you said I should rebase on top of https://review.openstack.org/#/c/365580/ but I don't see any relation between these two patches? | 10:29 |
jtomasek | akrivoka: pong | 10:29 |
akrivoka | jtomasek: am I missing something or did you mean to link to a different review? | 10:30 |
dtantsur | https://blueprints.launchpad.net/tripleo/+spec/ironic-integration marked done :) | 10:30 |
* jtomasek checks | 10:30 | |
akrivoka | jtomasek: perhaps https://review.openstack.org/#/c/360569/ ? | 10:30 |
*** thrash|g0ne is now known as thrash | 10:30 | |
openstackgerrit | Merged openstack/tripleo-common: Ironic Client: Bump the max_retries and retry_interval https://review.openstack.org/365694 | 10:30 |
*** ramishra has quit IRC | 10:32 | |
jtomasek | akrivoka: ah, yes, that is the one. Nodes listing patch is dependent on that too. I thought I changed nodes selectors in Nodes listing patch too but I see it happened before and now is merged in master | 10:32 |
jtomasek | akrivoka: so rebase it on top of Roles listing | 10:32 |
akrivoka | jtomasek: ok | 10:32 |
akrivoka | jtomasek: and another question | 10:33 |
akrivoka | jtomasek: I found a bug in my patch, line 94 here https://review.openstack.org/#/c/367562/1/src/js/components/deployment_plan/NodesAssignment.js | 10:33 |
*** ramishra has joined #tripleo | 10:33 | |
openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Update plan files when re-deploying https://review.openstack.org/366541 | 10:33 |
akrivoka | jtomasek: the count I'm getting from getAssignedNodes is the count before the update, but what I need is count after the update | 10:34 |
akrivoka | jtomasek: so e.g. if there are 2 nodes assigned, then I unassign them both, I will get 2 as count and not 0 | 10:35 |
jtomasek | akrivoka: hmm, that is probably because updateNode action did not finish by then. Which makes sense as it is async | 10:35 |
akrivoka | jtomasek: how can I make sure it gets called after the nodes counts are updated? | 10:36 |
*** Guest90983 is now known as rook | 10:36 | |
*** nyechiel has quit IRC | 10:38 | |
jtomasek | akrivoka: it is slightly complicated (because we're working around the actual workflow)... so | 10:38 |
*** rhallisey has joined #tripleo | 10:39 | |
jtomasek | akrivoka: you'd need to create new action which would get called in updateNodeSuccess action | 10:41 |
*** jaosorior_lunch is now known as jaosorior | 10:41 | |
*** leanderthal|afk is now known as leanderthal | 10:41 | |
jtomasek | akrivoka: to make it possible you'd need to turn that action into thunk action and at the same time as updateNode is too generic for this logic, you'd need to create completely new chain of actions to handle this workflow | 10:41 |
jtomasek | akrivoka: I think in this case a better approach is to update honza's mistral workflow with additional task which will call the udpate_parameters action | 10:42 |
jtomasek | akrivoka: what do you think? | 10:43 |
jtomasek | akrivoka: compared to what it turns out we'd have to do in GUI to achieve the same seems like much better option to me | 10:43 |
akrivoka | jtomasek: yeah probably | 10:43 |
jaosorior | anyone got time to review this t-h-t patch? https://review.openstack.org/#/c/367208/1 | 10:44 |
akrivoka | jtomasek: I'll look into that | 10:44 |
akrivoka | should have probably done that from the start... | 10:44 |
akrivoka | jtomasek: thanks for your input! | 10:44 |
jtomasek | akrivoka: thanks! I feel bad for these twists :/ | 10:44 |
akrivoka | jtomasek: heh, not your fault :) | 10:44 |
*** mhenkel has quit IRC | 10:45 | |
akrivoka | honza: mind if I update https://review.openstack.org/#/c/332132/ to add the necessary parameters setting? | 10:48 |
jaosorior | d0ugal: hey dude, did you figure out if the ovb gates are working? | 10:51 |
d0ugal | jaosorior: it seems not :) | 10:52 |
d0ugal | jaosorior: I can't tell why | 10:52 |
jaosorior | d0ugal: sure seems flaky though | 10:52 |
openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Add an action and workflow to update a plan https://review.openstack.org/367921 | 10:54 |
*** athomas has quit IRC | 10:56 | |
*** CaptTofu_ is now known as CaptTofu | 10:58 | |
derekh | d0ugal: it appears to be packets getting lost, which ia a particular problem for DNS and there is now retry, I've just now added some failback dns servers to the CI instances, so it should help things | 10:59 |
derekh | d0ugal: pretty much all of the problems I've seen today are yum install fails, which I think are failing on dns lookup | 11:00 |
d0ugal | derekh: cool, thanks | 11:00 |
jaosorior | thrash: hey dude, could you check out this commit? https://review.openstack.org/#/c/367362/ | 11:00 |
d0ugal | derekh: I'll let you know if I see anything different :) | 11:00 |
thrash | jaosorior: sure | 11:01 |
derekh | d0ugal: btw the changes I added will only take effect on jobs started from now, already running jobs have no backup | 11:01 |
thrash | jaosorior: ctlplane is on undercloud, no? | 11:02 |
derekh | bnemec: sshnaidm|pto ^ | 11:02 |
*** athomas has joined #tripleo | 11:02 | |
thrash | jaosorior: the overcloud shouldn't really "know" about the ctlplane network. | 11:03 |
jaosorior | thrash: how come? we have the keystone admin endpoint listening on that VIP | 11:03 |
tbarron | marios: when you get a chance I updated https://review.openstack.org/#/c/366760/ with a remark - no joy yet, but hopefully I'm doing something silly and easy to correct | 11:03 |
thrash | jaosorior: hmmm | 11:04 |
tbarron | marios: i see the patch getting downloaded and cached etc. in the overcloud image build but it's still not there in the overcloud image itself | 11:04 |
*** pkovar has joined #tripleo | 11:04 | |
jaosorior | thrash: yeah dude, we have a VIP dedicated to the control plane, keystone admin endpoint listening on it too :/ | 11:05 |
*** matbu is now known as matbu|lunch | 11:05 | |
thrash | jaosorior: basically, I'm agreeing with bnemec | 11:05 |
thrash | it should be in addition, not a replace. | 11:05 |
*** dsariel has joined #tripleo | 11:05 | |
jaosorior | thrash: but we don't have a VIP for the management network | 11:05 |
jaosorior | thrash: so it didn't make sense in the first place to add a name for that vip... since there is no vip for that | 11:06 |
jaosorior | thrash: that was actually my mess up | 11:06 |
thrash | jaosorior: I see... You're trying to hide your mistakes. :P | 11:06 |
jaosorior | thrash: it's not used by anything at the moment (but the stuff that sets /etc/hosts which I'm changing in that patch. So it's still a good time to replace it. | 11:06 |
jaosorior | thrash: haha s/hide/correct | 11:07 |
jaosorior | I did raise a bug on it | 11:07 |
thrash | lol | 11:07 |
*** egafford has joined #tripleo | 11:07 | |
thrash | jaosorior: I'm kidding of course. | 11:07 |
jaosorior | thrash: but yeah, there is no management vip, so that name doesn't make sense there. It should be for the ctlplane | 11:08 |
thrash | jaosorior: it seems to me that there should be a mgmt vip tho... | 11:09 |
*** pkovar has quit IRC | 11:09 | |
thrash | why wouldn't there be? | 11:09 |
jaosorior | thrash: if it exists I missed it | 11:09 |
*** akshai has joined #tripleo | 11:11 | |
thrash | jaosorior: I'm not saying that it already exists... I'm saying it either should be added, or we leave in place things that facilitate adding. | 11:11 |
thrash | jaosorior: although, thinking about it more, a management network is likely to be managed externally from ooo | 11:12 |
jaosorior | thrash: isn't the management network supposed to be just interfaces to access the overcloud nodes and that's it? I thought that was the reason for it not to have a vIP assigned to it | 11:12 |
*** aqkhan__ has quit IRC | 11:12 | |
thrash | jaosorior: I'm coming to the light | 11:13 |
*** egafford has quit IRC | 11:13 | |
jaosorior | thrash: yeah, and I had initially confused the management network to do the same thing as the control plane one | 11:14 |
jaosorior | thrash: which is what I'm trying to correct there | 11:14 |
thrash | jaosorior: gotcha | 11:14 |
jaosorior | thrash: this commit goes in the same premise https://review.openstack.org/#/c/367754/ | 11:15 |
*** ohamada has quit IRC | 11:15 | |
*** ohamada has joined #tripleo | 11:15 | |
*** pkovar has joined #tripleo | 11:15 | |
marios | was afk tbarron sorry ack will do in few | 11:16 |
*** absubram has quit IRC | 11:16 | |
*** ebarrera has quit IRC | 11:16 | |
tbarron | marios: np, and thanks | 11:17 |
*** ebarrera has joined #tripleo | 11:17 | |
openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Update plan files when re-deploying https://review.openstack.org/366541 | 11:24 |
openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Add an action and workflow to update a plan https://review.openstack.org/367921 | 11:24 |
jaosorior | thrash: so what do you think dude? | 11:25 |
*** fultonj has joined #tripleo | 11:27 | |
openstackgerrit | Merged openstack/tripleo-heat-templates: Add keystone networks for the different endpoints https://review.openstack.org/367176 | 11:28 |
*** mburned_out is now known as mburned | 11:31 | |
*** zoli|wfh is now known as zoli|lunch | 11:32 | |
snecklifter | hello, can a red hat person sanity check this when they have a moment please | 11:33 |
snecklifter | https://review.openstack.org/#/c/367628/ | 11:33 |
snecklifter | trying to understand is proxy parameter has been missed off on purpose... | 11:33 |
snecklifter | it has passed CI | 11:34 |
*** jprovazn has joined #tripleo | 11:36 | |
slagle | snecklifter: there was another patch for that already that I commented on: https://review.openstack.org/#/c/367554/ | 11:37 |
*** nyechiel has joined #tripleo | 11:38 | |
openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Add Zaqar to scenario002 https://review.openstack.org/365026 | 11:39 |
snecklifter | slagle: ah, thanks, missed that, will abandon mine as that one includes user/pass | 11:41 |
snecklifter | slagle: would be nice to see it backported though | 11:41 |
snecklifter | currently I'm just adding in the script/rhel-registration | 11:41 |
*** bvandenh_ has joined #tripleo | 11:45 | |
EmilienM | hello | 11:46 |
*** abregman has joined #tripleo | 11:47 | |
*** bvandenh has quit IRC | 11:48 | |
*** bvandenh_ has quit IRC | 11:50 | |
*** jprovazn has quit IRC | 11:50 | |
*** pkovar has quit IRC | 11:51 | |
jaosorior | EmilienM: hey dude. If you have time, can you check this out? https://review.openstack.org/#/c/327029/37/manifests/certmonger/httpd.pp It's actually a puppet-related question | 11:53 |
EmilienM | jaosorior: on my list | 11:53 |
*** lblanchard has joined #tripleo | 11:53 | |
*** lucasagomes is now known as lucas-hungry | 11:55 | |
openstackgerrit | Brad P. Crochet proposed openstack/tripleo-heat-templates: Composable Zaqar services https://review.openstack.org/331682 | 11:55 |
*** nyechiel has quit IRC | 11:55 | |
*** pkovar has joined #tripleo | 11:56 | |
*** jprovazn has joined #tripleo | 11:57 | |
openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo: Terminate Zaqar websocket endpoint in HAProxy https://review.openstack.org/360329 | 11:57 |
jaosorior | thrash: by the way... I think we're running into issues related to the zaqar websocket endpoint | 11:57 |
jaosorior | thrash: so... nonha is not being deployed with network isolation | 11:58 |
EmilienM | jaosorior: Alex is right in his review | 11:58 |
EmilienM | jaosorior: the spaceship is the solution | 11:58 |
jaosorior | thrash: which means that the ssl stuff will overwrite the non-ssl | 11:58 |
jaosorior | EmilienM: ok, no problem. I just wanted to know if that was fine... since there will be two steps (step 1 and 2) where the apache service won't appear at all in the service catalog | 11:59 |
jaosorior | so I was thinking it would break that notify | 11:59 |
thrash | jaosorior: hmmm | 11:59 |
jaosorior | thrash: wait a minute | 12:01 |
jaosorior | no | 12:01 |
jaosorior | it should be fine | 12:01 |
thrash | jaosorior: it shouldn't matter. all things should be port 9000. :) | 12:01 |
jaosorior | thrash: I got confused for a bit | 12:01 |
*** trown|outtypewww is now known as trown | 12:01 | |
jaosorior | thrash: yeah, this is how it looks like in the config | 12:01 |
jaosorior | listen zaqar_ws | 12:01 |
jaosorior | bind 192.0.2.2:9000 transparent ssl crt /etc/pki/tls/certs/undercloud-192.0.2.2.pem | 12:01 |
thrash | jaosorior: that happens to me a lot | 12:01 |
jaosorior | bind 192.0.2.3:9000 transparent | 12:01 |
jaosorior | mode http | 12:01 |
jaosorior | server 192.0.2.1 192.0.2.1:9000 check fall 5 inter 2000 rise 2 | 12:01 |
thrash | jaosorior: that looks fine to me. | 12:02 |
jaosorior | thrash: well... it's still failing in CI somehow :( | 12:02 |
jaosorior | which is strange, we saw that work yesterday | 12:03 |
openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 12:04 |
thrash | jaosorior: where are you seeing it fail? | 12:05 |
*** jayg|g0n3 is now known as jayg | 12:08 | |
openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: scenarios: set Debug to True https://review.openstack.org/366896 | 12:08 |
jaosorior | thrash: in the node registration part | 12:09 |
jaosorior | thrash: https://review.openstack.org/#/c/360350/ | 12:10 |
openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: scenario001: deploy Ceph https://review.openstack.org/366810 | 12:10 |
thrash | jaosorior: ahh yes | 12:10 |
*** jpena is now known as jpena|lunch | 12:12 | |
*** hjensas has quit IRC | 12:13 | |
jaosorior | dtrainor: you around? | 12:14 |
*** kjw3 has quit IRC | 12:15 | |
*** matbu|lunch is now known as matbu | 12:15 | |
openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 12:25 |
honza | akrivoka: no at all! do you want me to rebase it first? | 12:27 |
akrivoka | honza: no need, I've got it :) | 12:27 |
*** pradk has joined #tripleo | 12:28 | |
*** skramaja has joined #tripleo | 12:33 | |
openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Add config parameter to collect-logs ci-script https://review.openstack.org/364394 | 12:33 |
*** tobias-fiberdata has joined #tripleo | 12:36 | |
*** zoli|lunch is now known as zoli|wfh | 12:36 | |
*** tobias_fiberdata has quit IRC | 12:39 | |
jaosorior | thrash: hey dude, could you check this one out too? https://review.openstack.org/#/c/367754/2 the values from those facts come from the values we set in the templates, like here: https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/controller.yaml#L412 | 12:42 |
thrash | jaosorior: sure | 12:44 |
thrash | jaosorior: I don't +2 puppet. :) | 12:44 |
thrash | I know... not really puppet. :) | 12:45 |
jaosorior | I see | 12:45 |
jaosorior | thrash: a +1 would do :D | 12:45 |
marios | tbarron: sorry i keep getting interrupts, but will update momentarily, trying to come up with something to help... but agree you are not picking up the puppet-tripleo changes from what i can see | 12:46 |
marios | tbarron: long story short, i think we can just use virt-customize to upload the puppet manifest to the overcloud.yaml image so you can get passed this pain with trying to pull the review | 12:48 |
openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Convert UpdateWorkflow to support composable roles https://review.openstack.org/367282 | 12:48 |
openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Convert allNodesConfig properties to composable jinja2 https://review.openstack.org/365794 | 12:48 |
openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Move role ResourceGroups inside the jinja2 loop https://review.openstack.org/365793 | 12:48 |
openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Convert AllNodesExtraConfig to support composable roles https://review.openstack.org/367295 | 12:48 |
openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates: Convert deploy steps to jinja2 loop https://review.openstack.org/365796 | 12:48 |
*** lucas-hungry is now known as lucasagomes | 12:51 | |
jaosorior | thrash: thanks dude | 12:51 |
*** jaosorior has quit IRC | 12:52 | |
*** jaosorior has joined #tripleo | 12:53 | |
openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Use relative paths in ci-scripts for quickstart.sh https://review.openstack.org/360649 | 12:53 |
*** rlandy has joined #tripleo | 12:54 | |
*** Goneri has joined #tripleo | 12:54 | |
openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Document Third Party CI and developer usage https://review.openstack.org/360007 | 12:55 |
jtomasek | honza: I've updated shrinkwrap but jenkins still fail on some dependency https://review.openstack.org/#/c/366615/ | 12:56 |
* honza looking | 12:56 | |
openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: Add Role Detail https://review.openstack.org/367993 | 12:57 |
*** tobias-fiberdata has quit IRC | 12:58 | |
*** b00tcat has quit IRC | 12:58 | |
*** tzumainn has joined #tripleo | 12:58 | |
openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Make undercloud service workers configurable https://review.openstack.org/366116 | 12:58 |
*** jcoufal has joined #tripleo | 12:59 | |
honza | jtomasek: works fine for me locally, investigating further | 12:59 |
jtomasek | honza: yeah:/ | 13:00 |
honza | jtomasek: i feel like we've had this issue before --- something about phantom | 13:01 |
*** pgadiya has quit IRC | 13:01 | |
jtomasek | honza: it was colision of phantomjs and phantomjs-prebuilt afaik. there is no more phantomjs npm package, so it must be something different | 13:01 |
honza | jtomasek: sounds about right, good memory | 13:02 |
*** tobias_fiberdata has joined #tripleo | 13:02 | |
tbarron | marios: sure, the old virt tools vs dib thing :) | 13:02 |
openstackgerrit | Merged openstack/tripleo-common: Preserve the shell environment for run-validation https://review.openstack.org/365983 | 13:03 |
tbarron | marios: though if I'm not executing wrong and dib (as used by tripleo-ci) isn't getting review changes that could be an issue for others than me i'd think | 13:04 |
honza | jtomasek: seems like a lot of people are having installation issues with phantomjs-prebuilt https://github.com/Medium/phantomjs/issues | 13:05 |
jtomasek | honza: yep | 13:06 |
honza | jtomasek: let's try upgrading to the latest version first? | 13:06 |
jtomasek | honza: that was my thought, I am going to update that patch | 13:07 |
*** cdearborn has joined #tripleo | 13:08 | |
honza | jtomasek: finally found an issue that matches our output https://github.com/Medium/phantomjs/issues/519 | 13:09 |
*** jprovazn has quit IRC | 13:10 | |
openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui: ModalPanel component https://review.openstack.org/366615 | 13:12 |
jtomasek | honza: ^ update to latest phantomjs-prebuilt, hope it helps | 13:12 |
jtomasek | honza: if not then IIUC we need to clean the npm cache on the jenkins machine? | 13:13 |
*** nyechiel has joined #tripleo | 13:14 | |
*** dsariel has quit IRC | 13:14 | |
jtomasek | jrist: can you try mistral action-exectution-list and see what execution is failing? | 13:15 |
marios | tbarron: indeed, might be worth following up with a bug to report what you did so someone can cycle back. i updated the review https://review.openstack.org/#/c/366760/4 with a comment and added notes for virt-customize to https://etherpad.openstack.org/p/manila-overcloud-deploy-with-netapp-notes | 13:15 |
EmilienM | the ovb jobs are still unreliable, I see some transient issues on centos repositories | 13:15 |
EmilienM | next week is RC1 we need to merge patches | 13:15 |
jrist | jtomasek: hmm | 13:15 |
jrist | [stack@instack tripleo-common]$ mistral action-execution-list | 13:15 |
jrist | ERROR (app) Entity '<class 'mistral.db.v2.sqlalchemy.models.ActionExecution'>' has no property 'type' | 13:15 |
EmilienM | jistr: ^ I want to land the composable roles patches | 13:16 |
tbarron | marios: excellent, i'll read the comment and notes and move forwards | 13:16 |
jtomasek | jrist: something is broken hah | 13:17 |
jrist | jtomasek: yeah | 13:17 |
jrist | jtomasek: I'll just nuke | 13:18 |
jrist | :( | 13:18 |
jrist | ...again | 13:18 |
jrist | :) | 13:18 |
jistr | EmilienM: you mean to ignore the ovb jobs? hmm :/ i don't know what are the exact differences between that and multinode but e.g. the multinode jobs don't test HA at all, right? | 13:20 |
openstackgerrit | Jeff Peeler proposed openstack/tripleo-common: Set Deployment Parameters https://review.openstack.org/365625 | 13:21 |
EmilienM | jistr: no we can't do that | 13:21 |
EmilienM | jistr: the multinode deploy pacemaker but it's a all in one node, we merge controller and compute | 13:21 |
jistr | ah ok | 13:22 |
EmilienM | pabelanger is pushing for a centos repo mirror | 13:22 |
*** jprovazn has joined #tripleo | 13:22 | |
jistr | cool | 13:22 |
EmilienM | so we can kill this problem with package not found | 13:22 |
EmilienM | and i'm investigating if there are other problems | 13:22 |
jaosorior | EmilienM, jistr: Feed me reviews | 13:22 |
jaosorior | I have time to check out the composable roles remaining stuff | 13:23 |
EmilienM | jaosorior: it's not a matter of review | 13:23 |
jistr | jaosorior: you can just start here and go up the dependency chain on top right :) https://review.openstack.org/#/c/365763 | 13:23 |
EmilienM | but a matter of CI passing or not | 13:23 |
EmilienM | jaosorior: though you can review, for sure :) | 13:23 |
EmilienM | jaosorior: we're having network troubles, see on #rdo | 13:23 |
*** jpena|lunch is now known as jpena | 13:24 | |
larsks | EmilienM: it looks like verification on 353506 failed for reasons not related to the patch itself... | 13:25 |
EmilienM | larsks: this is true | 13:25 |
larsks | What do we do? | 13:25 |
EmilienM | larsks: on #rdo, we're chatting and trying to find solutions | 13:25 |
EmilienM | we're having networking issues in CI | 13:25 |
larsks | Okay. I will follow along. | 13:25 |
openstackgerrit | Justin Kilpatrick proposed openstack/tripleo-quickstart: Add documentation for add-on roles https://review.openstack.org/346733 | 13:26 |
pabelanger | derekh: re: DNS issues, did you see my comments yesterday about using tripleo-centos-7 nodes for DNS? We hace DNS caching setup already for google and should be straightforward to use them as servers for ovb nodes | 13:28 |
pabelanger | need to test, but should be able to open firewall on tripleo-centos-7 to accept udp/53 | 13:28 |
derekh | pabelanger: its worth a try but I don't think the DNS server we're using is the problem, we're just loosing packets somewhere, give me a few mintes to confirm | 13:30 |
pabelanger | derekh: yes, that is possible too. We've had this issue in the past I believe | 13:31 |
jaosorior | EmilienM: ah, crap :/ | 13:31 |
EmilienM | larsks: I know your stuff is urgent and I got reminders every day to merge it. We'll merge it when our CI will be working | 13:31 |
*** pradk has quit IRC | 13:32 | |
*** pradk has joined #tripleo | 13:32 | |
larsks | EmilienM: No worries. I just noticed the -1 this morning and wanted to check in! | 13:32 |
openstackgerrit | Merged openstack/tripleo-quickstart: inventory role no longer needed for the usbkey https://review.openstack.org/365048 | 13:33 |
EmilienM | jistr: I'm going to land https://review.openstack.org/#/c/365763/ it passes less than 48h ago and we didn't modify the code, just rebased | 13:34 |
EmilienM | larsks: I'm rebasing your patch | 13:35 |
openstackgerrit | Merged openstack/tripleo-quickstart: Add objectstorage flavor and extra blockdevices option https://review.openstack.org/359630 | 13:36 |
jistr | EmilienM: the rebase looks like it's in a different spot than the patch itself https://review.openstack.org/#/c/365763/13..14/overcloud.yaml so i'm fine with merging it | 13:37 |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Add fluentd client service https://review.openstack.org/353506 | 13:37 |
EmilienM | jistr: ack | 13:37 |
EmilienM | jistr: I'm currently exploring all we can to merge stuffs | 13:38 |
*** jschluet has joined #tripleo | 13:38 | |
jistr | EmilienM: i think this is the same case https://review.openstack.org/#/c/365792/ so once the parent merges we can merge this one too | 13:39 |
jistr | also passed on previous patchset, clean rebase it seems | 13:39 |
EmilienM | jistr: please look https://review.openstack.org/#/c/365735/ | 13:39 |
EmilienM | jistr: I just approved it, parent will merge, no need to wait | 13:39 |
EmilienM | when 365735 is merged, we can continue the rest of THT | 13:40 |
* rcrit hasn't had a successful tripleo install in days :-( | 13:40 | |
jistr | yea it has a depends-on too though, which didn't pass CI... i'll review them both | 13:41 |
rcrit | seeing a bunch of Error: Command exceeded timeout | 13:41 |
openstackgerrit | Merged openstack/tripleo-quickstart: Add back socket for ci-scripts - OVB and baremetal https://review.openstack.org/367545 | 13:41 |
*** jschluet is now known as jschlueter|afk | 13:41 | |
*** jschlueter|afk is now known as jschlueter | 13:41 | |
openstackgerrit | Lucas Alvares Gomes proposed openstack/tripleo-quickstart: Force the use of python/pip version 2 https://review.openstack.org/368013 | 13:42 |
*** rhallisey has quit IRC | 13:43 | |
EmilienM | jistr: https://review.openstack.org/#/c/362465/ can me merged | 13:43 |
EmilienM | jistr: because https://review.openstack.org/#/c/365735/ is full green | 13:44 |
*** rhallisey has joined #tripleo | 13:44 | |
EmilienM | slagle: no need to recheck imho | 13:48 |
EmilienM | slagle: CI was full green on https://review.openstack.org/#/c/365735/ | 13:48 |
EmilienM | slagle: which depends-on the patch you just rechecked | 13:48 |
EmilienM | jistr, slagle: though this one I have a doubt: https://review.openstack.org/#/c/315679/ I did recheck to make sure we don't break anything | 13:54 |
*** [1]cdearborn has joined #tripleo | 13:54 | |
jrist | ugh another phantomjs-prebuilt issue? | 13:58 |
jtomasek | honza: weird thing is that it is failing only on that single patch (and its dependent patches) | 13:59 |
*** fragatina has joined #tripleo | 14:01 | |
derekh | pabelanger: hmmm, maybe it is the named server, on the bastion (where named is running) I can see udp requests with no reponse | 14:03 |
derekh | pabelanger: line 9 (http://paste.openstack.org/show/570204/) | 14:03 |
dtrainor | jaosorior, ping | 14:03 |
*** pkovar has quit IRC | 14:03 | |
derekh | pabelanger: bnemec I'd imagine this would be relevant | 14:04 |
derekh | Sep 9 10:03:25 host01-rack01 kernel: nf_conntrack: table full, dropping packet. | 14:04 |
pabelanger | derekh: where is that from? | 14:05 |
derekh | pabelanger: bastion | 14:05 |
*** nyechiel has quit IRC | 14:05 | |
*** fragatina has quit IRC | 14:06 | |
*** skramaja has quit IRC | 14:06 | |
pabelanger | :( | 14:06 |
jaosorior | dtrainor: hey dude, you mentioned that you were gonna work on the UI SSL parts. Got any patches up already? Need any help? | 14:06 |
pabelanger | derekh: what is preventing us from removing the bastion server? | 14:07 |
dtrainor | I appreciate you reaching out, thanks. I got SSL working fine on the Undercloud, but I found what I believe to be a bug in UI that misconfigures the websocket url when the UI discovers that the Undercloud uses SSL/TLS | 14:07 |
*** akshai_ has joined #tripleo | 14:08 | |
jaosorior | dtrainor: what's the bug? | 14:08 |
jaosorior | dtrainor: did you take a look at the generate_service_certificate part? | 14:08 |
dtrainor | I need to run through the Undercloud SSL setup again a few more times.... I want to better understand how subaltname is used or referenced | 14:08 |
dtrainor | I did not because at the moment I'm tasked with some UI on OSP9 bits :) | 14:09 |
*** cdearborn has quit IRC | 14:09 | |
derekh | pabelanger: dunno, the rack was setup for all public traffic to come in via the bastion, I don't know how hard that would be to change | 14:09 |
derekh | pabelanger: looking into the connecting tracking limits now | 14:09 |
*** myoung has joined #tripleo | 14:09 | |
jaosorior | dtrainor: the websocket endpoint at the moment is not done by instack-undercloud, but that's coming. | 14:10 |
jaosorior | but then again, that's newton | 14:10 |
dtrainor | Understood. It's the UI that incorrectly generates the wss url | 14:10 |
dtrainor | give me a second here, let me go through some other notes, someone took a look at this overnight | 14:11 |
*** akshai has quit IRC | 14:11 | |
pabelanger | derekh: Ya, obviously we cannot remove it today. But we should maybe start the discussions around removing it. Not a fan of routing all tripleo-test-cloud-rh1 traffic thought it | 14:11 |
derekh | pabelanger: ok | 14:11 |
openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Use relative paths in ci-scripts for quickstart.sh https://review.openstack.org/360649 | 14:11 |
pabelanger | derekh: I'll start a thread on the ML about it, see if I can collect more information about it | 14:12 |
openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart: Document Third Party CI and developer usage https://review.openstack.org/360007 | 14:12 |
derekh | pabelanger: ack | 14:12 |
*** myoung is now known as myoung|bbl | 14:12 | |
jaosorior | bnemec: are you around yet? | 14:13 |
jaosorior | dtrainor: well, zaqar-websocket is not yet supported by haproxy | 14:14 |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: DNM: try to break tripleo-update job https://review.openstack.org/368025 | 14:14 |
jaosorior | dtrainor: got a patch that will hopefully do the trick, but that's not there yet | 14:14 |
*** skramaja has joined #tripleo | 14:14 | |
*** tobias-fiberdata has joined #tripleo | 14:14 | |
jaosorior | dtrainor: actually thrash and me were looking at that haproxy part | 14:14 |
jaosorior | dtrainor: but I do think you want to get the UI terminated by haproxy also | 14:14 |
*** mhenkel has joined #tripleo | 14:16 | |
*** tobias_fiberdata has quit IRC | 14:17 | |
EmilienM | slagle: can you approve https://review.openstack.org/#/c/366896/ please? | 14:18 |
EmilienM | pradk: AFIK scenario001 is red because of telemetry stuff (without ceph) - i'm looking into it | 14:22 |
EmilienM | http://logs.openstack.org/96/366896/4/check/gate-tripleo-ci-centos-7-scenario001-multinode-nv/97ed4b4/console.html#_2016-09-09_13_05_29_297314 | 14:22 |
*** pkovar has joined #tripleo | 14:22 | |
EmilienM | pradk: see this aodh error: http://logs.openstack.org/96/366896/4/check/gate-tripleo-ci-centos-7-scenario001-multinode-nv/97ed4b4/logs/subnode-2/var/log/aodh/app.txt.gz#_2016-09-09_13_05_24_956 | 14:23 |
EmilienM | ok I switch on telemetry channel | 14:24 |
*** jprovazn has quit IRC | 14:24 | |
*** mhenkel has quit IRC | 14:25 | |
EmilienM | ok so FTR about telemetry problem, they fixed it today, we need a promotion then | 14:28 |
dtrainor | jaosorior, yes i understand :) | 14:28 |
dtrainor | give me a few minutes here | 14:28 |
EmilienM | https://review.openstack.org/#/c/366638/ fixed it | 14:29 |
*** mburned is now known as mburned_out | 14:30 | |
bnemec | jaosorior: I am now. | 14:32 |
dtrainor | jaosorior, i understand that a lot of ssl related bits have changed between mitaka to newton and i'm totally ok with that. they're all great changes that help implement ssl much better. unfortunately i'm tasked with something that revolves around mitaka/osp9 right now, that's why i was working with those particular pieces of software. | 14:37 |
openstackgerrit | Justin Kilpatrick proposed openstack/tripleo-quickstart: Add documentation for add-on roles https://review.openstack.org/346733 | 14:37 |
jaosorior | bnemec: hey dude, so, actually based on your comment it turned out that I had really confused what the management network means :/ so I fixed up the patch to fix what had occured because of that, and to do what I really needed: https://review.openstack.org/#/c/367362/ what do you think? | 14:37 |
ccamacho | hey shardy, I finished to merge the first part of my changes for https://review.openstack.org/#/c/346799 can you check my last comment there? Now im hitting some un-related issues "Could not find class ::keystone::cors" but my env its kind of old, I'm going up update all and re launch the deployment. | 14:37 |
*** rajinir has joined #tripleo | 14:37 | |
jaosorior | dtrainor: oooh I see | 14:38 |
dtrainor | jaosorior, really looking forward to this one task of mine going away so i can immerse myself in all the osp10 ssl goodness | 14:44 |
bnemec | jaosorior: +2a | 14:44 |
slagle | EmilienM: hey, for the undercloud upgrades in tripleo.sh, i'm not sure we should be rm'ing all the delorean repo files, b/c delorean-current.repo is removed, which could actually be newer than what you're trying to upgrade to (rdo-trunk-master-tested) | 14:44 |
openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Add an action and workflow to update a plan https://review.openstack.org/367921 | 14:45 |
EmilienM | slagle: indeed | 14:45 |
slagle | EmilienM: why can't we just rerun repo_setup, and try an upgrade? | 14:45 |
pradk | EmilienM, sorry was afk.. cool you got it resolved | 14:45 |
EmilienM | slagle: I can't remember details of why I did that, but sure, let's try it ! | 14:45 |
*** mcornea has quit IRC | 14:46 | |
EmilienM | pradk: well, not really but see #rdo | 14:46 |
jaosorior | bnemec: can you check this one out too? https://review.openstack.org/#/c/367754/ it's along the same theme | 14:46 |
slagle | EmilienM: ok, i'll work on something | 14:46 |
bnemec | slagle: EmilienM: I think at one point our docs recommended that. | 14:46 |
EmilienM | slagle: merci | 14:46 |
pradk | EmilienM, yea new promotion needed for h fix | 14:46 |
bnemec | Why are we "upgrading" to an older repo anyway? | 14:46 |
slagle | bnemec: rerunning --repo-setup?. i think so | 14:46 |
EmilienM | pradk: promotion in upstream requirements | 14:46 |
bnemec | slagle: No, deleting the existing repo files. | 14:47 |
EmilienM | pradk: not in RDO, because RDO is waiting for upstream req update which requires FFE | 14:47 |
bnemec | Or maybe I just made that up when I did my undercloud upgrade. :-) | 14:47 |
slagle | bnemec: i guess it depends on what we're trying to test with this | 14:48 |
slagle | bnemec: i think it ought to be: not installed, --repo-setup, install. then: delorean-current for all packages, upgrade | 14:49 |
openstackgerrit | Martin André proposed openstack/tripleo-heat-templates: Have docker start script honor configuration https://review.openstack.org/366138 | 14:49 |
openstackgerrit | Martin André proposed openstack/tripleo-heat-templates: Add steps to containerized compute deployment https://review.openstack.org/346927 | 14:49 |
openstackgerrit | Martin André proposed openstack/tripleo-heat-templates: WIP: Containerized Services for Composable Roles https://review.openstack.org/330659 | 14:49 |
openstackgerrit | Martin André proposed openstack/tripleo-heat-templates: Bind mount files to run DiD in latest atomic host https://review.openstack.org/347218 | 14:49 |
openstackgerrit | Martin André proposed openstack/tripleo-heat-templates: Be more inclusive in insecure registry regex https://review.openstack.org/366111 | 14:49 |
slagle | bnemec: that would test that we can upgrade from our currently promoted repo to delorean-current | 14:49 |
*** mhenkel has joined #tripleo | 14:50 | |
bnemec | slagle: How would delorean-current be older than the pinned tripleo repo? | 14:50 |
slagle | it won't be | 14:51 |
*** dtrainor has quit IRC | 14:51 | |
*** dtrainor has joined #tripleo | 14:51 | |
bnemec | slagle: FWIW, that step came from http://docs.openstack.org/developer/tripleo-docs/installation/installation.html#updating-undercloud-components | 14:52 |
bnemec | I don't have a problem removing it as long as we're sure we won't end up with old repo files lying around. | 14:52 |
*** jcoufal has quit IRC | 14:53 | |
slagle | bnemec: ok, i think the issue is that what's done in undercloud_uprade doesnt match those docs | 14:53 |
slagle | in that it doesn't put back a delorean-current.repo and it uses rdo-trunk-master-tested instead of rdo-trunk-master-tripleo | 14:54 |
slagle | (not that i could even tell you the difference between those 2 fwiw) | 14:54 |
openstackgerrit | Carlos Camacho proposed openstack/puppet-tripleo: WIP - Add puppet-lint 2.0.0 to check puppet-tripleo manifests https://review.openstack.org/352847 | 14:55 |
bnemec | slagle: Yeah, the undercloud upgrade process is still pretty fscked, and I wish we weren't implementing it in tripleo.sh. | 14:55 |
*** myoung|bbl is now known as myoung | 14:56 | |
*** jcoufal has joined #tripleo | 14:56 | |
slagle | bnemec: we're not permanently. there is a note to clean it up, and EmilienM started a discussion about it on the ML | 14:56 |
slagle | i'd like to see it all in instack-undercloud | 14:57 |
bnemec | slagle: Yeah, but we're cutting Newton soon. What are the odds it gets moved to tripleoclient/instack-undercloud in time? | 14:57 |
*** pkovar has quit IRC | 14:58 | |
slagle | bnemec: i think that it is documentation until it is properly implemented, which doesnt have to be for Newton | 14:58 |
slagle | and that is what tripleo.sh is capturing | 14:58 |
slagle | same as always | 14:58 |
bnemec | I guess I'm not in a position to criticize since I completely fell of the undercloud upgrade wagon lately, but I feel like there are a lot of things that went into tripleo.sh "temporarily" and are still there a year+ later. | 14:58 |
bnemec | Anyway, I'll take a look at the ML thread. | 14:59 |
bnemec | Need to catch up on that anyway. | 14:59 |
slagle | bnemec: i think tripleo.sh matches the docs pretty closely tbh | 15:00 |
slagle | with the exception of this piece, which was just added | 15:00 |
*** jobewan has joined #tripleo | 15:00 | |
*** abehl has quit IRC | 15:01 | |
openstackgerrit | Dougal Matthews proposed openstack/python-tripleoclient: Update plan files when re-deploying https://review.openstack.org/366541 | 15:02 |
openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Add an action and workflow to update a plan https://review.openstack.org/367921 | 15:02 |
*** pkovar has joined #tripleo | 15:02 | |
openstackgerrit | Lucas Alvares Gomes proposed openstack/tripleo-quickstart: Force the use of python/pip version 2 https://review.openstack.org/368013 | 15:02 |
*** jaosorior has quit IRC | 15:07 | |
*** pcaruana has quit IRC | 15:08 | |
*** mburned_out is now known as mburned | 15:09 | |
*** abregman has quit IRC | 15:10 | |
*** ebarrera has quit IRC | 15:11 | |
*** egafford has joined #tripleo | 15:17 | |
bnemec | derekh: I see no recent nf_conntrack messages on the bastion. Is that fixed? | 15:19 |
*** mhenkel has quit IRC | 15:20 | |
derekh | bnemec: http://paste.openstack.org/show/570228/ | 15:20 |
derekh | bnemec: I've doubled the table size and reduced some of the timeouts, monitoring it at the moment to see how big the table grows | 15:21 |
*** egafford has quit IRC | 15:21 | |
bnemec | derekh: Cool. | 15:21 |
bnemec | derekh: How do you have root on the bastion? Do you have the password or did someone give your user sudo? | 15:22 |
*** mbound has quit IRC | 15:22 | |
*** jcoufal has quit IRC | 15:22 | |
*** egafford has joined #tripleo | 15:22 | |
derekh | bnemec: this would almost definitely have been causing all kinds of random stuff to go on | 15:22 |
bnemec | derekh: Yeah, now that you point out the conntrack messages I'm pretty sure I saw the same behavior when I was exhausting conntrack on my router at home. | 15:23 |
bnemec | At first I thought it was just bad package mirrors, but at some point I realized I was the only one having trouble with them. | 15:23 |
bnemec | So weird that we ran for a month, at times with a heavier workload than now, and never ran into this before. | 15:24 |
*** nyechiel has joined #tripleo | 15:24 | |
derekh | bnemec: I only got root recently, after we did the RAM upgrade and realized too few people have root (only wfoster and kambiz had it I think) | 15:24 |
derekh | bnemec: ya, something must have changed to all of a sudden start more connections | 15:25 |
bnemec | derekh: Dropping the timeouts seems to have helped. It's down to 60000. | 15:27 |
derekh | bnemec: ya, i'd love to know which one did the trick, is there a way to see what the connections the kernel was tracking I wonder | 15:28 |
openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Check the result of the parent action when subclassing https://review.openstack.org/368081 | 15:29 |
bnemec | derekh: conntrack -L seems to do that. | 15:33 |
*** leanderthal is now known as leanderthal|afk | 15:35 | |
*** jistr is now known as jistr|biab | 15:35 | |
marios | bandini: i'll have a more 'proper' look https://review.openstack.org/#/c/358626 monday sorry didn't get to cycle back yet | 15:37 |
*** jcoufal has joined #tripleo | 15:37 | |
bandini | marios: ah no worries at all, I will try to implement option C) as suggested by Jiri | 15:38 |
bandini | marios: it'd be great if we could land your rework asap | 15:39 |
marios | bandini: yes i know i wanted today but looks like monday it has to be | 15:39 |
bandini | marios: aye! you have a nice weekend ;) | 15:39 |
marios | you too bandini | 15:39 |
*** nyechiel has quit IRC | 15:40 | |
openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart: Pin setuptools to <27.1.1 https://review.openstack.org/368090 | 15:45 |
openstackgerrit | Merged openstack/puppet-tripleo: Add Ceph RGW profile https://review.openstack.org/334081 | 15:46 |
derekh | bnemec: /tmp/conntrack.list | 15:48 |
trown | adarazs: mind giving a quick look to https://review.openstack.org/#/c/368090/ before you take off, all our jobs will start failing without that | 15:48 |
derekh | bnemec: its almost all connections for DNS | 15:48 |
*** openstackgerrit has quit IRC | 15:49 | |
*** tesseract- has quit IRC | 15:49 | |
*** openstackgerrit has joined #tripleo | 15:49 | |
derekh | bnemec: every dns request must open a udp stream that stays open until it timesout | 15:49 |
*** jlinkes has quit IRC | 15:49 | |
trown | ah yep linter jobs failed on the same thing, though not sure how we fix those | 15:50 |
trown | hmm yep #openstack-infra has seen the same | 15:51 |
bnemec | derekh: Yeah, we must have done something that is resulting in more DNS lookups recently. | 15:51 |
*** aufi has quit IRC | 15:52 | |
-openstackstatus- NOTICE: New setuptools release appears to have a circular import which is breaking many jobs - check for ImportError: cannot import name monkey. | 15:56 | |
*** ChanServ changes topic to "New setuptools release appears to have a circular import which is breaking many jobs - check for ImportError: cannot import name monkey." | 15:56 | |
*** jcoufal has quit IRC | 15:56 | |
trown | hmm seems odd to change the topic of every channel | 15:56 |
*** jcoufal has joined #tripleo | 15:56 | |
*** dtantsur is now known as dtantsur|afk | 15:59 | |
openstackgerrit | Merged openstack/tripleo-heat-templates: Move role deployment steps into puppet/post.yaml https://review.openstack.org/365763 | 16:00 |
lucasagomes | trown, have you seem this running ooo-quickstart: http://paste.openstack.org/show/570247/ ? | 16:00 |
bkero | lucasagomes: means python-tripleoclient isn't installed IIRC | 16:00 |
trown | lucasagomes: that would probably mean you are using an undercloud image that does not have python-tripleoclient installed | 16:00 |
openstackgerrit | Merged openstack/tripleo-heat-templates: Remove *ExtraConfig parameters from overcloud.yaml https://review.openstack.org/365792 | 16:00 |
openstackgerrit | Merged openstack/tripleo-common: Wire in jinja templating for custom roles https://review.openstack.org/362465 | 16:00 |
lucasagomes | bkero, trown right, just wondering if you seem it cause I'm just using the normal devmode thingy | 16:01 |
lucasagomes | it should get the images from ooo-ci right? | 16:01 |
trown | lucasagomes: ya, unfortunately devmode is in a bit of a rough state, it was constructed around having an instack.qcow2 from ooo-ci, but that is no longer being produced and quite out of day | 16:02 |
trown | date | 16:02 |
lucasagomes | trown, right | 16:02 |
trown | I am working on making it consume the overcloud-full.qcow2 as an undercloud image now | 16:02 |
lucasagomes | ack, I will look into it | 16:02 |
trown | it is a bit tricky, but I have had some success today | 16:02 |
zoli|wfh | have a good weekend | 16:04 |
*** zoli|wfh is now known as zoli|gone | 16:04 | |
*** yamahata has joined #tripleo | 16:06 | |
*** limao has joined #tripleo | 16:08 | |
*** skramaja has quit IRC | 16:09 | |
*** akshai_ has quit IRC | 16:12 | |
*** limao has quit IRC | 16:12 | |
openstackgerrit | James Slagle proposed openstack/tripleo-docs: Update docs for undercloud upgrade https://review.openstack.org/368138 | 16:16 |
*** lmiccini has quit IRC | 16:21 | |
*** florianf has quit IRC | 16:21 | |
*** zoli|gone is now known as zoli_gone-proxy | 16:21 | |
*** jcoufal has quit IRC | 16:22 | |
*** mbound has joined #tripleo | 16:22 | |
*** jcoufal has joined #tripleo | 16:24 | |
openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: Use current-tripleo for undercloud upgrades https://review.openstack.org/368142 | 16:24 |
EmilienM | nice ^ | 16:24 |
*** bana_k has joined #tripleo | 16:25 | |
*** akshai has joined #tripleo | 16:25 | |
*** jistr|biab is now known as jistr | 16:25 | |
openstackgerrit | Julie Pichon proposed openstack/tripleo-common: Mistral workflow for creating a deployment plan from JSON input https://review.openstack.org/347830 | 16:26 |
*** mbound has quit IRC | 16:28 | |
*** jbadiapa has quit IRC | 16:30 | |
*** fragatina has joined #tripleo | 16:32 | |
*** snecklifter has quit IRC | 16:33 | |
*** athomas has quit IRC | 16:34 | |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: DNM: try to break tripleo-update job https://review.openstack.org/368025 | 16:35 |
*** dsneddon has joined #tripleo | 16:38 | |
*** akrivoka has quit IRC | 16:44 | |
*** lucasagomes is now known as lucas-afk | 16:44 | |
*** lmiccini has joined #tripleo | 16:47 | |
openstackgerrit | Julie Pichon proposed openstack/tripleo-common: Remove the old, deprecated Mistral action names. https://review.openstack.org/366529 | 16:48 |
*** akrivoka has joined #tripleo | 16:52 | |
*** tosky has quit IRC | 16:52 | |
*** jpich has quit IRC | 16:52 | |
*** derekh has quit IRC | 16:54 | |
*** trown is now known as trown|lunch | 16:56 | |
openstackgerrit | Ryan Brady proposed openstack/tripleo-common: [WIP] Port password generation from tripleoclient to tripleo-common https://review.openstack.org/368150 | 16:56 |
*** ohamada has quit IRC | 16:58 | |
*** ChanServ changes topic to "TripleO : http://tripleo.org/ | https://wiki.openstack.org/wiki/TripleO | Meetings On Tuesdays at 14:00 UTC in #openstack-meeting-alt" | 16:59 | |
-openstackstatus- NOTICE: setuptools 27.1.2 addresses the circular import | 16:59 | |
*** rodrigods has quit IRC | 17:03 | |
*** rodrigods has joined #tripleo | 17:03 | |
openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui: Increase severity of all checks to error https://review.openstack.org/368152 | 17:03 |
*** yamahata has quit IRC | 17:06 | |
*** bana_k has quit IRC | 17:08 | |
*** chem has quit IRC | 17:10 | |
*** jpena is now known as jpena|off | 17:11 | |
bnemec | EmilienM: Just a heads up, the ovb jobs look good now that we've resolved the conntrack issues on the bastion. | 17:17 |
EmilienM | oh what a good news | 17:17 |
*** pkovar has quit IRC | 17:19 | |
slagle | EmilienM: instead of rechecking all the custom roles patches, maybe we can just recheck every 3rd or 4th one | 17:21 |
slagle | EmilienM: e.g., i rechecked https://review.openstack.org/#/c/364749/, if that passes, we can merge that and the 3 prior | 17:21 |
EmilienM | yes, this is a good idea | 17:21 |
bnemec | slagle: EmilienM: Note that a bunch of those failed the multinode job the same way they did the ovb jobs. e.g. https://review.openstack.org/#/c/315679/37 | 17:22 |
bnemec | They seem to be hanging at overcloud deploy time. | 17:22 |
*** bana_k has joined #tripleo | 17:22 | |
EmilienM | not great, though we haven't change it recently | 17:22 |
slagle | bnemec: oh, is that something on our side? | 17:24 |
slagle | i guess it might be | 17:25 |
slagle | let me try a local test | 17:27 |
bnemec | slagle: I don't know, but the fact that it's happening consistently in both ovb and multinode makes me think it's not an infrastructure problem. | 17:27 |
bnemec | Maybe we have an infinite jinja loop. | 17:28 |
* bnemec trademarks that as his future band name | 17:28 | |
*** bnemec is now known as beekneemech | 17:28 | |
slagle | i totally rave to that | 17:29 |
*** jcoufal has quit IRC | 17:30 | |
*** flepied has quit IRC | 17:30 | |
EmilienM | and https://review.openstack.org/#/c/365735/ | 17:32 |
EmilienM | mhh | 17:33 |
openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui: Increase severity of all checks to error https://review.openstack.org/368152 | 17:35 |
EmilienM | I'm trying to find what we could have broken | 17:37 |
EmilienM | https://review.openstack.org/#/c/362465/ was full green | 17:38 |
*** fragatina has quit IRC | 17:39 | |
*** fragatina has joined #tripleo | 17:43 | |
*** yamahata has joined #tripleo | 17:44 | |
slagle | yes, but that's not actually used until https://review.openstack.org/#/c/315679/ is tested | 17:47 |
slagle | and the depends-on tripleoclient patch | 17:47 |
*** fragatina has quit IRC | 17:48 | |
EmilienM | I ran recheck on https://review.openstack.org/#/c/365735/ to see if it was transient | 17:48 |
slagle | k. i'm trying it locally as well | 17:49 |
slagle | had to rebuild new oc images though, so it will be a few | 17:49 |
openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui: Remove unused import https://review.openstack.org/368167 | 17:53 |
openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui: Increase severity of all checks to error https://review.openstack.org/368152 | 17:55 |
*** trown|lunch is now known as trown | 17:55 | |
*** akshai has quit IRC | 18:05 | |
*** akshai has joined #tripleo | 18:12 | |
slagle | EmilienM: ok, so testing the tht and tripleoclient patch together, it does just hang for me | 18:27 |
slagle | at "Creating plan from template files in: tripleo-root/tripleo-heat-templates/" | 18:27 |
slagle | according to mistral, the error is Failure caused by error in tasks: notify_zaqar | 18:27 |
EmilienM | damn | 18:28 |
EmilienM | slagle: have you this one too ? https://review.openstack.org/#/c/362465/ | 18:28 |
slagle | yes | 18:28 |
EmilienM | in http://tripleo.org/cistatus.html I don't see much red | 18:29 |
EmilienM | I'm wondering if it's transient or not | 18:29 |
EmilienM | rbrady: do you have any idea of what we could miss here? | 18:30 |
EmilienM | slagle: see https://review.openstack.org/#/c/366541/8 also | 18:30 |
*** flepied has joined #tripleo | 18:30 | |
EmilienM | are you redeploying? | 18:30 |
EmilienM | https://bugs.launchpad.net/tripleo/+bug/1620932 | 18:30 |
openstack | Launchpad bug 1620932 in tripleo "openstack overcloud deploy will use the old plan and wont update on a second deploy" [Critical,In progress] - Assigned to Dougal Matthews (d0ugal) | 18:30 |
EmilienM | it also depends on https://review.openstack.org/#/c/367921/ | 18:31 |
slagle | ZaqarAction.queue_post failed: <class 'zaqarclient.transport.errors.MalformedRequest'>: Error response from Zaqar. Code: 400. Title: Invalid API request. Description: Message collection size is too large. Max size 524288. | 18:31 |
slagle | ] (execution_id=e36ebe4e-cce7-4ea6-be79-4ac64156fb62) | 18:31 |
EmilienM | oh I saw that already | 18:31 |
slagle | i'm not updating, i'm starting clean each time | 18:31 |
EmilienM | we increased it recently | 18:31 |
slagle | is there a fix? | 18:31 |
slagle | i updated my undercloud, maybe i didnt get it | 18:32 |
EmilienM | let me find it again | 18:32 |
EmilienM | https://review.openstack.org/#/c/344537/ | 18:33 |
EmilienM | dprince did that 7 weeks ago | 18:33 |
slagle | EmilienM: look at the error i posted though :( | 18:33 |
slagle | we're exceeding it again | 18:33 |
EmilienM | sounds like 524288 is too small | 18:34 |
*** fragatina has joined #tripleo | 18:34 | |
*** jcoufal has joined #tripleo | 18:34 | |
*** fragatina has quit IRC | 18:34 | |
EmilienM | can we have thoughts from d0ugal, jrist, rbrady on this thing? ^ | 18:34 |
EmilienM | do we need to increase it? | 18:34 |
*** fragatina has joined #tripleo | 18:34 | |
*** noslzzp has quit IRC | 18:36 | |
slagle | dunno enough right now to know if it's reasonable to increase it or not | 18:37 |
EmilienM | slagle: do have the size of the message by any chance? | 18:37 |
EmilienM | maybe in mistral or zaqar's logs | 18:37 |
EmilienM | thrash: any idea? | 18:37 |
slagle | EmilienM: no. zaqar doesnt even seem to be fully configured to log to it's logfile | 18:38 |
slagle | and in the journal output all it shows is it returning 400's for bad request | 18:39 |
thrash | EmilienM: slagle sounds to me like we are trying to pass too much around in zaqar... | 18:39 |
slagle | flaper87: any thoughts about max zaqar message size? how big can we safely make it? | 18:40 |
EmilienM | vkmc: ^ too :) | 18:41 |
slagle | this is random ping friday | 18:41 |
thrash | lol | 18:41 |
EmilienM | at this stage of cycle, I have no shame to ping people | 18:41 |
EmilienM | and find solutions | 18:42 |
EmilienM | :) | 18:42 |
EmilienM | slagle: can you just for fun increase the size in zaqar.conf and retry? | 18:42 |
slagle | sure, let me clean up first | 18:43 |
slagle | just need to delete 10 different things | 18:43 |
slagle | using 15 different api's | 18:44 |
EmilienM | I guess the size is in bytes? | 18:44 |
* EmilienM looks in zaqar code | 18:44 | |
EmilienM | in doc they set 262144 by default | 18:46 |
EmilienM | the default in code is 262144 too | 18:47 |
slagle | and we doubled that | 18:47 |
slagle | i'll try doubling again | 18:47 |
EmilienM | I'm not sure if I'm true, but iirc the most templates we have, more message is big | 18:48 |
EmilienM | and I guess bigger is mistral plan, bigger is zaqar message | 18:48 |
slagle | ok, it's deploying now, but I got an error on the cli | 18:48 |
slagle | Object GET failed: https://192.0.2.2:13808/v1/AUTH_870ac3192305413ea9758ce9419350b9/overcloud/overcloud-without-mergepy.yaml 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.< | 18:48 |
slagle | it's still going though, so that must not be fatal | 18:49 |
EmilienM | it's only 524,588 kb | 18:50 |
EmilienM | maybe the plan is bigger | 18:50 |
beekneemech | Wait, we completely removed without-mergepy. Why is the shiny brand new api looking for it at all? | 18:50 |
beekneemech | EmilienM: You mean bytes? | 18:51 |
EmilienM | yes | 18:51 |
slagle | beekneemech: tripleoclient still prefers that name | 18:51 |
EmilienM | slagle: also, your comment, have you seen this error? http://logs.openstack.org/79/315679/37/check/gate-tripleo-ci-centos-7-nonha-multinode/4233b5f/console.html#_2016-09-09_14_28_59_666566 | 18:51 |
openstackgerrit | Merged openstack/tripleo-common: Run copy_ssh_keys after overcloud finishes https://review.openstack.org/367422 | 18:52 |
openstackgerrit | Merged openstack-infra/tripleo-ci: scenarios: set Debug to True https://review.openstack.org/366896 | 18:52 |
slagle | EmilienM: it means the image build failed, and that did not fail the CI job for some reason | 18:52 |
slagle | EmilienM: or possibly the image failed to upload to glance | 18:53 |
beekneemech | That's a multi-node job, there aren't supposed to be images. | 18:53 |
openstackgerrit | Merged openstack/tripleo-heat-templates: Generate VIP info for ctlplane VIP, not management https://review.openstack.org/367362 | 18:53 |
slagle | oh, it was a trick question | 18:53 |
beekneemech | :-) | 18:53 |
EmilienM | yes | 18:53 |
slagle | yea, you can ignore that | 18:53 |
slagle | EmilienM: that job is timing out because of the same zaqar message size | 18:53 |
openstackgerrit | Merged openstack/tripleo-common: Remove "type: direct" from workflows as it is the default https://review.openstack.org/341617 | 18:54 |
EmilienM | this is sad we don't have logs | 18:54 |
slagle | you can see it hung at the same place | 18:54 |
openstackgerrit | Justin Kilpatrick proposed openstack/tripleo-quickstart: Add documentation for add-on roles https://review.openstack.org/346733 | 19:17 |
*** isq has quit IRC | 19:17 | |
bandini | matbu: ok so I narrowed down the commit that broke it, but I am quite unsure if it is a heat issue or a tht one | 19:21 |
*** lmiccini has quit IRC | 19:22 | |
EmilienM | therve: tl;dr: we are seeing errors in tripleo because zaqar messages reached the size limit | 19:26 |
EmilienM | we already doubled the default size on the undercloud | 19:26 |
EmilienM | and were wondering if it would be safe to increase it | 19:26 |
therve | Sure I don't see why not, especially in the undercloud | 19:26 |
EmilienM | current size is 524288 | 19:26 |
EmilienM | ok | 19:26 |
EmilienM | we're seeing transient issues with this thing | 19:26 |
therve | That sounds like not too small for config transport, if you're using it for that | 19:27 |
EmilienM | therve: we use it for plans | 19:27 |
EmilienM | so yes, config transport I guess | 19:28 |
slagle | why does the whole plan have to go through the zaqar notification though? | 19:28 |
therve | We had that issue in the CFN transport IIRC, and it was 1M | 19:28 |
therve | I don't know what a tripleo plan is :/ | 19:29 |
slagle | yea it was mostly a rhetorical question :) | 19:31 |
slagle | nobody knows | 19:31 |
*** tzumainn has quit IRC | 19:32 | |
therve | The config transport still seems to be POLL_SERVER_CFN in tht | 19:34 |
therve | EmilienM, Have you managed to raise it? | 19:35 |
EmilienM | therve: raise the bug you mean? | 19:35 |
slagle | i raised it to 1M in my environment, and it seemed to work fine | 19:35 |
EmilienM | or raise the size? | 19:35 |
therve | The size | 19:35 |
EmilienM | slagle: good | 19:35 |
slagle | i'll submit a patch | 19:35 |
slagle | and add a depends-on | 19:35 |
*** isq has joined #tripleo | 19:35 | |
EmilienM | thanks, I think we can go for it. Worst case we have bad feedback from Mistral folks | 19:36 |
EmilienM | therve: thx for your help also | 19:36 |
*** kjw3 has joined #tripleo | 19:36 | |
therve | Np | 19:36 |
openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates: Add contrail services to the resource registry defaulted to OS::Heat::None https://review.openstack.org/368196 | 19:39 |
*** fultonj has quit IRC | 19:39 | |
openstackgerrit | James Slagle proposed openstack/instack-undercloud: Raise zaqar max message size https://review.openstack.org/368197 | 19:41 |
*** fultonj has joined #tripleo | 19:41 | |
openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Convert overcloud.yaml to support jinja2 templating https://review.openstack.org/315679 | 19:42 |
ayoung | rcrit, I wonder if other people working on tripleo have seen the database failures you had here https://paste.fedoraproject.org/424979/14734499/ | 19:43 |
openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Convert overcloud.yaml to support jinja2 templating https://review.openstack.org/315679 | 19:43 |
ayoung | rcrit, alee said he saw it too, but in a different place. I wonder if there is a problem with alembic | 19:44 |
*** rbrady has quit IRC | 19:44 | |
rcrit | this is a new place for me too previous was in lbaas_something or other | 19:45 |
openstackgerrit | James Slagle proposed openstack/python-tripleoclient: Get template contents from plan, not local path https://review.openstack.org/365735 | 19:45 |
openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Convert overcloud.yaml to support jinja2 templating https://review.openstack.org/315679 | 19:46 |
*** ccamacho has quit IRC | 19:49 | |
ayoung | rcrit, asking the quickstart folks, although I doubt this is a quickstart issue. That error comes during the undercloud deploy | 19:50 |
rcrit | where do they live? | 19:51 |
openstackgerrit | Merged openstack/tripleo-quickstart: Fix all ci-scripts to use $LOCATION https://review.openstack.org/367300 | 19:53 |
*** saneax is now known as saneax-_-|AFK | 19:54 | |
ayoung | rcrit, #oooq | 19:55 |
rcrit | wow | 19:55 |
trown | this channel is fine for quickstart too | 19:58 |
openstackgerrit | Merged openstack/puppet-tripleo: Added Contrail Composable Services https://review.openstack.org/348905 | 20:10 |
openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Limit Heat engine workers to 4 in rh1 https://review.openstack.org/360650 | 20:13 |
openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Limit heat, neutron, and nova workers https://review.openstack.org/368220 | 20:13 |
*** jayg is now known as jayg|g0n3 | 20:18 | |
*** mbound has joined #tripleo | 20:20 | |
jrist | EmilienM: I'm not sure we're effected too much, but bigger is ok with us too I bet | 20:23 |
jrist | 2^20! | 20:23 |
jrist | 2^21! | 20:23 |
openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Convert UpdateWorkflow to support composable roles https://review.openstack.org/367282 | 20:24 |
openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Move AllNodesValidationDeployments into jinja template loop https://review.openstack.org/337587 | 20:24 |
openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Convert overcloud.yaml to support jinja2 templating https://review.openstack.org/315679 | 20:24 |
openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Convert allNodesConfig properties to composable jinja2 https://review.openstack.org/365794 | 20:24 |
openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Move AllNodesDeployments into jinja template loop https://review.openstack.org/337267 | 20:24 |
openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Move role ResourceGroups inside the jinja2 loop https://review.openstack.org/365793 | 20:24 |
openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Move per-role NetIpListMap's into jinja template loop https://review.openstack.org/364749 | 20:24 |
openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Convert AllNodesExtraConfig to support composable roles https://review.openstack.org/367295 | 20:24 |
openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates: Convert deploy steps to jinja2 loop https://review.openstack.org/365796 | 20:24 |
*** dciabrin has quit IRC | 20:26 | |
*** pabelanger is now known as pall | 20:29 | |
*** lblanchard has quit IRC | 20:32 | |
*** lmiccini has joined #tripleo | 20:39 | |
d0ugal | https://review.openstack.org/#/c/367921/ and https://review.openstack.org/#/c/366541/ are green and ready for reviews if anyone wants to look at two critical bugs :) | 20:39 |
openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Set Deployment Parameters https://review.openstack.org/365625 | 20:40 |
openstackgerrit | Dougal Matthews proposed openstack/tripleo-common: Check the result of the parent action when subclassing https://review.openstack.org/368081 | 20:40 |
*** dciabrin has joined #tripleo | 20:40 | |
*** jcoufal has quit IRC | 20:43 | |
slagle | d0ugal: +A'd. | 20:44 |
*** mnaser has joined #tripleo | 20:44 | |
slagle | d0ugal: now if you have time you can review https://review.openstack.org/#/c/365735/ and https://review.openstack.org/#/c/368197/ | 20:44 |
slagle | :-) | 20:44 |
mnaser | how crazy of an idea is it to migrate an existing deployment to tripleo? (if we have the services running already) | 20:45 |
mnaser | i'm thinking moving the control plane to tripleo could be as (probably not) simple as moving dbs over, restarting services, repointing compute/agents to new control plane | 20:45 |
thrash | d0ugal: slagle by all means https://review.openstack.org/#/c/339048/ | 20:46 |
thrash | :) | 20:46 |
thrash | lol | 20:46 |
thrash | a bug that's been sitting there a while. | 20:46 |
slagle | mnaser: if you're planning on redeploying the control plane on new hardware, it might be possible, but would not be very straightforward | 20:48 |
thrash | slagle: I went ahead and +2+A on 368197 since that's holding up CI | 20:49 |
slagle | thrash: yea, good idea | 20:49 |
d0ugal | slagle: +2ed the first, thrash beat me to the second | 20:51 |
d0ugal | thrash: LGTM | 20:52 |
d0ugal | thrash: Any idea why Juan didn't +2? | 20:52 |
d0ugal | OKay, I need to go now anyway, kinda late here :) | 20:53 |
* d0ugal leaves for the weekend | 20:53 | |
openstackgerrit | Merged openstack/tripleo-specs: Update bug tag policy to match actual tag names https://review.openstack.org/363477 | 20:55 |
*** openstackstatus has quit IRC | 20:57 | |
*** pall is now known as pabelanger | 21:00 | |
*** openstackstatus has joined #tripleo | 21:00 | |
*** ChanServ sets mode: +v openstackstatus | 21:00 | |
*** akrivoka has quit IRC | 21:06 | |
openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates: Adjust MTU to compensate for VLAN tag issue https://review.openstack.org/368233 | 21:16 |
*** jeckersb_gone is now known as jeckersb | 21:16 | |
openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates: Adjust MTU to compensate for VLAN tag issue https://review.openstack.org/368233 | 21:16 |
*** mbound has quit IRC | 21:19 | |
*** trown is now known as trown|outtypewww | 21:21 | |
*** mburned is now known as mburned_out | 21:29 | |
*** mwhahaha has quit IRC | 21:32 | |
*** akshai has quit IRC | 21:32 | |
*** abregman has joined #tripleo | 21:34 | |
*** mwhahaha has joined #tripleo | 21:52 | |
*** [1]cdearborn has quit IRC | 22:15 | |
*** yamahata has quit IRC | 22:22 | |
*** ipsecguy has quit IRC | 22:24 | |
*** ipsecguy has joined #tripleo | 22:25 | |
*** abregman has quit IRC | 22:28 | |
*** ipsecguy has quit IRC | 22:31 | |
*** ipsecguy has joined #tripleo | 22:31 | |
*** rlandy has quit IRC | 22:33 | |
*** yamahata has joined #tripleo | 22:37 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-common: Updated from global requirements https://review.openstack.org/361501 | 22:54 |
*** jobewan has quit IRC | 23:11 | |
*** Ryjedo has quit IRC | 23:16 | |
openstackgerrit | Merged openstack/tripleo-heat-templates: add composable services for Contrail https://review.openstack.org/346799 | 23:19 |
*** limao has joined #tripleo | 23:20 | |
*** limao has quit IRC | 23:24 | |
*** pradk has quit IRC | 23:27 | |
openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates: Add fluentd client service https://review.openstack.org/353506 | 23:37 |
*** mnaser has left #tripleo | 23:38 | |
*** bana_k has quit IRC | 23:43 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!