| *** harlowja has quit IRC | 00:06 | |
| *** VW has joined #openstack-operators | 00:15 | |
| *** VW has quit IRC | 00:20 | |
| *** marst has quit IRC | 00:20 | |
| *** harlowja has joined #openstack-operators | 00:34 | |
| *** saneax is now known as saneax-_-|AFK | 00:58 | |
| *** markvoelker has joined #openstack-operators | 01:14 | |
| *** mriedem has joined #openstack-operators | 01:17 | |
| *** markvoelker has quit IRC | 01:19 | |
| *** harlowja has quit IRC | 01:25 | |
| *** bradm has quit IRC | 01:55 | |
| *** bradm has joined #openstack-operators | 01:58 | |
| mnaser | does anyone run multiple nova-schedulers? i think i ran into an issue where our database server was overwhelmed | 02:06 |
|---|---|---|
| mnaser | because every update was happening N times (# of nova-scheduler processes) | 02:06 |
| mnaser | as hypervisors submitted their state, every scheduler instance picked it up and updated it | 02:06 |
| mnaser | this is proven by running tail -f /var/log/nova/nova-scheduler.log across all 3 controllers and we can see all 3 running update for the same | 02:07 |
| *** karad has joined #openstack-operators | 02:15 | |
| *** armax has quit IRC | 02:27 | |
| *** markvoelker has joined #openstack-operators | 02:31 | |
| *** markvoelker has quit IRC | 02:36 | |
| *** ducttape_ has joined #openstack-operators | 02:58 | |
| *** ducttape_ has quit IRC | 03:02 | |
| *** ducttape_ has joined #openstack-operators | 03:04 | |
| *** fragatina has quit IRC | 03:07 | |
| *** fragatina has joined #openstack-operators | 03:08 | |
| *** fragatina has quit IRC | 03:12 | |
| *** fragatina has joined #openstack-operators | 03:12 | |
| *** fragatin_ has joined #openstack-operators | 03:15 | |
| *** fragatina has quit IRC | 03:15 | |
| *** fragatin_ has quit IRC | 03:18 | |
| *** sudipto has joined #openstack-operators | 03:19 | |
| *** fragatina has joined #openstack-operators | 03:19 | |
| *** fragatina has quit IRC | 03:24 | |
| *** ducttape_ has quit IRC | 03:26 | |
| *** VW has joined #openstack-operators | 03:43 | |
| *** catintheroof has quit IRC | 04:06 | |
| *** catintheroof has joined #openstack-operators | 04:06 | |
| *** catintheroof has quit IRC | 04:06 | |
| *** ducttape_ has joined #openstack-operators | 04:26 | |
| *** slaweq_ has quit IRC | 04:31 | |
| *** ducttape_ has quit IRC | 04:32 | |
| *** ducttape_ has joined #openstack-operators | 04:33 | |
| *** karad has quit IRC | 04:37 | |
| *** VW has quit IRC | 04:37 | |
| haukebruno | mnaser, after a quick check I can confirm that | 04:44 |
| haukebruno | updates appears on both of our controllers too | 04:44 |
| *** udesale has joined #openstack-operators | 04:51 | |
| *** ducttape_ has quit IRC | 05:00 | |
| *** fragatina has joined #openstack-operators | 05:26 | |
| *** sticker has quit IRC | 05:33 | |
| *** ecrehar has joined #openstack-operators | 05:36 | |
| *** fragatina has quit IRC | 05:36 | |
| *** fragatina has joined #openstack-operators | 05:36 | |
| *** simon-AS559 has joined #openstack-operators | 05:42 | |
| *** saneax-_-|AFK is now known as saneax | 05:44 | |
| *** ducttape_ has joined #openstack-operators | 06:00 | |
| *** sorrison has quit IRC | 06:04 | |
| *** ducttape_ has quit IRC | 06:05 | |
| *** rcernin has quit IRC | 06:13 | |
| *** rcernin has joined #openstack-operators | 06:37 | |
| *** armax has joined #openstack-operators | 06:38 | |
| *** rcernin has quit IRC | 06:43 | |
| *** rcernin has joined #openstack-operators | 06:54 | |
| *** Oku_OS-away is now known as Oku_OS | 06:55 | |
| *** ducttape_ has joined #openstack-operators | 07:01 | |
| *** ducttape_ has quit IRC | 07:07 | |
| *** paramite has joined #openstack-operators | 07:09 | |
| *** derekh has joined #openstack-operators | 07:40 | |
| *** derekh has quit IRC | 07:47 | |
| *** simon-AS559 has quit IRC | 07:54 | |
| *** pcaruana has joined #openstack-operators | 07:56 | |
| *** tesseract has joined #openstack-operators | 07:57 | |
| *** tesseract is now known as Guest31304 | 07:58 | |
| *** fragatina has quit IRC | 08:09 | |
| *** jsheeren has joined #openstack-operators | 08:11 | |
| *** matrohon has joined #openstack-operators | 08:23 | |
| *** udesale has quit IRC | 08:44 | |
| *** med_ has quit IRC | 08:53 | |
| *** med_ has joined #openstack-operators | 08:57 | |
| *** med_ is now known as Guest67717 | 08:57 | |
| *** udesale has joined #openstack-operators | 09:00 | |
| *** ducttape_ has joined #openstack-operators | 09:04 | |
| *** ducttape_ has quit IRC | 09:09 | |
| *** rmart04 has joined #openstack-operators | 09:14 | |
| *** simon-AS559 has joined #openstack-operators | 09:24 | |
| *** simon-AS5591 has joined #openstack-operators | 09:37 | |
| *** simon-AS559 has quit IRC | 09:37 | |
| *** derekh has joined #openstack-operators | 09:41 | |
| yankcrime | mnaser haukebruno yup, sadly i see the same here | 09:55 |
| yankcrime | i'd never noticed that before :( | 09:56 |
| yankcrime | thanks mnaser ;/ | 09:56 |
| yankcrime | have you come across any kind of bug report for this? | 09:56 |
| haukebruno | me neither. but this isn't a big deal for me now :p our stacks are small and we have an overpowered galera cluster in each of it | 09:56 |
| yankcrime | yeah, i'm in the same boat | 09:57 |
| yankcrime | we've a fair number of hypervisors but our galera cluster is meaty enough to not break much of a sweat | 09:57 |
| yankcrime | i guess this could be solved by loadbalancer configuration for now, but i like distributing load as evenly as possible for obvious reasons | 09:58 |
| yankcrime | especially for busier components such as nova | 09:58 |
| yankcrime | ah no wait, it couldn't in this case - that's a stupid suggestion | 09:59 |
| haukebruno | jep. we run our galera clusers with 5 nodes, all nova services talk to a specific db nod and all other services use other nodes to talk too | 09:59 |
| yankcrime | hmm, now i'm doing some reading i'm coming across some other considerations for nova-scheduler in active/active | 10:03 |
| yankcrime | i.e scheduler_host_subset_size | 10:03 |
| *** ducttape_ has joined #openstack-operators | 10:05 | |
| *** ducttape_ has quit IRC | 10:09 | |
| haukebruno | what is the suggest there? set it to a higher value? while reading the nova.conf reference I am not sure how that can help in active/active situations | 10:11 |
| *** simon-AS5591 has quit IRC | 10:17 | |
| yankcrime | haukebruno: https://bugzilla.redhat.com/show_bug.cgi?id=1104219 | 10:33 |
| openstack | bugzilla.redhat.com bug 1104219 in openstack-foreman-installer "[RFE][nova] Optionally allow nova-scheduler to be deployed as Active/Active or Active/Passive" [Unspecified,Closed: errata] - Assigned to jguiditt | 10:33 |
| *** ducttape_ has joined #openstack-operators | 11:05 | |
| *** vijaykc4 has joined #openstack-operators | 11:10 | |
| *** ducttape_ has quit IRC | 11:10 | |
| *** bdeschenes has joined #openstack-operators | 11:26 | |
| *** VW has joined #openstack-operators | 11:38 | |
| *** VW has quit IRC | 11:42 | |
| *** ducttape_ has joined #openstack-operators | 12:06 | |
| *** ducttape_ has quit IRC | 12:11 | |
| *** paramite_ has joined #openstack-operators | 12:27 | |
| *** catintheroof has joined #openstack-operators | 12:34 | |
| *** karad has joined #openstack-operators | 12:38 | |
| *** markvoelker has joined #openstack-operators | 12:45 | |
| *** jamesdenton has joined #openstack-operators | 12:47 | |
| *** jamesden_ has joined #openstack-operators | 12:50 | |
| *** jamesdenton has quit IRC | 12:52 | |
| *** vijaykc4 has quit IRC | 12:54 | |
| *** zioproto has joined #openstack-operators | 13:02 | |
| *** simon-AS559 has joined #openstack-operators | 13:05 | |
| zioproto | Hey there | 13:06 |
| zioproto | I am in liberty | 13:06 |
| zioproto | I have few hypervisors with VMs with local SSDs | 13:06 |
| *** simon-AS5591 has joined #openstack-operators | 13:06 | |
| zioproto | I would like to live migrate them | 13:06 |
| zioproto | but of course I have | 13:07 |
| zioproto | Live Migration failure: Cannot access storage file | 13:07 |
| zioproto | is it a common nova bug ? | 13:07 |
| *** ducttape_ has joined #openstack-operators | 13:07 | |
| zioproto | It is funny because permissions are root:root and the qemu processes run as root | 13:07 |
| zioproto | simon-AS559: hey there | 13:07 |
| *** jsheeren has quit IRC | 13:09 | |
| *** simon-AS559 has quit IRC | 13:10 | |
| *** jamesden_ has quit IRC | 13:12 | |
| *** karad has quit IRC | 13:13 | |
| *** simon-AS5591 has quit IRC | 13:25 | |
| *** armax has quit IRC | 13:28 | |
| *** fragatina has joined #openstack-operators | 13:29 | |
| *** ducttape_ has quit IRC | 13:40 | |
| *** VW has joined #openstack-operators | 13:45 | |
| *** VW has quit IRC | 13:46 | |
| mnaser | yankcrime / haukebruno -- we have tweaked scheduler_host_subset_size | 13:52 |
| mnaser | because when we would get a big surge of VM creation, all schedulers would take the same decision | 13:52 |
| mnaser | we set our scheduler_host_subset_size to 3 | 13:53 |
| mnaser | haukebruno / yankcrime the problem in this issue is that when we do a mass restart of all our nova-compute nodes (at least for a small 50 node deployment) | 13:53 |
| mnaser | the scheduler is stuck spinning trying to keep up with the queue of updates | 13:54 |
| mnaser | also, do you always hit the same node in your galera cluster? we have it configured so it hits only 1 (active/passive) to avoid deadlock issues with galera | 13:54 |
| *** chlong has quit IRC | 14:00 | |
| *** sudipto has quit IRC | 14:01 | |
| *** jsheeren has joined #openstack-operators | 14:03 | |
| *** ckonstanski has joined #openstack-operators | 14:04 | |
| *** ducttape_ has joined #openstack-operators | 14:17 | |
| *** ducttape_ has quit IRC | 14:21 | |
| *** jamesdenton has joined #openstack-operators | 14:28 | |
| *** dansmith is now known as superdan | 14:38 | |
| *** vijaykc4 has joined #openstack-operators | 14:39 | |
| *** VW has joined #openstack-operators | 14:40 | |
| *** marst has joined #openstack-operators | 14:41 | |
| *** vijaykc4 has quit IRC | 14:42 | |
| *** vijaykc4 has joined #openstack-operators | 14:42 | |
| *** vijaykc4 has quit IRC | 14:45 | |
| *** ducttape_ has joined #openstack-operators | 14:47 | |
| *** liverpooler has joined #openstack-operators | 14:47 | |
| *** vijaykc4 has joined #openstack-operators | 14:55 | |
| *** vijaykc4 has quit IRC | 14:56 | |
| *** ecrehar has quit IRC | 14:56 | |
| *** paramite_ has quit IRC | 14:57 | |
| *** paramite has quit IRC | 14:58 | |
| yankcrime | mnaser: yeah, we have one designated as the master (via haproxy) - although the cluster is configured as multi-writer mode | 15:02 |
| mnaser | same here, we only write to 1 master | 15:03 |
| yankcrime | this was built before jay pipes had solved the deadlock issues, but we've left it as it's worked ok | 15:03 |
| mnaser | oh deadlock is not a thing anymore? didn't know | 15:03 |
| *** rpr has quit IRC | 15:08 | |
| *** ckonstanski has quit IRC | 15:09 | |
| *** rpr has joined #openstack-operators | 15:09 | |
| *** paramite_ has joined #openstack-operators | 15:12 | |
| *** derekh has quit IRC | 15:15 | |
| *** paramite has joined #openstack-operators | 15:16 | |
| *** chlong has joined #openstack-operators | 15:21 | |
| *** armax has joined #openstack-operators | 15:25 | |
| *** vijaykc4 has joined #openstack-operators | 15:31 | |
| *** jsheeren has quit IRC | 15:31 | |
| *** rmart04 has quit IRC | 15:32 | |
| zioproto | so, is it really the case that VMs with LVM local storage cannot be migrated ? :) | 15:38 |
| zioproto | it looks like it ! | 15:38 |
| zioproto | https://review.openstack.org/#/c/73387/ | 15:38 |
| *** vijaykc4 has quit IRC | 15:51 | |
| *** armax has quit IRC | 15:53 | |
| *** Guest31304 has quit IRC | 16:02 | |
| *** zioproto has quit IRC | 16:04 | |
| *** VW has quit IRC | 16:05 | |
| *** VW has joined #openstack-operators | 16:05 | |
| *** ducttape_ has quit IRC | 16:06 | |
| *** saneax is now known as saneax-_-|AFK | 16:07 | |
| *** VW has quit IRC | 16:10 | |
| *** ducttape_ has joined #openstack-operators | 16:13 | |
| *** ckonstanski has joined #openstack-operators | 16:15 | |
| *** udesale has quit IRC | 16:16 | |
| *** VW has joined #openstack-operators | 16:19 | |
| *** vinsh has quit IRC | 16:22 | |
| *** rcernin has quit IRC | 16:23 | |
| *** VW has quit IRC | 16:24 | |
| *** ecrehar has joined #openstack-operators | 16:24 | |
| *** pcaruana has quit IRC | 16:25 | |
| *** chlong has quit IRC | 16:29 | |
| *** Oku_OS is now known as Oku_OS-away | 16:32 | |
| *** slaweq_ has joined #openstack-operators | 16:33 | |
| *** karad has joined #openstack-operators | 16:36 | |
| *** kstev has joined #openstack-operators | 16:44 | |
| *** kelv has joined #openstack-operators | 16:45 | |
| *** bdeschenes has quit IRC | 16:46 | |
| *** matrohon has quit IRC | 16:51 | |
| *** chlong has joined #openstack-operators | 16:53 | |
| *** VW has joined #openstack-operators | 17:00 | |
| *** VW has quit IRC | 17:03 | |
| *** VW has joined #openstack-operators | 17:03 | |
| *** rcernin has joined #openstack-operators | 17:04 | |
| *** armax has joined #openstack-operators | 17:30 | |
| *** vinsh has joined #openstack-operators | 17:50 | |
| *** bdeschenes has joined #openstack-operators | 17:51 | |
| jproulx | zioproto: I no longer use LVM backend so not 100% sure of current state, I think the '--block-migration' flag in nova live-migration [--block-migrate] <server> [<host>] is meant to handle that (I honestly forget if we were using LVM or File before we went to RBD but we did have migrations working if slowly) | 17:54 |
| *** bdeschenes has quit IRC | 17:55 | |
| *** bdeschenes has joined #openstack-operators | 17:58 | |
| *** bdeschenes has quit IRC | 18:03 | |
| *** slaweq_ has quit IRC | 18:28 | |
| *** VW has quit IRC | 18:29 | |
| *** VW has joined #openstack-operators | 18:30 | |
| *** slaweq_ has joined #openstack-operators | 18:30 | |
| *** ducttape_ has quit IRC | 18:34 | |
| *** VW has quit IRC | 18:34 | |
| *** karad has quit IRC | 18:36 | |
| *** bdeschenes has joined #openstack-operators | 18:41 | |
| *** fragatina has quit IRC | 18:47 | |
| *** VW has joined #openstack-operators | 18:57 | |
| *** VW has quit IRC | 19:01 | |
| *** VW has joined #openstack-operators | 19:06 | |
| *** slaweq_ has quit IRC | 19:28 | |
| *** slaweq_ has joined #openstack-operators | 19:30 | |
| *** electrofelix has quit IRC | 19:37 | |
| *** ducttape_ has joined #openstack-operators | 19:42 | |
| *** ducttape_ has quit IRC | 19:47 | |
| *** ducttape_ has joined #openstack-operators | 19:47 | |
| *** jamesdenton has quit IRC | 19:52 | |
| *** jamesdenton has joined #openstack-operators | 19:53 | |
| *** fragatina has joined #openstack-operators | 19:55 | |
| *** jamesdenton has quit IRC | 19:58 | |
| *** ckonstanski has quit IRC | 19:58 | |
| *** ckonstanski has joined #openstack-operators | 19:59 | |
| *** karad has joined #openstack-operators | 20:20 | |
| *** paramite has quit IRC | 20:23 | |
| *** paramite_ has quit IRC | 20:24 | |
| *** ducttape_ has quit IRC | 20:41 | |
| *** VW has quit IRC | 20:46 | |
| *** VW has joined #openstack-operators | 20:47 | |
| *** slaweq_ has quit IRC | 20:47 | |
| *** chlong has quit IRC | 20:48 | |
| *** slaweq_ has joined #openstack-operators | 20:48 | |
| *** vijaykc4 has joined #openstack-operators | 20:51 | |
| *** VW has quit IRC | 20:51 | |
| *** fragatina has quit IRC | 20:53 | |
| *** VW has joined #openstack-operators | 21:00 | |
| *** liverpooler has quit IRC | 21:07 | |
| *** karad has quit IRC | 21:08 | |
| *** catintheroof has quit IRC | 21:10 | |
| *** catintheroof has joined #openstack-operators | 21:11 | |
| *** catintheroof has quit IRC | 21:15 | |
| *** ducttape_ has joined #openstack-operators | 21:20 | |
| *** bdeschenes has quit IRC | 21:31 | |
| *** ckonstanski has quit IRC | 21:50 | |
| *** vinsh has quit IRC | 21:55 | |
| *** VW has quit IRC | 21:57 | |
| *** VW has joined #openstack-operators | 21:58 | |
| *** slaweq_ has quit IRC | 22:01 | |
| *** VW has quit IRC | 22:02 | |
| *** slaweq_ has joined #openstack-operators | 22:03 | |
| *** VW has joined #openstack-operators | 22:03 | |
| *** jproulx has quit IRC | 22:04 | |
| *** ecrehar has quit IRC | 22:05 | |
| *** slaweq_ has quit IRC | 22:09 | |
| *** jamesdenton has joined #openstack-operators | 22:34 | |
| *** jamesden_ has joined #openstack-operators | 22:37 | |
| *** jamesdenton has quit IRC | 22:38 | |
| *** vijaykc4 has quit IRC | 22:41 | |
| *** fragatin_ has joined #openstack-operators | 22:51 | |
| *** fragatin_ has quit IRC | 22:53 | |
| *** fragatina has joined #openstack-operators | 22:56 | |
| *** fragatina has quit IRC | 23:00 | |
| *** VW_ has joined #openstack-operators | 23:02 | |
| *** VW has quit IRC | 23:05 | |
| *** VW_ has quit IRC | 23:07 | |
| *** saneax-_-|AFK is now known as saneax | 23:12 | |
| *** marst has quit IRC | 23:21 | |
| *** VW has joined #openstack-operators | 23:36 | |
| *** ducttape_ has quit IRC | 23:40 | |
| *** dalees has quit IRC | 23:40 | |
| *** VW has quit IRC | 23:41 | |
| *** armax has quit IRC | 23:42 | |
| *** dalees has joined #openstack-operators | 23:43 | |
| *** dalees has quit IRC | 23:48 | |
| *** dalees has joined #openstack-operators | 23:50 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!