| *** rebase has quit IRC | 00:31 | |
| *** rebase has joined #openstack-operators | 00:32 | |
| *** catintheroof has joined #openstack-operators | 00:33 | |
| *** mriedem has joined #openstack-operators | 01:03 | |
| *** trungnv has joined #openstack-operators | 01:05 | |
| *** aojea has joined #openstack-operators | 01:08 | |
| *** aojea has quit IRC | 01:13 | |
| *** r-daneel has joined #openstack-operators | 01:18 | |
| *** r-daneel has quit IRC | 02:06 | |
| *** catintheroof has quit IRC | 02:22 | |
| *** r-daneel has joined #openstack-operators | 02:23 | |
| *** rebase has quit IRC | 02:31 | |
| *** rebase has joined #openstack-operators | 02:32 | |
| *** mriedem has quit IRC | 02:43 | |
| *** aojea has joined #openstack-operators | 03:09 | |
| *** aojea has quit IRC | 03:14 | |
| *** vijaykc4 has joined #openstack-operators | 03:21 | |
| *** vijaykc4 has quit IRC | 03:24 | |
| *** gkadam has joined #openstack-operators | 03:30 | |
| *** fragatina has quit IRC | 03:38 | |
| *** fragatina has joined #openstack-operators | 03:39 | |
| *** Caterpillar has joined #openstack-operators | 03:44 | |
| *** simon-AS5591 has joined #openstack-operators | 04:09 | |
| *** simon-AS559 has quit IRC | 04:10 | |
| *** rebase has quit IRC | 04:31 | |
| *** fragatina has quit IRC | 04:31 | |
| *** fragatina has joined #openstack-operators | 04:32 | |
| *** rebase has joined #openstack-operators | 04:32 | |
| *** arnewiebalck_ has joined #openstack-operators | 04:38 | |
| *** Dinesh_Bhor has joined #openstack-operators | 04:56 | |
| *** wasmum has quit IRC | 05:01 | |
| *** wasmum has joined #openstack-operators | 05:02 | |
| *** aojea has joined #openstack-operators | 05:10 | |
| *** jproulx has quit IRC | 05:12 | |
| *** jproulx has joined #openstack-operators | 05:13 | |
| *** aojea has quit IRC | 05:15 | |
| *** arnewiebalck_ has quit IRC | 05:35 | |
| *** aojea has joined #openstack-operators | 05:44 | |
| *** Oku_OS-away is now known as Oku_OS | 06:07 | |
| *** jbadiapa has joined #openstack-operators | 06:08 | |
| *** aojea has quit IRC | 06:10 | |
| *** yprokule has joined #openstack-operators | 06:19 | |
| *** rcernin has joined #openstack-operators | 06:23 | |
| *** makowals has joined #openstack-operators | 06:23 | |
| *** simon-AS5591 has quit IRC | 07:09 | |
| *** wasmum has quit IRC | 07:28 | |
| *** aojea has joined #openstack-operators | 07:28 | |
| *** wasmum has joined #openstack-operators | 07:28 | |
| *** rmart04 has joined #openstack-operators | 07:29 | |
| *** rmart04 has quit IRC | 07:34 | |
| *** belmoreira has joined #openstack-operators | 07:39 | |
| *** rmart04 has joined #openstack-operators | 07:43 | |
| zioproto | good morning | 07:57 |
|---|---|---|
| *** wasmum has quit IRC | 08:13 | |
| *** wasmum has joined #openstack-operators | 08:14 | |
| *** derekh has joined #openstack-operators | 08:23 | |
| *** rebase has quit IRC | 08:31 | |
| *** rebase has joined #openstack-operators | 08:32 | |
| *** dbecker has joined #openstack-operators | 08:40 | |
| yankcrime | hey zioproto | 08:48 |
| zioproto | yankcrime: this morning I am looking into some Neutron warnings that seem cosmetic, because everything works. And we are trying to understand if we can safely ignore them. The OS version is Newton. Anyone interested if I paste this log lines ? | 08:52 |
| yankcrime | yeah go for it | 08:53 |
| zioproto | first file is /var/log/neutron/neutron-openvswitch-agent.log | 08:54 |
| zioproto | Message=2017-07-21 01:39:31.375 5119 WARNING neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-ba16c36c-aab9-4e7a-ab69-9c98cc16d36e - - - - -] Device 21228b06-28bf-46ba-a486-8cf5646cdaf7 not defined on plugin or binding failed ... | 08:54 |
| zioproto | we have this line popping up some time | 08:57 |
| zioproto | but we dont really have a problem | 08:57 |
| zioproto | everything seems to work | 08:57 |
| zioproto | yankcrime: do you have anything like this in your logs ? | 08:57 |
| yankcrime | doesn't look familiar zioproto | 08:57 |
| * yankcrime searches elasticsearch just in case | 08:58 | |
| yankcrime | nope | 08:58 |
| yankcrime | has this just started appearing after an upgrade? | 08:58 |
| zioproto | So after the upgrade the filenames of the log files changed | 09:18 |
| zioproto | all filenames have now 'neutron-' prefix | 09:18 |
| zioproto | so our Nagios was not picking up all the information, we noticed recently and we fixed it, that is why we see the warnings popping up in Nagios now. most likely this started with the upgrade, we did not have these warnings in Mitaka | 09:19 |
| zioproto | Another one is in the file /var/log/neutron/neutron-dhcp-agent.log | 09:19 |
| zioproto | 2017-07-21 09:37:07.657 28328 WARNING stevedore.named [req-df33ac9d-77ad-4b7f-a2b7-6c7a883eff95 - - - - -] Could not load neutron.agent.linux.interface.OVSInterfaceDriver | 09:19 |
| zioproto | These are warnings that appear on the compute nodes | 09:21 |
| zioproto | sorry, I misstyped, I meant on the network nodes | 09:24 |
| *** electrofelix has joined #openstack-operators | 09:24 | |
| *** rebase has quit IRC | 10:31 | |
| *** rebase has joined #openstack-operators | 10:32 | |
| *** electrofelix has quit IRC | 10:38 | |
| *** racedo has joined #openstack-operators | 10:38 | |
| *** electrofelix has joined #openstack-operators | 10:43 | |
| *** r-daneel has quit IRC | 10:55 | |
| *** Dinesh_Bhor has quit IRC | 11:07 | |
| *** belmoreira has quit IRC | 11:58 | |
| *** khappone has quit IRC | 12:19 | |
| *** khappone has joined #openstack-operators | 12:20 | |
| *** VW has quit IRC | 12:21 | |
| *** VW has joined #openstack-operators | 12:22 | |
| *** VW has quit IRC | 12:27 | |
| *** rebase has quit IRC | 12:31 | |
| *** rebase has joined #openstack-operators | 12:32 | |
| zioproto | anyone successfully started 500 instances with 1 openstack server create command line ? :) | 12:32 |
| arnewiebalck | I started a 500 node Magnum cluster once, so 1 command line, but not ‘openstack server create’ ;) | 12:37 |
| *** gkadam has quit IRC | 12:41 | |
| *** catintheroof has joined #openstack-operators | 12:42 | |
| *** chlong has joined #openstack-operators | 12:53 | |
| *** mriedem has joined #openstack-operators | 12:59 | |
| *** r-daneel has joined #openstack-operators | 13:09 | |
| *** ChiTo has joined #openstack-operators | 13:09 | |
| *** r-daneel has quit IRC | 13:13 | |
| *** VW has joined #openstack-operators | 13:15 | |
| *** VW has quit IRC | 13:16 | |
| *** VW has joined #openstack-operators | 13:16 | |
| *** VW has quit IRC | 13:16 | |
| *** VW has joined #openstack-operators | 13:17 | |
| *** jamesdenton has joined #openstack-operators | 13:18 | |
| *** jamesdenton has quit IRC | 13:25 | |
| *** jamesdenton has joined #openstack-operators | 13:25 | |
| *** zz9pzza has quit IRC | 13:29 | |
| *** arnewiebalck_ has joined #openstack-operators | 13:30 | |
| mriedem | does anyone here modify the wsgi_log_format option in nova for request logging nova-api stuff under eventlet.wsgi? | 13:34 |
| *** Dinesh_Bhor has joined #openstack-operators | 13:40 | |
| *** chyka has joined #openstack-operators | 13:51 | |
| yankcrime | zioproto seem to recall someone else from cern did a presentation at the paris summit which was about scaling bits of openstack to be able to boot 1k instances simultaneously | 13:53 |
| *** MVenesio has joined #openstack-operators | 13:53 | |
| yankcrime | https://www.openstack.org/summit/openstack-paris-summit-2014/session-videos/presentation/cold-start-booting-of-1000-vms-under-10-minutes | 13:54 |
| zioproto | yankcrime: thanks. I am stuck at 500. it makes dnsmasq stop responding correctly, it cant handle that many DHCP requests at once | 13:56 |
| yankcrime | i think a few other people moaned about that in boston, there was some talk about swapping dnsmasq out for something else i.e kea | 13:56 |
| *** chyka has quit IRC | 13:57 | |
| *** zz9pzza has joined #openstack-operators | 14:03 | |
| *** vijaykc4 has joined #openstack-operators | 14:04 | |
| zioproto | the video si quite old, CERN at that time was using nova-network. I guess they did not have dnsmasq problems. | 14:08 |
| zioproto | I am watchig at the video right now | 14:08 |
| zioproto | watching | 14:09 |
| zioproto | from the video it looks like their bottleneck was the glance images, that is not a problem for me with the ceph backend | 14:09 |
| * jproulx HATES dnsmasq, but has nothing productive to add :) | 14:10 | |
| *** emccormick has joined #openstack-operators | 14:14 | |
| *** vijaykc4 has quit IRC | 14:15 | |
| *** rebase has quit IRC | 14:31 | |
| *** rebase has joined #openstack-operators | 14:32 | |
| zioproto | jproulx: I can confirm that dnsmasq when you 200 or more instances booting, just takes a long time to give out IP addresses via DHCP | 14:35 |
| zioproto | it is really slow | 14:35 |
| zioproto | I have no idea how may I tune it to make it faster | 14:35 |
| zioproto | anyone knows if I can have multiple dnsmasq instances in the qdhcp namespace ? | 14:37 |
| zioproto | I mean, I know the DHCP protocol, you can have multiple servers, that is no problem. But I am not aware about details if the specific dnsmasq implementation | 14:38 |
| zioproto | basically all VMs boot correctly, but a bunch of them timeout after 5 minutes on DHCP | 14:38 |
| *** emccormick has quit IRC | 14:39 | |
| jproulx | I think you can run multiple neutron-dhcp-agents (i mean I know you can but I'm not 100% they are active/active) | 14:39 |
| *** marst has quit IRC | 14:39 | |
| *** marst has joined #openstack-operators | 14:47 | |
| *** chyka has joined #openstack-operators | 14:50 | |
| *** jamesdenton has quit IRC | 14:51 | |
| *** arnewiebalck_ has quit IRC | 14:51 | |
| *** makowals has quit IRC | 14:51 | |
| *** jamesdenton has joined #openstack-operators | 14:52 | |
| *** makowals has joined #openstack-operators | 14:53 | |
| *** dtrainor has quit IRC | 14:57 | |
| *** jamesdenton has quit IRC | 14:59 | |
| *** zz9pzza has quit IRC | 15:00 | |
| *** zz9pzza has joined #openstack-operators | 15:00 | |
| *** dtrainor has joined #openstack-operators | 15:02 | |
| *** jamesdenton has joined #openstack-operators | 15:02 | |
| *** MVenesio has quit IRC | 15:04 | |
| *** MVenesio has joined #openstack-operators | 15:04 | |
| *** rmart04 has quit IRC | 15:13 | |
| *** Oku_OS is now known as Oku_OS-away | 15:18 | |
| *** Apoorva has joined #openstack-operators | 15:21 | |
| *** chlong_ has joined #openstack-operators | 15:21 | |
| *** chlong has quit IRC | 15:24 | |
| *** emccormick has joined #openstack-operators | 15:27 | |
| *** rcernin has quit IRC | 15:34 | |
| *** wasmum has quit IRC | 15:36 | |
| *** wasmum has joined #openstack-operators | 15:39 | |
| *** Apoorva has quit IRC | 15:46 | |
| *** jamesden_ has joined #openstack-operators | 15:55 | |
| *** jamesdenton has quit IRC | 15:57 | |
| *** rcernin has joined #openstack-operators | 15:58 | |
| *** aojea has quit IRC | 16:03 | |
| aakso | it seems that dnsmasq still wants to maintain the lease file even when every address bound to a specific hw addr | 16:06 |
| aakso | has there been any work on serving the dhcp from OVS? I know OVN can do it but are there any plans to do it with current neutron-openvswitch-agent setup? | 16:08 |
| *** yprokule has quit IRC | 16:09 | |
| *** aojea has joined #openstack-operators | 16:12 | |
| *** chlong__ has joined #openstack-operators | 16:12 | |
| *** chlong_ has quit IRC | 16:13 | |
| *** aojea has quit IRC | 16:17 | |
| *** jamesden_ has quit IRC | 16:21 | |
| *** aojea has joined #openstack-operators | 16:21 | |
| *** jamesdenton has joined #openstack-operators | 16:23 | |
| aakso | zioproto: your first warning might be related to stale port entries in your db | 16:23 |
| *** aojea has quit IRC | 16:26 | |
| *** aojea has joined #openstack-operators | 16:31 | |
| *** rebase has quit IRC | 16:31 | |
| aakso | then again it could also be a interface present in the compute/network node that doesn't resolve into a port instance on neutron-server | 16:31 |
| *** rebase has joined #openstack-operators | 16:32 | |
| aakso | in that case you should see "Device <name> requested by agent <agentid> not found in database" in your neutron-server logs | 16:33 |
| aakso | provided you have debug logging enabled. | 16:34 |
| *** aojea has quit IRC | 16:36 | |
| *** craigs has joined #openstack-operators | 16:37 | |
| *** chlong_ has joined #openstack-operators | 16:40 | |
| *** aojea has joined #openstack-operators | 16:40 | |
| *** chlong__ has quit IRC | 16:41 | |
| *** Apoorva has joined #openstack-operators | 16:42 | |
| *** aojea has quit IRC | 16:44 | |
| *** derekh has quit IRC | 16:56 | |
| *** fragatina has quit IRC | 17:06 | |
| *** ircuser-1 has joined #openstack-operators | 17:08 | |
| *** chlong_ has quit IRC | 17:08 | |
| *** harlowja has joined #openstack-operators | 17:09 | |
| *** chlong_ has joined #openstack-operators | 17:10 | |
| *** electrofelix has quit IRC | 17:27 | |
| *** fragatin_ has joined #openstack-operators | 17:34 | |
| *** fragatin_ has quit IRC | 17:34 | |
| *** fragatin_ has joined #openstack-operators | 17:34 | |
| *** fragatina has joined #openstack-operators | 17:37 | |
| *** fragatina has quit IRC | 17:37 | |
| *** fragatin_ has quit IRC | 17:37 | |
| *** fragatina has joined #openstack-operators | 17:38 | |
| serverascode | zioproto you can and in fact should probably have more than one dhcp instance for HA | 17:38 |
| serverascode | you could also use config drive if you really needed it to work now | 17:38 |
| *** gkadam has joined #openstack-operators | 18:00 | |
| *** wasmum has quit IRC | 18:09 | |
| *** wasmum has joined #openstack-operators | 18:10 | |
| *** ChiTo has quit IRC | 18:11 | |
| *** rcernin has quit IRC | 18:16 | |
| *** emccormick has quit IRC | 18:21 | |
| *** emccormick has joined #openstack-operators | 18:25 | |
| *** rebase has quit IRC | 18:31 | |
| *** rebase has joined #openstack-operators | 18:32 | |
| *** gkadam has quit IRC | 18:38 | |
| *** wasmum has quit IRC | 18:41 | |
| *** wasmum has joined #openstack-operators | 18:43 | |
| *** jamesden_ has joined #openstack-operators | 18:49 | |
| *** jamesdenton has quit IRC | 18:49 | |
| *** racedo has quit IRC | 19:18 | |
| *** racedo has joined #openstack-operators | 19:18 | |
| *** racedo has quit IRC | 19:19 | |
| *** ChiTo has joined #openstack-operators | 19:24 | |
| *** Apoorva_ has joined #openstack-operators | 20:08 | |
| *** Apoorva_ has quit IRC | 20:09 | |
| *** aojea has joined #openstack-operators | 20:09 | |
| *** Apoorva_ has joined #openstack-operators | 20:09 | |
| *** Apoorva_ has quit IRC | 20:10 | |
| *** Apoorva_ has joined #openstack-operators | 20:10 | |
| *** Apoorva has quit IRC | 20:11 | |
| *** aojea has quit IRC | 20:13 | |
| *** aojea has joined #openstack-operators | 20:18 | |
| *** aojea has quit IRC | 20:22 | |
| *** rebase has quit IRC | 20:31 | |
| *** rebase has joined #openstack-operators | 20:36 | |
| *** rockyg has joined #openstack-operators | 20:41 | |
| *** emccormick has quit IRC | 20:42 | |
| *** aojea has joined #openstack-operators | 20:45 | |
| *** aojea has quit IRC | 20:50 | |
| *** aojea has joined #openstack-operators | 21:03 | |
| *** aojea has quit IRC | 21:09 | |
| *** MVenesio has quit IRC | 21:19 | |
| *** aojea has joined #openstack-operators | 21:21 | |
| *** emccormick has joined #openstack-operators | 21:24 | |
| *** aojea has quit IRC | 21:26 | |
| *** Apoorva_ has quit IRC | 21:34 | |
| *** Apoorva has joined #openstack-operators | 21:34 | |
| *** aojea has joined #openstack-operators | 21:40 | |
| *** aojea has quit IRC | 21:44 | |
| *** Apoorva has quit IRC | 21:47 | |
| *** Apoorva has joined #openstack-operators | 21:48 | |
| *** chyka has quit IRC | 21:52 | |
| *** chyka has joined #openstack-operators | 21:52 | |
| *** marst has quit IRC | 21:52 | |
| *** aojea has joined #openstack-operators | 22:07 | |
| *** aojea has quit IRC | 22:12 | |
| *** fragatin_ has joined #openstack-operators | 22:14 | |
| *** fragatina has quit IRC | 22:18 | |
| *** emccormick has quit IRC | 22:22 | |
| *** rockyg has quit IRC | 22:26 | |
| *** sliver has joined #openstack-operators | 22:27 | |
| *** VW_ has joined #openstack-operators | 22:27 | |
| *** VW has quit IRC | 22:31 | |
| *** rebase has quit IRC | 22:31 | |
| *** VW_ has quit IRC | 22:31 | |
| *** rebase has joined #openstack-operators | 22:32 | |
| *** fragatin_ has quit IRC | 22:42 | |
| *** fragatina has joined #openstack-operators | 22:43 | |
| *** Caterpillar has quit IRC | 23:05 | |
| *** craigs has quit IRC | 23:11 | |
| *** aojea has joined #openstack-operators | 23:20 | |
| *** catintheroof has quit IRC | 23:21 | |
| *** aojea has quit IRC | 23:24 | |
| *** chyka has quit IRC | 23:26 | |
| *** aojea has joined #openstack-operators | 23:29 | |
| *** jproulx1 has joined #openstack-operators | 23:30 | |
| *** ChiTo has quit IRC | 23:32 | |
| *** aojea has quit IRC | 23:33 | |
| *** chlong__ has joined #openstack-operators | 23:36 | |
| *** chlong_ has quit IRC | 23:37 | |
| *** aojea has joined #openstack-operators | 23:38 | |
| *** aojea has quit IRC | 23:42 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!