| *** e0ne has quit IRC | 00:02 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Remove nova-consoleauth task in U cycle https://review.opendev.org/735446 | 00:22 |
|---|---|---|
| *** dasp_ has joined #openstack-kolla | 00:28 | |
| *** dasp has quit IRC | 00:29 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Remove the congress roles since it has been retired https://review.opendev.org/729461 | 01:02 |
| *** wuchunyang has joined #openstack-kolla | 01:02 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Remove mongodb integration https://review.opendev.org/735030 | 01:10 |
| *** wuchunyang has quit IRC | 01:36 | |
| *** wuchunyang has joined #openstack-kolla | 01:37 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: fix deploy nova failed when use kolla_dev_mod https://review.opendev.org/735437 | 01:45 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Remove mongodb supported for panko backend https://review.opendev.org/735449 | 01:46 |
| *** wuchunyang has quit IRC | 02:35 | |
| *** bsanjeewa has quit IRC | 02:49 | |
| *** k_mouza has joined #openstack-kolla | 03:11 | |
| *** k_mouza has quit IRC | 03:16 | |
| *** wuchunyang has joined #openstack-kolla | 03:29 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Remove nova-consoleauth task in U cycle https://review.opendev.org/735446 | 03:38 |
| *** cah_link has joined #openstack-kolla | 03:52 | |
| *** ykarel|away is now known as ykarel | 03:55 | |
| *** skramaja has joined #openstack-kolla | 03:58 | |
| *** wuchunyang has quit IRC | 04:04 | |
| *** wuchunyang has joined #openstack-kolla | 04:08 | |
| *** cah_link has quit IRC | 04:09 | |
| *** wuchunyang has quit IRC | 04:14 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-kolla | 04:33 | |
| *** ricolin has joined #openstack-kolla | 04:42 | |
| *** wuchunyang has joined #openstack-kolla | 04:47 | |
| openstackgerrit | Merged openstack/kolla-ansible master: fix deploy nova failed when use kolla_dev_mod https://review.opendev.org/735437 | 05:04 |
| *** wuchunyang has quit IRC | 05:05 | |
| *** wuchunyang has joined #openstack-kolla | 05:19 | |
| *** wuchunyang has quit IRC | 05:38 | |
| *** cah_link has joined #openstack-kolla | 05:49 | |
| *** seco_ has joined #openstack-kolla | 05:53 | |
| sorin-mihai_ | morning | 06:01 |
| sorin-mihai_ | any idea what is causing this error? http://paste.openstack.org/raw/Zzl2vXVRB5Nhitk7ybW7/ | 06:01 |
| *** diurnalist has joined #openstack-kolla | 06:04 | |
| *** irclogbot_2 has quit IRC | 06:05 | |
| *** irclogbot_0 has joined #openstack-kolla | 06:06 | |
| *** diurnalist has quit IRC | 06:08 | |
| *** k_mouza has joined #openstack-kolla | 06:12 | |
| *** wuchunyang has joined #openstack-kolla | 06:14 | |
| *** k_mouza has quit IRC | 06:16 | |
| *** priteau has joined #openstack-kolla | 06:34 | |
| sorin-mihai_ | for quite a few days now, when trying to deploy new instances, i'm always getting 'No valid host was found', everything in the logs seems to be pointing to placement, like it would not allow over commitment and because of that it's ignoring the (only) available node. previously i had 98vCPU on 96threads, so i know it shouldn't be an issue to over commit, today i even removed one instance and now it's 96/96, but i'm still not | 06:54 |
| sorin-mihai_ | able to launch any new instance. any idea? | 06:54 |
| *** wuchunyang has quit IRC | 06:55 | |
| *** ykarel is now known as ykarel|afk | 06:56 | |
| priteau | sorin-mihai_: you could install the osc-placement plugin and check directly with placement the amount of allocated resources | 06:59 |
| *** ykarel|afk is now known as ykarel | 07:00 | |
| *** vishalmanchanda has joined #openstack-kolla | 07:03 | |
| sorin-mihai_ | priteau, i don't see anything relevant. what should i look for? | 07:13 |
| priteau | Check the overcommit settings for your host | 07:14 |
| priteau | Check if existing allocations match the usage you're expecting | 07:14 |
| *** bengates has joined #openstack-kolla | 07:16 | |
| *** jbadiapa has joined #openstack-kolla | 07:18 | |
| sorin-mihai_ | priteau, still doesn't make sense why previously i could launch instances and even get 98/96 while now i can't even if it's 96/96. it is supposed to be 16:1 cpu by default, right? | 07:18 |
| priteau | Yes that's the default setting for CPU allocation ratio | 07:19 |
| priteau | Have you checked that you have enough RAM and disk? | 07:19 |
| yoctozepto | admin0: image version so just 'stein' or 'train' there would work | 07:19 |
| *** jbalciunas has joined #openstack-kolla | 07:22 | |
| jbalciunas | hi, has anyone looked at this bug: https://bugs.launchpad.net/kolla-ansible/+bug/1882460 ? | 07:24 |
| openstack | Launchpad bug 1882460 in kolla-ansible "Prometheus task "Copying over prometheus config file" has been broken since 10.0.0.0rc1" [Undecided,New] | 07:24 |
| sorin-mihai_ | priteau, there seems to be enough ram in 'free -h': Mem: 754Gi Used: 189Gi Free: 170Gi Shared: 1.0Gi Buf/Cache 394Gi Available 563Gi, the storage is on cinder on top of thin LVM, the pool has about 12TB, meta is 0.25, while data is 6.49. plenty of storage, plenty of ram | 07:25 |
| *** amoralej|off is now known as amoralej | 07:30 | |
| *** bsanjeewa has joined #openstack-kolla | 07:31 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/ussuri: Remove nova-consoleauth task in U cycle https://review.opendev.org/735451 | 07:32 |
| *** wuchunyang has joined #openstack-kolla | 07:40 | |
| -openstackstatus- NOTICE: uWSGI made a new release that breaks devstack, please refrain from rechecking until a devstack fix is merged. | 07:43 | |
| *** dosaboy has joined #openstack-kolla | 07:43 | |
| *** dougsz has joined #openstack-kolla | 07:44 | |
| yoctozepto | jbalciunas: I'll take a look then | 07:45 |
| *** wuchunyang has quit IRC | 07:46 | |
| *** eliaswimmer has joined #openstack-kolla | 07:46 | |
| *** born2bake has joined #openstack-kolla | 07:50 | |
| eliaswimmer | morning | 07:53 |
| *** rpittau|afk is now known as rpittau | 08:00 | |
| *** ykarel is now known as ykarel|lunch | 08:04 | |
| jbalciunas | yoctozepto - thanks | 08:04 |
| *** sorin-mihai_ has quit IRC | 08:04 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/rocky: Allow cinder coordination backend to be configured https://review.opendev.org/735520 | 08:06 |
| *** sorin-mihai has joined #openstack-kolla | 08:07 | |
| *** sorin-mihai has quit IRC | 08:11 | |
| *** sorin-mihai has joined #openstack-kolla | 08:11 | |
| *** sorin-mihai has quit IRC | 08:13 | |
| *** sorin-mihai has joined #openstack-kolla | 08:13 | |
| *** sorin-mihai has quit IRC | 08:15 | |
| *** sorin-mihai has joined #openstack-kolla | 08:16 | |
| *** kevko has joined #openstack-kolla | 08:18 | |
| kevko | hi guys :), thanks for merge , i would like to ask for this patch https://review.opendev.org/#/c/733612/ and this https://review.opendev.org/#/c/733607/ | 08:20 |
| patchbot | patch 733612 - kolla-ansible - Fix STATIC_ROOT for Debian, Ubuntu - 2 patch sets | 08:20 |
| patchbot | patch 733607 - kolla - Fix horizon debian binary docker image - 2 patch sets | 08:20 |
| jbalciunas | have you guys noticed that with stable/ussuri the TASK [grafana : Enable grafana datasources] always fails on kolla-ansible reconfigure? | 08:22 |
| jbalciunas | TASK [grafana : Enable grafana datasources] ***************************************************************************************************************************************************************** | 08:23 |
| jbalciunas | 'telegraf', 'type': 'influxdb', 'url': 'http://10.81.0.10:8086', 'access': 'proxy', 'basicAuth': False}}}) | 08:23 |
| jbalciunas | 'timeField': '@timestamp'}}}}) => {"action": "uri", "ansible_loop_var": "item", "cache_control": "no-cache", "changed": false, "connection": "close", "content_length": "59", "content_type": "application/json", "date": "Mon, 15 Jun 2020 08:18:21 GMT", "elapsed": 0, "expires": "-1", "failed_when_result": true, "item": {"key": "elasticsearch", | 08:23 |
| jbalciunas | "value": {"data": {"access": "proxy", "database": "flog-*", "jsonData": {"esVersion": 5, "timeField": "@timestamp"}, "name": "elasticsearch", "type": "elasticsearch", "url": "http://10.81.0.10:9200"}, "enabled": true}}, "json": {"message": "Data source with the same name already exists"}, "msg": "HTTP Error 409: Conflict", "pragma": "no-cache", | 08:23 |
| jbalciunas | "redirected": false, "status": 409, "url": "http://10.81.0.10:3000/api/datasources", "x_frame_options": "deny"} | 08:23 |
| jbalciunas | the workaround is to delete the elasticsearch source via API before running kolla-ansible reconfigure | 08:23 |
| *** sorin-mihai_ has joined #openstack-kolla | 08:24 | |
| jbalciunas | curl -XDELETE http://10.81.0.10:3000/api/datasources/name/elasticsearch -u admin:[password_here] | 08:25 |
| jbalciunas | once you do this - reconfigure succeeds as expected | 08:25 |
| *** sorin-mihai has quit IRC | 08:26 | |
| yoctozepto | jbalciunas: no, please report :-) | 08:27 |
| jbalciunas | sure, ok | 08:27 |
| *** seco_ has quit IRC | 08:30 | |
| jbalciunas | ah, same issue has already been registered: https://bugs.launchpad.net/kolla-ansible/+bug/1881890 | 08:32 |
| openstack | Launchpad bug 1881890 in kolla-ansible "Task Enable grafana datasources fail after reconfigure" [Undecided,New] | 08:32 |
| yoctozepto | ah, so it is broken even in stein | 08:33 |
| yoctozepto | then just no core noticed :-) | 08:33 |
| jbalciunas | :) | 08:34 |
| hrw | morning | 08:35 |
| sorin-mihai_ | here are some logs from both placement and nova http://paste.openstack.org/raw/RAQvSD7zvuAGvEx9iHlE/ if they are too short, i can provide longer than 100lines each. i'm still clueless as to what is the cause, i can only see the effect... | 08:36 |
| *** seco_ has joined #openstack-kolla | 08:37 | |
| *** shyamb has joined #openstack-kolla | 08:37 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/ussuri: Remove nova-consoleauth task in U cycle https://review.opendev.org/735451 | 08:38 |
| *** bsanjeewa_ has joined #openstack-kolla | 08:44 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Fix issues with Prometheus config generation https://review.opendev.org/735529 | 08:45 |
| openstackgerrit | Merged openstack/kolla-ansible stable/ussuri: Fix group_vars and cinder template per documentation https://review.opendev.org/735474 | 08:47 |
| *** bsanjeewa has quit IRC | 08:48 | |
| yoctozepto | jbalciunas: I fixed that prometheus, please check the referenced patch ^^ | 08:48 |
| *** ykarel|lunch is now known as ykarel | 08:49 | |
| *** shyamb has quit IRC | 08:52 | |
| jbalciunas | yoctozepto - thanks, will put that to test :) | 08:55 |
| mgoddard | morning | 08:57 |
| *** gfidente has joined #openstack-kolla | 09:01 | |
| *** k_mouza has joined #openstack-kolla | 09:02 | |
| sorin-mihai_ | yoctozepto, not sure if it's related to that prometheus patch or not, but at least since January or so, whenever i tried to deploy vitrage it failed with "TASK [vitrage : Copying over prometheus_conf.yml file for service] [localhost]: FAILED! => {"changed": false, "msg": "Could not find or access '/etc/kolla/config/vitrage/prometheus_conf.yaml' on the Ansible Controller.\nIf you are using a module and expect the file to exist | 09:08 |
| sorin-mihai_ | on the remote, see the remote_src option"} | 09:08 |
| *** kemopq has joined #openstack-kolla | 09:08 | |
| *** wuchunyang has joined #openstack-kolla | 09:08 | |
| *** e0ne has joined #openstack-kolla | 09:09 | |
| *** priteau has quit IRC | 09:11 | |
| *** greuceanu has joined #openstack-kolla | 09:12 | |
| *** greuceanu has left #openstack-kolla | 09:12 | |
| *** Manheim has joined #openstack-kolla | 09:13 | |
| Manheim | hello | 09:13 |
| Manheim | anyone using grafana deployed with kolla? if yes, what collector are you using/in combination with what other service? | 09:13 |
| *** shyamb has joined #openstack-kolla | 09:17 | |
| *** priteau has joined #openstack-kolla | 09:21 | |
| yoctozepto | sorin-mihai_: no, it's something different there | 09:21 |
| yoctozepto | sorin-mihai_: please report if not reported yet | 09:22 |
| jbalciunas | Manheim: yes, prometheus | 09:26 |
| Manheim | I see, can it work with gnocchi only? I already have that deployed | 09:30 |
| jbalciunas | haven't tried that | 09:34 |
| Manheim | ok, thanks | 09:34 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Fix Grafana datasource update https://review.opendev.org/735540 | 09:36 |
| jbalciunas | anyone else tried to add more than 2 DHCP agents to a network on Ussuri? because upon adding third I'm hitting a duplicate binding_index error :-/ | 09:38 |
| jbalciunas | https://bugs.launchpad.net/neutron/+bug/1883513 | 09:39 |
| openstack | Launchpad bug 1883513 in neutron "Adding a third DHCP agent to the network fails due to binding index error" [Undecided,New] | 09:39 |
| jbalciunas | feels like I'm on a QA mission here :) | 09:39 |
| hrw | https://review.opendev.org/#/c/730331/ anyone? | 09:42 |
| patchbot | patch 730331 - kolla - CI: use wheels from infra mirror - 6 patch sets | 09:42 |
| sorin-mihai_ | yoctozepto, i think that vitrage issue is not reported, i never got that far in my deployment. i only noticed that the file path is required in ansible/roles/vitrage/tasks/precheck.yml and /ansible/roles/vitrage/tasks/config.yml, creating an empty file would make the error go away, but i'm almost sure that's not the real solution | 09:44 |
| sorin-mihai_ | tbh, i'm more concerned by the nova/placement issue that i can't figure out right now since the logs don't make any sense to me... | 09:45 |
| yoctozepto | sorin-mihai_: well, vitrage is not the most popular service here ;-) what nova/placement issue? | 09:48 |
| sorin-mihai_ | pasted some logs here: http://paste.openstack.org/raw/RAQvSD7zvuAGvEx9iHlE/ | 09:49 |
| yoctozepto | jbalciunas: oh, yes, you are | 09:49 |
| yoctozepto | jbalciunas: i'm betting neutron tested only two-agent setups | 09:50 |
| jbalciunas | yeah, looks like | 09:57 |
| jbalciunas | https://github.com/openstack/neutron/blob/stable/ussuri/neutron/scheduler/dhcp_agent_scheduler.py | 09:57 |
| jbalciunas | get_vacant_network_dhcp_agent_binding_index method | 09:58 |
| *** k_mouza has quit IRC | 09:59 | |
| *** rpittau is now known as rpittau|bbl | 10:03 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Fix kolla-ansible config when a network is named 'external' https://review.opendev.org/735551 | 10:07 |
| *** seco_ has quit IRC | 10:15 | |
| *** wuchunyang has quit IRC | 10:17 | |
| *** wuchunyang has joined #openstack-kolla | 10:18 | |
| *** k_mouza has joined #openstack-kolla | 10:19 | |
| *** wuchunyang has quit IRC | 10:23 | |
| *** seco_ has joined #openstack-kolla | 10:23 | |
| sorin-mihai_ | yoctozepto, maybe this is a bit more clear: http://paste.openstack.org/raw/1h9Iz4iob998SPpuIVH9/ | 10:49 |
| yoctozepto | sorin-mihai_: I see "Conflicting resource provider name: stk1-blr1.blockgemini.net already exists" | 10:53 |
| sorin-mihai_ | my logic fails, i don't understand why would it try to create it if it's already there | 10:55 |
| openstackgerrit | Michal Arbet proposed openstack/kolla-ansible master: Remove chrony package if containerized chrony is enabled https://review.opendev.org/734042 | 10:58 |
| openstackgerrit | Michal Arbet proposed openstack/kolla-ansible master: Remove chrony package if containerized chrony is enabled https://review.opendev.org/734042 | 11:00 |
| *** shyamb has quit IRC | 11:01 | |
| sorin-mihai_ | yoctozepto, the only thing that changed was to have all kolla images rebuilt and deployed, nothing actually changed compared to whatever was built a week earlier. tried with both local and master images, same thing happens | 11:02 |
| yoctozepto | sorin-mihai_: me neither, tried talking to nova/placements folks? | 11:02 |
| sorin-mihai_ | yoctozepto, not yet, i was hopping this is a just deployment issue, everything i've seen so far was because of my inability to deploy things properly. lol | 11:04 |
| openstackgerrit | Michal Arbet proposed openstack/kolla-ansible master: Remove chrony package if containerized chrony is enabled https://review.opendev.org/734042 | 11:07 |
| *** kevko has quit IRC | 11:13 | |
| jbalciunas | any ideas why neutron.conf values from /etc/kolla/config/neutron/neutron.conf are not merged and placed for neutron-server ? | 11:14 |
| *** bsanjeewa has joined #openstack-kolla | 11:15 | |
| mgoddard | jbalciunas: /etc/kolla/config/neutron.conf | 11:15 |
| jbalciunas | aha, so I should move it one level up, to /etc/kolla/neutron.conf instead of /etc/kolla/neutron/neutron.conf ? | 11:17 |
| jbalciunas | and the agent specific configs should stay in /etc/kolla/neutron? like ml2_conf.ini, dhcp_agent.ini | 11:18 |
| *** bsanjeewa_ has quit IRC | 11:18 | |
| *** shyamb has joined #openstack-kolla | 11:18 | |
| *** bsanjeewa_ has joined #openstack-kolla | 11:19 | |
| sorin-mihai_ | is this normal? it's one and the same node http://paste.openstack.org/raw/768gxdJCvXWGSa90X67E/ | 11:19 |
| *** bsanjeewa has quit IRC | 11:19 | |
| mgoddard | jbalciunas: you missed config/, but yes | 11:20 |
| jbalciunas | I did, thanks for clearing this up Mark | 11:21 |
| *** bsanjeewa has joined #openstack-kolla | 11:21 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Fix prometheus-elasticsearch-exporter flags https://review.opendev.org/735560 | 11:22 |
| mgoddard | jbalciunas: while you're here, I just hit your bug: https://bugs.launchpad.net/kolla-ansible/+bug/1880242 | 11:22 |
| openstack | Launchpad bug 1880242 in kolla-ansible victoria "prometheus elasticsearch exporter template error" [Medium,In progress] - Assigned to Mark Goddard (mgoddard) | 11:22 |
| mgoddard | which version are you running? | 11:22 |
| jbalciunas | just a sec... | 11:23 |
| jbalciunas | root@kolla-ussuri:/etc/kolla/config# pip3 show kolla-ansible | 11:23 |
| mgoddard | ussuri is enough :) | 11:23 |
| mgoddard | to be clear, what version were you running when you hit that bug? | 11:24 |
| *** bsanjeewa_ has quit IRC | 11:24 | |
| jbalciunas | from the very beginning with Ussuri | 11:25 |
| admin0 | i hit some bugs with Ussuri .. switched to train | 11:26 |
| jbalciunas | I mean early dev versions, not even 10.x | 11:26 |
| jbalciunas | admin0: I have a pre-prod on Ussuri where I'm debugging and fine tuning the deployment before transitioning to prod | 11:29 |
| *** shyam89 has joined #openstack-kolla | 11:30 | |
| *** ykarel is now known as ykarel|afk | 11:30 | |
| jbalciunas | the most nagging issues were OVN and multipathd. OVN deploys fine but has multiple issues, therefore had to revert to OVS, and multipathd can be easily resolved by modifying the jinja template directly - not a good approach with upgrades but is ok as a workaround | 11:31 |
| mgoddard | jbalciunas: did you raise bugs on the OVN issues? were they caused by kolla or OVN? | 11:32 |
| *** shyamb has quit IRC | 11:32 | |
| jbalciunas | mgoddard - don't know really, I didn't assume Kolla to be the culprit for the OVN as the deployment process is fine, there were subsequent issues with neutron-server and southdb | 11:34 |
| jbalciunas | as well as this one: https://bugs.launchpad.net/kolla-ansible/+bug/1875223 | 11:35 |
| openstack | Launchpad bug 1875223 in kolla-ansible "OVN relies on one host address (SPoF)" [High,Triaged] - Assigned to Michal Nasiadka (mnasiadka) | 11:35 |
| jbalciunas | therefore I though it would be safer to revert to OVS | 11:36 |
| *** lvdombrkr has joined #openstack-kolla | 11:36 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Safeguard kolla_source_version https://review.opendev.org/730604 | 11:37 |
| yoctozepto | ah, Michał did not finish the OVN HA | 11:39 |
| yoctozepto | eh | 11:39 |
| yoctozepto | sorin-mihai_: hmm, that is very interesting | 11:40 |
| yoctozepto | I am pretty sure there was another folk here with a similar issue | 11:40 |
| yoctozepto | also claiming nothing was done :D | 11:41 |
| *** Luzi has joined #openstack-kolla | 11:41 | |
| yoctozepto | could really use talking to nova/placement on that | 11:41 |
| yoctozepto | document everything | 11:41 |
| yoctozepto | report bug | 11:41 |
| yoctozepto | against kolla and others and we'll handle it there | 11:41 |
| sorin-mihai_ | all my problems started when my centos decided to ignore /etc/hosts, had to add related entries in public DNS. when i say 'nothing changed' that means i'm not actually doing anything beside running kolla-build and kolla-deploy once in a while. last week i wanted to enable some things that were disabled (monitoring related, i never got those to work) and then i saw i can't launch instances. logs i have a ton, debug is enabled | 11:45 |
| *** seco_ has quit IRC | 11:46 | |
| sorin-mihai_ | s/kolla-deploy/kolla-ansible\ deploy/ | 11:46 |
| openstackgerrit | Merged openstack/kayobe master: Fix seed VM interface ordering https://review.opendev.org/734040 | 11:46 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe stable/ussuri: Fix seed VM interface ordering https://review.opendev.org/735564 | 11:49 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe stable/train: Fix seed VM interface ordering https://review.opendev.org/735565 | 11:49 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe stable/stein: Fix seed VM interface ordering https://review.opendev.org/735566 | 11:51 |
| *** dswebb has joined #openstack-kolla | 11:53 | |
| openstackgerrit | Merged openstack/kayobe stable/ussuri: Update feature flags https://review.opendev.org/735365 | 12:00 |
| yoctozepto | sorin-mihai_: aye, I don't blame you, there are things changing from time to time w/o us noticing, known issue but too general to fix :D | 12:02 |
| *** jbalciunas has quit IRC | 12:02 | |
| *** amoralej is now known as amoralej|lunch | 12:03 | |
| *** rpittau|bbl is now known as rpittau | 12:04 | |
| *** jbalciunas has joined #openstack-kolla | 12:05 | |
| sorin-mihai_ | yoctozepto, yeah, i wouldn't mind if it was my fault. in fact, if i knew that, it would've been so easy to fix, i'd just time travel before making the changes. lol | 12:06 |
| *** dougsz has quit IRC | 12:13 | |
| openstackgerrit | Merged openstack/kolla stable/ussuri: Make Cyborg buildable (again) https://review.opendev.org/731285 | 12:15 |
| *** dougsz has joined #openstack-kolla | 12:28 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/ussuri: Remove nova-consoleauth task in U cycle https://review.opendev.org/735451 | 12:32 |
| shyam89 | mgoddard: Hi | 12:32 |
| shyam89 | Which containers people use in production? source or binary? | 12:33 |
| shyam89 | or they should use? | 12:33 |
| *** jbalciunas has quit IRC | 12:38 | |
| priteau | I use binary most of the time (just because it is the default) unless I need to run a version that isn't packaged (bug fix needed, RPMs not yet released, etc.), in which case I use source for specific services | 12:41 |
| shyam89 | okay, got it | 12:42 |
| shyam89 | source has virtual env with pip packages | 12:43 |
| shyam89 | and binary had rpm/debian packages | 12:43 |
| shyam89 | one of our client is using source containers in production env | 12:43 |
| shyam89 | not sure, if it's the good practice | 12:43 |
| *** wuchunyang has joined #openstack-kolla | 12:45 | |
| *** ykarel|afk is now known as ykarel | 12:46 | |
| *** dswebb has quit IRC | 12:52 | |
| *** shyam89 has quit IRC | 12:53 | |
| *** priteau has quit IRC | 12:54 | |
| *** priteau has joined #openstack-kolla | 12:55 | |
| priteau | I don't think it's bad practice :) | 12:56 |
| hrw | neither do I | 13:03 |
| *** jbalciunas has joined #openstack-kolla | 13:15 | |
| jbalciunas | can confirm that kolla-ansible reconfigure job completes succesfully without workarounds with https://review.opendev.org/#/c/735540/ on Ussuri | 13:17 |
| patchbot | patch 735540 - kolla-ansible - Fix Grafana datasource update - 1 patch set | 13:17 |
| *** amoralej|lunch is now known as amoralej | 13:20 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: CI: use wheels from infra mirror https://review.opendev.org/730331 | 13:23 |
| hrw | now it is working as expected ;D | 13:23 |
| *** wuchunyang has quit IRC | 13:33 | |
| *** wuchunya_ has joined #openstack-kolla | 13:36 | |
| jbalciunas | aplogies if I'm asking something obvious or widely discussed previously - what are the plans for moving from Ubuntu bionic to focal? | 13:38 |
| *** Manheim has quit IRC | 13:38 | |
| *** kemopq has quit IRC | 13:49 | |
| priteau | jbalciunas: this was discussed in the kolla PTG, see notes starting line 323: https://etherpad.opendev.org/p/kolla-victoria-ptg | 13:50 |
| *** Luzi has quit IRC | 13:54 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix Grafana datasource update https://review.opendev.org/735540 | 13:58 |
| *** bsanjeewa has quit IRC | 13:59 | |
| hrw | 0:05:21.497262 versus 0:23:49.306074 is nice improvement (openstack-base on CI) | 14:07 |
| *** happyhemant has joined #openstack-kolla | 14:08 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/ussuri: Fix Grafana datasource update https://review.opendev.org/735595 | 14:11 |
| *** priteau has quit IRC | 14:12 | |
| openstackgerrit | wu.chunyang proposed openstack/kolla-ansible master: add support of octavia dev mod https://review.opendev.org/735470 | 14:12 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/train: Fix Grafana datasource update https://review.opendev.org/735597 | 14:13 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible stable/stein: Fix Grafana datasource update https://review.opendev.org/735598 | 14:14 |
| lvdombrkr | hello guys. trying to inject passwoword with --admin-pass but without any successs. hw_qemu_guest_agent='True' in images enabled. in nova logs no errors. more information : http://paste.openstack.org/raw/794771/ | 14:14 |
| lvdombrkr | any ideaS? | 14:14 |
| wuchunya_ | I have the same problem | 14:23 |
| *** wuchunya_ is now known as wuchunyang | 14:24 | |
| openstackgerrit | jacky06 proposed openstack/kolla-ansible master: Remove mongodb supported for panko backend https://review.opendev.org/735449 | 14:25 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: CI: use wheels from infra mirror https://review.opendev.org/730331 | 14:27 |
| *** cah_link has quit IRC | 14:28 | |
| lvdombrkr | wuchunya_: same as me? | 14:32 |
| sorin-mihai_ | yoctozepto, i think my placement issue is caused by centos not reading /etc/hosts ... | 14:36 |
| jbalciunas | vdombrkr: have you set encryption keys for the setting/retrieving passwords in nova? | 14:40 |
| jbalciunas | other things that I can think of are cloud-init does not include password setting hooks | 14:42 |
| jbalciunas | and checking that metadata actually includes the admin_pass | 14:43 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: CI: use wheels from infra mirror https://review.opendev.org/730331 | 14:46 |
| *** diurnalist has joined #openstack-kolla | 14:50 | |
| *** wuchunyang has quit IRC | 14:52 | |
| lvdombrkr | jbalciunas: dont see pass in metadata if trying retrieve it from instance | 14:53 |
| lvdombrkr | curl http://169.254.169.254/openstack/latest/meta_data.json | 14:53 |
| lvdombrkr | jbalciunas: how i check is there encruption is set for the setting/retrieving passwords in nova? | 14:54 |
| lvdombrkr | i mean how i can check it | 14:54 |
| *** ykarel is now known as ykarel|away | 14:56 | |
| *** bengates_ has joined #openstack-kolla | 14:56 | |
| *** bengates has quit IRC | 14:56 | |
| *** wuchunyang has joined #openstack-kolla | 14:57 | |
| jbalciunas | if it's in the metadata, please check that cloud-init picks it up | 14:58 |
| jbalciunas | cloud.cfg in /etc/cloud | 14:59 |
| *** wuchunyang has quit IRC | 14:59 | |
| *** wuchunyang has joined #openstack-kolla | 15:00 | |
| *** wuchunyang has quit IRC | 15:01 | |
| *** wuchunyang has joined #openstack-kolla | 15:02 | |
| lvdombrkr | jbalciunas: this is my cloud-init config http://paste.openstack.org/raw/794774/ | 15:02 |
| jbalciunas | yeah, it's the default cloud.cfg from ubuntu public image | 15:03 |
| jbalciunas | so no, it will not set a password for you | 15:04 |
| lvdombrkr | jbalciunas: so what i need to change? i didnt found any documentation about this | 15:05 |
| lvdombrkr | just that i can pass password by | 15:05 |
| lvdombrkr | --admin-pass | 15:05 |
| lvdombrkr | and hw_qemu_guest_agent='True' should be enabled on image | 15:06 |
| *** bsanjeewa has joined #openstack-kolla | 15:10 | |
| sorin-mihai_ | this issue with centos not reading /etc/hosts is really bad for me, and #centos insists on saying that 'host' is not supposed to read /etc/hosts. how is everything working in kolla-ansible deploy if not reading /etc/hosts ? | 15:14 |
| mgoddard | sorin-mihai_: generally we don't rely on hostnames | 15:17 |
| mgoddard | sorin-mihai_: we use IP addresses everywhere | 15:17 |
| k_mouza | mgoddard: I can see this blueprint is approved and not started: https://blueprints.launchpad.net/kolla-ansible/+spec/docker-upgrade-reconfigure-command. I can start working on it if nobody has already made some progress and possibly not marked it? | 15:20 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Remove max count from Cinder online schema migration https://review.opendev.org/735617 | 15:23 |
| mgoddard | k_mouza: I don't believe anyone has started it yet. Feel free to pick it up | 15:23 |
| sorin-mihai_ | mgoddard, i'm just trying to solve this weird issue, not entirely sure this is is the cause... i have hostname, internal_fqdn and external_fqdn set in this node's /etc/hosts. they where all pointing to different internal private IPs, i have always tested with 'host hostname' and 'host fqdn', it always worked and got the same things as in /etc/hosts. public DNS had no entries for this, in fact there was a catch all set in public | 15:29 |
| sorin-mihai_ | DNS for fqdn. couple of weeks ago 'host' started say 'Host hostname not found: 3(NXDOMAIN)', although i know 100% that it did before. reading around the internet and even asking couple of times in #centos everything is leading me to the same answer, i'm supposed to keep these entries in public DNS, because reasons. done that, got burnt, placement keeps trying to create a new resource for the hostname.fqdn even if there already | 15:29 |
| sorin-mihai_ | exists. i can't recall what was the other issue i had caused by this same thing, but i recall i mendioned in here, i'll have to search the logs | 15:29 |
| openstackgerrit | jacky06 proposed openstack/kolla master: Remove mongodb dockerfile from kolla https://review.opendev.org/735447 | 15:32 |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Allow cinder coordination backend to be configured https://review.opendev.org/735520 | 15:33 |
| *** JamesBenson has joined #openstack-kolla | 15:40 | |
| *** e0ne has quit IRC | 15:51 | |
| *** e0ne_ has joined #openstack-kolla | 15:51 | |
| *** e0ne_ has quit IRC | 15:58 | |
| *** cah_link has joined #openstack-kolla | 16:03 | |
| *** sorin-mihai has joined #openstack-kolla | 16:13 | |
| *** rpittau is now known as rpittau|afk | 16:17 | |
| *** Manheim has joined #openstack-kolla | 16:24 | |
| openstackgerrit | caoyuan proposed openstack/kolla master: Remove mongodb dockerfile from kolla https://review.opendev.org/735447 | 16:26 |
| *** wuchunyang has quit IRC | 16:29 | |
| *** Manheim has quit IRC | 16:30 | |
| *** wuchunyang has joined #openstack-kolla | 16:30 | |
| *** cah_link has quit IRC | 16:32 | |
| *** e0ne has joined #openstack-kolla | 16:32 | |
| *** e0ne has quit IRC | 16:34 | |
| *** e0ne has joined #openstack-kolla | 16:34 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Remove max count from Cinder online schema migration https://review.opendev.org/735617 | 16:35 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Install Fluentd Prometheus plugin out of the box https://review.opendev.org/706042 | 16:35 |
| *** ricolin has quit IRC | 16:36 | |
| openstackgerrit | Merged openstack/kayobe master: Revert "Use OpenStack Ussuri release" https://review.opendev.org/735331 | 16:39 |
| openstackgerrit | Pierre Riteau proposed openstack/kolla-ansible master: Remove max count from Cinder online schema migration https://review.opendev.org/735617 | 16:41 |
| *** Manheim has joined #openstack-kolla | 16:43 | |
| *** wuchunyang has quit IRC | 16:48 | |
| sorin-mihai | mgoddard, found in the IRC logs the other issue, which I think it's also caused by /etc/hosts not being read properly, it was when mariadb would fail to deploy on my node, as the haproxy would keep trying/waiting on the public DNS catchall, because /etc/host was not read. that's when i added records in the public DNS for for hostname, internal_fqdn and external_fqdn pointing 3 private IPs, because i realised that the catchall is | 16:50 |
| sorin-mihai | killing it. now, the thing is, the catchall was there all along since January, and only /etc/hosts was used for the 3 records on private IPs | 16:50 |
| * sorin-mihai note to self, build a bloody local dns resolver, stop relying to cloudflare | 16:51 | |
| *** k_mouza has quit IRC | 16:53 | |
| *** lvdombrkr has quit IRC | 16:53 | |
| sorin-mihai | the question remains though, how was /etc/hosts read by the entire deployment properly before it started to fail, and why things keep failing even with the proper records in the public DNS... | 16:54 |
| *** e0ne has quit IRC | 16:54 | |
| *** skramaja has quit IRC | 16:55 | |
| mgoddard | hrw, mnasiadka, yoctozepto, osmanlicilegi, dougsz, priteau, jovial[m] & anyone else, please see for ussuri release blockers http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015456.html | 16:56 |
| dougsz | ACK | 16:56 |
| *** happyhemant has quit IRC | 16:58 | |
| *** seco_ has joined #openstack-kolla | 16:59 | |
| yoctozepto | mgoddard: ack, also - are we then unblocking cyborg in Train as well? | 17:01 |
| yoctozepto | mgoddard: I could remove OPAE altogether to make it simple | 17:02 |
| yoctozepto | mgoddard: or limit it to centos7 | 17:02 |
| *** dougsz has quit IRC | 17:02 | |
| mgoddard | yoctozepto: they didn't say they would backport to train on ML | 17:02 |
| yoctozepto | mgoddard: ok, less work, /me happy | 17:06 |
| yoctozepto | I don't think cyborg had much power before ussuri anyhow | 17:07 |
| *** bengates_ has quit IRC | 17:09 | |
| mgoddard | yoctozepto: you have a million DNM patches open : | 17:18 |
| mgoddard | :) | 17:18 |
| yoctozepto | mgoddard: man at work ;-) | 17:18 |
| mgoddard | yoctozepto: how can it be work if it is not to be merged? :p | 17:19 |
| yoctozepto | mgoddard: testing, testing, testing :-) | 17:19 |
| yoctozepto | mgoddard: could use reviewing the cool kvm patch to get rid of that high heat bug | 17:23 |
| yoctozepto | mgoddard: and now we can also test kvm :-) | 17:23 |
| *** gfidente is now known as gfidente|afk | 17:24 | |
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Generate Root CA for Self-Signed Certificates https://review.opendev.org/731344 | 17:26 |
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Generate Root CA for Self-Signed Certificates https://review.opendev.org/731344 | 17:30 |
| *** seco_ has quit IRC | 17:31 | |
| *** amoralej is now known as amoralej|off | 17:32 | |
| *** bengates has joined #openstack-kolla | 17:43 | |
| hrw | https://marcin.juszkiewicz.com.pl/2020/06/15/opendev-ci-speed-up-for-aarch64/ | 17:44 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/train: Escape table names in mariadb upgrade procedure https://review.opendev.org/735646 | 17:47 |
| *** bengates has quit IRC | 17:48 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/train: Escape table names in mariadb upgrade procedure https://review.opendev.org/735646 | 17:48 |
| yoctozepto | hrw: clap, clap, clap | 17:58 |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: Fix Grafana datasource update https://review.opendev.org/735597 | 18:06 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Fix Grafana datasource update https://review.opendev.org/735598 | 18:06 |
| sorin-mihai | since 'openstack resource provider list' listed 2 UUIDs, i said to myself 'why not rename them and see what happens'... at most i'd expect i can't do much with the instances allocated to those, but how bad can i break something already broken and not functional, right? so, i renamed both and moments after that a 3rd UUID got created. tried to launch new instance, check. previous instances still working just fine, check. now | 18:20 |
| sorin-mihai | let's see what happens at the next upgrade/deploy... | 18:20 |
| jbalciunas | those resources will remain | 18:22 |
| jbalciunas | I'd be very surprised if they didn't | 18:22 |
| sorin-mihai | yeah, i don't see why not. i wonder if there some way to move everything in a single resource provider. this is basically one and the same node, and has been all along | 18:25 |
| openstackgerrit | Merged openstack/kolla-ansible master: Remove chrony package if containerized chrony is enabled https://review.opendev.org/734042 | 18:48 |
| Manheim | is there a way to make grafana highly available? | 19:07 |
| jbalciunas | three instances behind haproxy, manually though | 19:08 |
| jbalciunas | a good idead for RFE, I guess | 19:08 |
| Manheim | oh, so It uses the OS DB | 19:13 |
| *** vishalmanchanda has quit IRC | 19:22 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/ussuri: Remove chrony package if containerized chrony is enabled https://review.opendev.org/735663 | 19:23 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/train: Remove chrony package if containerized chrony is enabled https://review.opendev.org/735664 | 19:24 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/stein: Remove chrony package if containerized chrony is enabled https://review.opendev.org/735665 | 19:24 |
| *** e0ne has joined #openstack-kolla | 19:24 | |
| *** e0ne has quit IRC | 19:25 | |
| *** e0ne has joined #openstack-kolla | 19:25 | |
| *** bsanjeewa_ has joined #openstack-kolla | 19:34 | |
| *** bsanjeewa has quit IRC | 19:38 | |
| *** eplatt has joined #openstack-kolla | 19:47 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/ussuri: Fix Grafana datasource update https://review.opendev.org/735595 | 19:48 |
| jbalciunas | well done lads, many thanks! | 19:52 |
| jbalciunas | I'm thinking if it would be possible to control whether Neutron is configured to use keystone_authtoken with memcached | 20:03 |
| jbalciunas | because currently it leaks memcached connections on a massive scal | 20:03 |
| jbalciunas | scale* | 20:04 |
| jbalciunas | in-process token caching in Neutron works fine but when used in conjunction with memcached - the connection count to memcached increases until memcached runs out of connections | 20:05 |
| *** e0ne has quit IRC | 20:06 | |
| jbalciunas | with the Kolla default of 5000 connections my test env runs out of connections overnight | 20:06 |
| jbalciunas | and then all the bad things happen in successsion | 20:07 |
| jbalciunas | so currently, as I want to use memcached for Keystone and Horizon, I need to manually run a job that comments out keystone_authtoken section entries that relate to memcached in neutron.conf in order to keep things stable | 20:09 |
| jbalciunas | otherwise, as said, in about 8-9 hours after deployment, bad things start to happen | 20:10 |
| jbalciunas | so it's not a thing that automated tests would pick up immediately | 20:10 |
| *** evrardjp has quit IRC | 20:11 | |
| jbalciunas | Grafana memcached connection graphs: https://ibb.co/p3TCJqC AND https://ibb.co/nmmvvH4 - just an example as to what happens | 20:11 |
| jbalciunas | the drops in the graphs indicate the restart of the neutron-server and the subsequent reduction of connections to memcached | 20:12 |
| *** evrardjp has joined #openstack-kolla | 20:13 | |
| *** eplatt has quit IRC | 20:24 | |
| openstackgerrit | Pierre Riteau proposed openstack/kolla master: Remove unused variable MAX_NUMBER https://review.opendev.org/735676 | 20:28 |
| *** e0ne has joined #openstack-kolla | 20:32 | |
| *** bsanjeewa has joined #openstack-kolla | 20:55 | |
| openstackgerrit | Merged openstack/kolla stable/ussuri: add libibverbs package for openvswitch https://review.opendev.org/735438 | 20:57 |
| *** bsanjeewa_ has quit IRC | 20:58 | |
| *** Manheim has quit IRC | 21:00 | |
| *** e0ne has quit IRC | 21:12 | |
| *** bsanjeewa has quit IRC | 21:17 | |
| *** sorin-mihai has quit IRC | 21:30 | |
| *** sorin-mihai has joined #openstack-kolla | 21:31 | |
| *** threestrands has joined #openstack-kolla | 21:36 | |
| *** gmann is now known as gmann_afk | 21:39 | |
| *** gfidente|afk has quit IRC | 21:47 | |
| *** Manheim has joined #openstack-kolla | 21:49 | |
| *** eplatt has joined #openstack-kolla | 22:15 | |
| eplatt | hi everyone, I had a question about VIF plugging that I hope someone can answer. I'm having an issue with where VM boot/rebuild time is taking a long time when the VM itself has multiple interfaces. It seems that there is a 5sec delay between each VIF unplug and 10sec delay between each VIF plug. so far I can only reproduce this behaviour on kolla | 22:18 |
| eplatt | deployed clouds - packstack with linuxbridge is able to VIF plug in parallel, as well as RHEL OSP with OVS. If anyone has some experience with this I'd really appreciate it! | 22:18 |
| *** also_stingrayza has quit IRC | 22:37 | |
| *** stingrayza has joined #openstack-kolla | 22:37 | |
| openstackgerrit | Merged openstack/kayobe stable/ussuri: Fix seed VM interface ordering https://review.opendev.org/735564 | 22:37 |
| *** JamesBenson has quit IRC | 23:10 | |
| *** Manheim has quit IRC | 23:10 | |
| *** JamesBenson has joined #openstack-kolla | 23:11 | |
| *** JamesBenson has quit IRC | 23:16 | |
| *** JamesBenson has joined #openstack-kolla | 23:46 | |
| *** born2bake has quit IRC | 23:48 | |
| *** JamesBenson has quit IRC | 23:51 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!