| *** sorin-mihai has quit IRC | 00:00 | |
| *** threestrands has joined #openstack-kolla | 00:00 | |
| *** JamesBenson has joined #openstack-kolla | 00:06 | |
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Add support for encrypting glance/heat api https://review.opendev.org/707131 | 00:12 |
|---|---|---|
| *** cdearborn has quit IRC | 00:23 | |
| *** ricolin has joined #openstack-kolla | 01:13 | |
| *** ricolin_ has joined #openstack-kolla | 01:18 | |
| *** ricolin has quit IRC | 01:21 | |
| openstackgerrit | Xinliang Liu proposed openstack/kolla-ansible stable/stein: Fix kolla_source_version value https://review.opendev.org/720117 | 02:14 |
| *** k_mouza has joined #openstack-kolla | 02:23 | |
| *** k_mouza has quit IRC | 02:27 | |
| openstackgerrit | Xinliang Liu proposed openstack/kolla-ansible stable/train: Fix kolla_source_version value https://review.opendev.org/720121 | 03:00 |
| openstackgerrit | Xinliang Liu proposed openstack/kolla-ansible stable/stein: Fix kolla_source_version value https://review.opendev.org/720117 | 03:01 |
| *** seco has joined #openstack-kolla | 03:26 | |
| *** hamzy_ has joined #openstack-kolla | 03:29 | |
| *** seco has quit IRC | 03:31 | |
| *** hamzy has quit IRC | 03:31 | |
| *** ykarel|away is now known as ykarel | 04:25 | |
| *** KeithMnemonic has quit IRC | 04:30 | |
| *** bline has quit IRC | 04:30 | |
| *** evrardjp has quit IRC | 04:37 | |
| *** evrardjp has joined #openstack-kolla | 04:37 | |
| *** skramaja has joined #openstack-kolla | 05:02 | |
| *** bline has joined #openstack-kolla | 05:02 | |
| *** cah_link has joined #openstack-kolla | 05:29 | |
| *** vishalmanchanda has joined #openstack-kolla | 05:40 | |
| osmanlicilegi | morning | 05:40 |
| sri_ | morning | 05:41 |
| *** threestrands has quit IRC | 05:54 | |
| andrein | Morning! | 05:56 |
| *** sorin-mihai has joined #openstack-kolla | 06:08 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Custom haproxy script for monitoring galera https://review.opendev.org/710213 | 06:15 |
| cosmicsound | Morning | 06:25 |
| mnasiadka | morning | 06:35 |
| *** sorin-mihai_ has joined #openstack-kolla | 06:40 | |
| cosmicsound | http://paste.openstack.org/show/792132/ | 06:43 |
| cosmicsound | ugly morning here | 06:43 |
| cosmicsound | just found out i cant seem to be able to run reconfigure or anything | 06:43 |
| *** sorin-mihai has quit IRC | 06:43 | |
| cosmicsound | any suggestion is welcomed | 06:44 |
| andrein | cosmicsound: looks like public key authentication is misconfigured on that server. can you log in via ssh manually? | 06:46 |
| cosmicsound | yes | 06:46 |
| cosmicsound | this is a one server only | 06:47 |
| andrein | what does your inventory look like? | 06:47 |
| andrein | Hi everyone, yesterday I've tried to add a few new compute servers that I've provisioned via bifrost. The issue is, my old servers have their networking configured using team devices, while the old servers are configured using bond devices (cloud-init only supports bonding). I thought, ok, I'll just set network_interface and neutron_external_interface using hostvars in the inventory, but to my surprise it | 06:48 |
| andrein | didn't work. | 06:48 |
| cosmicsound | http://paste.openstack.org/show/792133/ | 06:49 |
| andrein | cosmicsound: add ansible_connection=local | 06:49 |
| andrein | >> https://github.com/openstack/kolla-ansible/blob/master/ansible/inventory/all-in-one | 06:50 |
| cosmicsound | thi8ng is | 06:50 |
| cosmicsound | it worked util yesterday just great | 06:50 |
| cosmicsound | ... | 06:50 |
| cosmicsound | with this config | 06:51 |
| cosmicsound | weird now works | 06:52 |
| cosmicsound | anyhow i still do not realise why it does not work and it stopped suddenly | 06:52 |
| andrein | so, just to confirm, when logged in as root, ssh root@compute-1 should work without a password | 06:54 |
| andrein | right? | 06:54 |
| cosmicsound | yes | 06:54 |
| cosmicsound | i always runned it from root | 06:54 |
| cosmicsound | and worked without any issues | 06:55 |
| cosmicsound | until now | 06:55 |
| andrein | that's odd, i have no idea why that would happen | 06:55 |
| andrein | yoctozepto: ping | 06:55 |
| andrein | yoctozepto: if you have a few minutes, I'd like to talk to you about https://bugs.launchpad.net/kolla-ansible/+bug/1864803 | 06:55 |
| openstack | Launchpad bug 1864803 in kolla-ansible "Docs: clarify network config (globals.yml confusing text)" [Medium,Triaged] | 06:55 |
| *** abdysn has joined #openstack-kolla | 07:11 | |
| *** tonythomas has joined #openstack-kolla | 07:12 | |
| *** Abdallahyas has joined #openstack-kolla | 07:18 | |
| *** abdysn has quit IRC | 07:19 | |
| *** bengates has joined #openstack-kolla | 07:20 | |
| *** bengates has quit IRC | 07:20 | |
| *** bengates has joined #openstack-kolla | 07:20 | |
| hrw | morning | 07:27 |
| hrw | https://review.opendev.org/718714 needs some love and help | 07:27 |
| patchbot | patch 718714 - kolla-ansible - CI: do not build images on aarch64 - 7 patch sets | 07:27 |
| *** rpittau|afk is now known as rpittau | 07:28 | |
| openstackgerrit | zhangmeng proposed openstack/kolla-ansible master: Add support for encrypting cinder api. https://review.opendev.org/709663 | 07:29 |
| *** abdysn has joined #openstack-kolla | 07:32 | |
| andrein | mgoddard: ping! | 07:33 |
| *** Abdallahyas has quit IRC | 07:36 | |
| mnasiadka | andrein: it's 08:40 in UK, give him time to wake up :) | 07:40 |
| andrein | mnasiadka: :) I guess I'm just too high on caffeine :) | 07:41 |
| hrw | looks like ubunut/aarch64 builds again. ceph issue sorted | 07:44 |
| hrw | andrein: do some running to burn it. room->kitchen->room. repeat nth times | 07:45 |
| andrein | hrw: it's actually yesterday's caffeine, must've been up until 4 am or something. I resisted the urge to ping people then, though. they should feel lucky :) | 07:46 |
| *** born2bake has joined #openstack-kolla | 07:50 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Custom haproxy script for monitoring galera https://review.opendev.org/710213 | 07:51 |
| *** ykarel is now known as ykarel|lunch | 07:54 | |
| openstackgerrit | Xinliang Liu proposed openstack/kolla-ansible stable/train: Fix kolla_source_version value https://review.opendev.org/720121 | 07:58 |
| openstackgerrit | Xinliang Liu proposed openstack/kolla-ansible stable/stein: Fix kolla_source_version value https://review.opendev.org/720117 | 08:00 |
| openstackgerrit | zhangmeng proposed openstack/kolla-ansible master: Add support for encrypting cinder api. https://review.opendev.org/709663 | 08:08 |
| *** e0ne has joined #openstack-kolla | 08:12 | |
| mgoddard | andrein: o/ | 08:13 |
| andrein | mgoddard: morning! | 08:14 |
| andrein | mgoddard: any thoughts on handling this issue? https://bugs.launchpad.net/kolla-ansible/+bug/1864803 | 08:14 |
| openstack | Launchpad bug 1864803 in kolla-ansible "Docs: clarify network config (globals.yml confusing text)" [Medium,Triaged] | 08:14 |
| andrein | I'm willing to put in the work to get it fixed, but I need some guidance :) | 08:15 |
| mgoddard | andrein: simple answer: you have to either use globals.yml or host_vars/group_vars | 08:16 |
| openstackgerrit | zhangmeng proposed openstack/kolla-ansible master: Add support for encrypting cinder api. https://review.opendev.org/709663 | 08:16 |
| mgoddard | andrein: globals.yml always beats host_vars | 08:16 |
| *** skramaja has quit IRC | 08:17 | |
| *** skramaja has joined #openstack-kolla | 08:17 | |
| andrein | the issue is that we can't use group_vars in inventory because they're always overridden by the group_vars in the playbook | 08:17 |
| mgoddard | andrein: playbook group_vars beats inventory group_vars | 08:18 |
| mgoddard | yes | 08:18 |
| mgoddard | except, any group beats 'all' | 08:18 |
| mgoddard | so if your hosts were in another group, e.g. 'control/compute' | 08:18 |
| mgoddard | you could use inventory/group_vars/control | 08:18 |
| mgoddard | it's a bit of a nightmare, I've always thought inventory should beat playbook, but ansible disagrees | 08:19 |
| andrein | yeah, I feel the same way... was totally confused last night because of this | 08:19 |
| mgoddard | andrein: if you think the docs/globals.yml could be more clear about this (I'm sure they could), feel free to propose an update | 08:21 |
| andrein | what do you think about removing the network stuff from group_vars/all.yml and moving them to role defaults? | 08:22 |
| andrein | that would allow overriding the values from inventory group_vars/all.yml | 08:23 |
| andrein | since role defaults have very low precedence | 08:23 |
| yoctozepto | mgoddard, andrein: would probably make sense to have that configurable in ansible (the precedence) | 08:26 |
| yoctozepto | that said, globals.yml act like all.yml | 08:26 |
| yoctozepto | so it's not all lost | 08:26 |
| yoctozepto | and morning folks ;D | 08:26 |
| mgoddard | yoctozepto: precedence is configurable, but recommending that seems dangerous | 08:27 |
| yoctozepto | mgoddard: inventory over playbook is? | 08:27 |
| mgoddard | yoctozepto: globals.yml takes highest precedence, so it's not the same | 08:27 |
| yoctozepto | I must've slept that over | 08:27 |
| yoctozepto | mgoddard: yeah, I meant in this particular case | 08:27 |
| yoctozepto | when one wants to force their config on all the hosts | 08:27 |
| yoctozepto | well, it makes it non-overridable per group | 08:28 |
| mgoddard | yeah I looked into it because kayobe has a similar issue | 08:28 |
| mgoddard | or per-host | 08:28 |
| yoctozepto | deprecate ansible? | 08:28 |
| yoctozepto | yeah, indeed | 08:28 |
| *** jbadiapa has joined #openstack-kolla | 08:28 | |
| *** seco has joined #openstack-kolla | 08:28 | |
| mgoddard | andrein: role defaults would be annoying because we'd have to duplicate the default for every role where it's required | 08:29 |
| cosmicsound | 2020-04-15T08:20:49.273727Z qemu-system-x86_64: failed to initialize KVM: Permission denied _do_build_and_run_instance /var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py:2164 | 08:29 |
| *** ricolin_ is now known as ricolin | 08:30 | |
| andrein | mgoddard: https://docs.ansible.com/ansible/latest/reference_appendices/config.html#variable-precedence | 08:30 |
| andrein | this looks interesting | 08:30 |
| mgoddard | andrein: most environments should have a relevant non-all group that you could use to get a higher precedence | 08:30 |
| mgoddard | andrein: yeah, largely committed to memory :D | 08:30 |
| mgoddard | yeah, that was the config option I mentioned to yoctozepto. You could change that, but it's quite invasive and non-standard so I don't think we'd want to recommend it | 08:31 |
| *** eliaswimmer has joined #openstack-kolla | 08:31 | |
| mgoddard | inventory/group_vars/control is the way to go | 08:32 |
| andrein | mgoddard: In my case, the old hosts are in multiple groups, that's why I just wanted to override the all group | 08:32 |
| mgoddard | add a new group containing all the old groups | 08:32 |
| mgoddard | or a file per group | 08:33 |
| hrw | mgoddard, yoctozepto: https://review.opendev.org/718714 - any suggestions/reviews? improvements? | 08:34 |
| patchbot | patch 718714 - kolla-ansible - CI: do not build images on aarch64 - 7 patch sets | 08:34 |
| andrein | mgoddard: how do you feel about removing https://github.com/openstack/kolla-ansible/blob/master/ansible/group_vars/all.yml#L186-L187 | 08:34 |
| andrein | I think this would fix my immediate issue | 08:35 |
| mgoddard | andrein: I feel it would be wrong | 08:35 |
| andrein | and globals.yaml already has those defined | 08:35 |
| mgoddard | that's how we define our defaults | 08:35 |
| mgoddard | globals.yml is commented out by default | 08:35 |
| andrein | ah, right | 08:36 |
| *** ykarel|lunch is now known as ykarel | 08:37 | |
| mgoddard | andrein: do you understand how to make it work? | 08:39 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: CI: do not build images on aarch64 https://review.opendev.org/718714 | 08:40 |
| andrein | mgoddard: yep, I'm giving it a go now | 08:40 |
| mgoddard | cool | 08:40 |
| andrein | I'll take a stab at updating the docs also | 08:40 |
| yoctozepto | ah, I read the new comment in that bug report, sorry for misleading advice, mgoddard group suggestions are valid (TM) | 08:40 |
| mgoddard | andrein: great, thanks | 08:41 |
| yoctozepto | a stab or a few :-) | 08:41 |
| *** skramaja has quit IRC | 08:42 | |
| *** skramaja has joined #openstack-kolla | 08:42 | |
| *** iniazi has quit IRC | 08:42 | |
| *** iniazi has joined #openstack-kolla | 08:43 | |
| mnasiadka | yoctozepto: the OVN change is only waiting for you :) | 08:45 |
| yoctozepto | mnasiadka: I'm glad! :D a review is hereby promised for today :-) | 08:45 |
| cosmicsound | anyone anything on this: https://bugs.launchpad.net/kolla/+bug/1681461 | 08:49 |
| openstack | Launchpad bug 1681461 in kolla "Instance fails to boot: Could not access KVM kernel module: Permission denied" [High,Confirmed] - Assigned to Chason Chan (chen-xing) | 08:49 |
| cosmicsound | i seem to have also this issue | 08:50 |
| yoctozepto | oh what an oldie | 09:00 |
| *** k_mouza has joined #openstack-kolla | 09:01 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: gnocchi: install missing python3-wheel package https://review.opendev.org/720159 | 09:02 |
| hrw | with this patch we should have ubuntu/aarch64 working | 09:03 |
| hrw | in kolla | 09:05 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla master: Add mariadb_clustercheck image https://review.opendev.org/710217 | 09:05 |
| hrw | for k-a I do not plan to add !debian CI | 09:05 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Custom haproxy script for monitoring galera https://review.opendev.org/710213 | 09:08 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Remove support for CentOS 7 https://review.opendev.org/692450 | 09:08 |
| hrw | comments sorted | 09:08 |
| hrw | gave RP+1 to this patch | 09:11 |
| hrw | we need to get rid of c7 before ussuri release | 09:11 |
| *** cosmicsound9 has joined #openstack-kolla | 09:14 | |
| *** cosmicsound has quit IRC | 09:15 | |
| *** cosmicsound9 is now known as cosmicsound | 09:15 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Remove support for CentOS 7 https://review.opendev.org/692450 | 09:15 |
| hrw | mgoddard: zuul jobs read ansible/* vars file at all? | 09:26 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: CentOS 8: enable elasticsearch and kibana images https://review.opendev.org/720164 | 09:26 |
| mgoddard | hrw: nope | 09:26 |
| hrw | mgoddard: then your hint in https://review.opendev.org/718714 is weird | 09:27 |
| patchbot | patch 718714 - kolla-ansible - CI: do not build images on aarch64 - 8 patch sets | 09:27 |
| mgoddard | hrw: I'm suggesting that you set openstack_tag_suffix based on docker_image_tag_suffix | 09:27 |
| hrw | k | 09:28 |
| hrw | mgoddard: still have to set it twice anyway as we need it in two blocks | 09:28 |
| mgoddard | hrw: set_fact is persistent | 09:29 |
| hrw | mgoddard: ok, so will move it to start of file to make sure it gets global ;d | 09:29 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla-ansible master: CI: do not build images on aarch64 https://review.opendev.org/718714 | 09:31 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: Remove support for CentOS 7 https://review.opendev.org/692450 | 09:32 |
| hrw | I have 22 open patches now. feels far too many ;D | 09:33 |
| mgoddard | hrw: sure | 09:33 |
| mgoddard | yoctozepto: centos7 removal - let's rip this plaster off | 09:34 |
| hrw | mgoddard: checking your elk patch on aarch64 | 09:34 |
| hrw | have to get centos8 image for aarch64 into zuul | 09:34 |
| yoctozepto | mgoddard: yeah, let's remove c7 | 09:35 |
| hrw | 'max-servers: 0' looks funny | 09:38 |
| yoctozepto | hrw: 0 waste policy in its prime time | 09:40 |
| *** iniazi has quit IRC | 09:46 | |
| *** gfidente|afk is now known as gfidente | 09:49 | |
| openstackgerrit | Pierre Riteau proposed openstack/kayobe-config stable/queens: Synchronise configuration with stable/queens Kayobe https://review.opendev.org/720170 | 09:59 |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible master: Deprecate rabbitmq_hipe_compile https://review.opendev.org/720172 | 10:10 |
| *** sorin-mihai has joined #openstack-kolla | 10:17 | |
| *** sorin-mihai has quit IRC | 10:19 | |
| *** sorin-mihai_ has quit IRC | 10:19 | |
| *** sorin-mihai has joined #openstack-kolla | 10:19 | |
| *** rpittau is now known as rpittau|bbl | 10:23 | |
| *** sorin-mihai_ has joined #openstack-kolla | 10:37 | |
| *** scottsol has joined #openstack-kolla | 10:37 | |
| *** sorin-mihai has quit IRC | 10:41 | |
| *** skramaja has quit IRC | 10:42 | |
| *** dcapone2004 has quit IRC | 10:42 | |
| *** e0ne has quit IRC | 10:52 | |
| *** e0ne has joined #openstack-kolla | 10:53 | |
| openstackgerrit | Fabian Zimmermann proposed openstack/kolla-ansible master: Don't enforce metadata-api to run on dhcp-agents https://review.opendev.org/720186 | 11:11 |
| openstackgerrit | Scott Solkhon proposed openstack/kolla-ansible master: Support Neutron policy file in both .json and .yaml format https://review.opendev.org/702682 | 11:13 |
| cosmicsound | Traceback (most recent call last): File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/conductor/manager.py", line 651, in build_instances raise exception.MaxRetriesExceeded(reason=msg) nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance | 11:15 |
| cosmicsound | 213bc6bc-20ec-40e5-8844-bb2e53e398cd. | 11:15 |
| cosmicsound | anyone seen this before? | 11:15 |
| *** cdearborn has joined #openstack-kolla | 11:15 | |
| devfaz | cosmicsound: well, I would check the nova-conductor, nova-sheduler or placement-api logs to see why nova didnt find a suitable host. | 11:18 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla-ansible master: CI: do not build images on aarch64 https://review.opendev.org/718714 | 11:19 |
| kevinz | mgoddard: hrw: Thanks for the review of https://review.opendev.org/#/c/719702/. I am a bit curious about put the parameters into config override, do we have some demo for this? | 11:25 |
| patchbot | patch 719702 - kolla-ansible - Make nums_pcie_ports as a paratem - 1 patch set | 11:25 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: CI: do not build images on aarch64 https://review.opendev.org/718714 | 11:26 |
| cosmicsound | 2020-04-15 11:24:57.327 6 DEBUG oslo_concurrency.processutils [req-0cac0116-ba20-4674-8c9f-e0323d1fdcb8 - - - - -] CMD "ceph df --format=json --id nova --conf /etc/ceph/ceph.conf" returned: 0 in 0.214s execute /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_concurrency/processutils.py:409 | 11:26 |
| *** sylar has joined #openstack-kolla | 11:26 | |
| *** kplant has quit IRC | 11:27 | |
| mgoddard | kevinz: https://docs.openstack.org/kolla-ansible/latest/admin/advanced-configuration.html#openstack-service-configuration-in-kolla | 11:32 |
| hrw | thx mgoddard | 11:33 |
| mgoddard | hrw: when you say elasticsearch and kibana not available for !x86, do you mean just CentOS 8? | 11:33 |
| *** sylar is now known as kplant | 11:34 | |
| devfaz | cosmicsound: I think you should try your luck in #openstack - I think here are mostly kolla-dev-guys | 11:34 |
| kevinz | mgoddard: aha, thanks, use it quite often actually :-D | 11:34 |
| mgoddard | :) | 11:34 |
| hrw | mgoddard: yes | 11:35 |
| hrw | mgoddard: your patch enabled them for centos8 | 11:35 |
| mgoddard | hrw: ok. we have them in unbuildable images | 11:35 |
| hrw | mgoddard: your patch dropped them from c8 without adding c8-aarch64 c8-ppc64le | 11:36 |
| mgoddard | hrw: I'll remove the repo from base | 11:36 |
| hrw | mgoddard: thx | 11:36 |
| mgoddard | also rebased on C7 drop | 11:38 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: gnocchi: install missing python3-wheel package https://review.opendev.org/720159 | 11:46 |
| hrw | added note | 11:46 |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: CentOS 8: enable elasticsearch and kibana images https://review.opendev.org/720164 | 11:59 |
| hrw | +2 | 12:02 |
| mnasiadka | yoctozepto, mgoddard: Got somewhere with clustercheck served by xinetd, which checks Galera status - and haproxy checks this service - would appreciate some thoughts if this is something we want to implement (tripleo and OSA use the same approach) - https://review.opendev.org/#/q/topic:mariadb-clustercheck+(status:open+OR+status:merged) | 12:03 |
| cosmicsound | if centos 7 is dropped means 8 is fully on? | 12:10 |
| *** factor has joined #openstack-kolla | 12:16 | |
| hrw | cosmicsound: c8 is enabled for quite a while | 12:20 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla-ansible master: CI: add Debian source upgrade job on AArch64 https://review.opendev.org/713854 | 12:27 |
| hrw | rebased | 12:27 |
| *** cah_link has quit IRC | 12:30 | |
| *** bline has quit IRC | 12:30 | |
| openstackgerrit | Merged openstack/kolla stable/train: Add elasticsearch6 and kibana6 images https://review.opendev.org/718696 | 12:30 |
| cosmicsound | well it seems today unlucky day | 12:31 |
| cosmicsound | nova stopped working due to some kvm crap | 12:32 |
| cosmicsound | and now no machines will even start | 12:32 |
| cosmicsound | Traceback (most recent call last): File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/conductor/manager.py", line 651, in build_instances raise exception.MaxRetriesExceeded(reason=msg) nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance | 12:32 |
| cosmicsound | e453143e-62cb-4a1e-aef0-b02e242239d1. | 12:32 |
| cosmicsound | what are right permissions for /dev/kvm ? | 12:32 |
| cosmicsound | on compute host | 12:32 |
| *** Abdallahyas has joined #openstack-kolla | 12:43 | |
| *** cah_link has joined #openstack-kolla | 12:45 | |
| *** abdysn has quit IRC | 12:47 | |
| *** cah_link has quit IRC | 12:48 | |
| *** cah_link has joined #openstack-kolla | 12:49 | |
| *** rpittau|bbl is now known as rpittau | 12:49 | |
| *** jimcrowleyibm has joined #openstack-kolla | 12:54 | |
| *** KeithMnemonic has joined #openstack-kolla | 12:54 | |
| *** ykarel is now known as ykarel|afk | 12:55 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla master: CentOS 8: Switch last mentions of yum to dnf https://review.opendev.org/713266 | 13:17 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Support Neutron policy file in both .json and .yaml format https://review.opendev.org/702682 | 13:23 |
| openstackgerrit | Merged openstack/kayobe stable/rocky: Switch to stackhpc fork of resmo.ntp https://review.opendev.org/718546 | 13:29 |
| openstackgerrit | Merged openstack/kayobe-config stable/queens: Synchronise configuration with stable/queens Kayobe https://review.opendev.org/720170 | 13:29 |
| openstackgerrit | Merged openstack/kayobe-config master: Add pip_proxy variable to pip.yml https://review.opendev.org/709665 | 13:29 |
| *** abdysn has joined #openstack-kolla | 13:30 | |
| openstackgerrit | Scott Solkhon proposed openstack/kolla-ansible master: Support Neutron policy file in both .json and .yaml format https://review.opendev.org/702682 | 13:32 |
| *** Abdallahyas has quit IRC | 13:33 | |
| *** TrevorV has joined #openstack-kolla | 13:37 | |
| hrw | yoctozepto, mnasiadka: https://review.opendev.org/#/c/720159/ is trivial in a need of +w+2 ;D | 13:42 |
| patchbot | patch 720159 - kolla - gnocchi: install missing python3-wheel package - 2 patch sets | 13:42 |
| mnasiadka | again non-x86 quirks? approved :) | 13:43 |
| hrw | thx | 13:44 |
| hrw | mnasiadka: in parallel world pypi.org is done right... | 13:45 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: CI: do not build images on aarch64 https://review.opendev.org/718714 | 13:54 |
| kplant | hrw: in that same world, python ignores whitespace | 13:54 |
| hrw | kplant: you mean Pythonscript | 13:54 |
| mnasiadka | hrw: how do I migrate to this parallel world? ;) | 13:56 |
| hrw | mnasiadka: ;d | 13:56 |
| hrw | may be late for meeting | 14:02 |
| hrw | have two meetings at same time again | 14:02 |
| openstackgerrit | Merged openstack/kolla master: Remove support for CentOS 7 https://review.opendev.org/692450 | 14:02 |
| *** ykarel|afk is now known as ykarel | 14:03 | |
| mgoddard | ^ so long CentOS 7, nice to know you | 14:06 |
| kplant | is there going to be a stable/train-c8 branch? | 14:07 |
| *** jovial[m] has joined #openstack-kolla | 14:08 | |
| mgoddard | kplant: nope, it's all in stable/train | 14:09 |
| kplant | oof | 14:10 |
| kplant | so i guess i shouldn't do a git pull on my c7 machines | 14:10 |
| kplant | or is c7 support going to remain in stable/train ?' | 14:11 |
| cosmicsound | http://paste.openstack.org/show/792164/ | 14:15 |
| cosmicsound | now everyday something new | 14:15 |
| cosmicsound | it seems glance is rejecting images with --protected on | 14:17 |
| cosmicsound | is that normal behaviour? | 14:17 |
| kplant | nope, that's what i set on all of mine | 14:18 |
| *** bodgix has left #openstack-kolla | 14:19 | |
| mgoddard | kplant: train supports both 7 and 8 | 14:26 |
| kplant | nice! is c8 fully supported in stable/train? | 14:27 |
| openstackgerrit | Merged openstack/kolla master: Fix glance-api privsep errors https://review.opendev.org/715051 | 14:28 |
| kplant | that would give me a good excuse to update some hosts | 14:28 |
| mgoddard | kplant: it's almost there. Still testing the migration process, but a greenfield should mostly work | 14:28 |
| kplant | ah okay, i'll wait for the migration process to be ironed out. let me know if there's any testing i can do in my lab to help | 14:29 |
| *** weshay|ruck has joined #openstack-kolla | 14:47 | |
| mgoddard | mgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi | 14:52 |
| mgoddard | ^ meeting in 10 | 14:52 |
| hrw | right | 14:52 |
| * hrw -> bluejeans | 14:52 | |
| cosmicsound | yup | 14:52 |
| yoctozepto | hrw -> khakijeans | 14:55 |
| openstackgerrit | Mark Goddard proposed openstack/kolla stable/train: CentOS 8: Add rabbitmq-3.7.24 image (CentOS 7 only) https://review.opendev.org/715224 | 14:58 |
| *** priteau has joined #openstack-kolla | 15:00 | |
| mgoddard | #startmeeting kolla | 15:01 |
| openstack | Meeting started Wed Apr 15 15:01:01 2020 UTC and is due to finish in 60 minutes. The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
| *** openstack changes topic to " (Meeting topic: kolla)" | 15:01 | |
| openstack | The meeting name has been set to 'kolla' | 15:01 |
| mgoddard | #topic rollcall | 15:01 |
| *** openstack changes topic to "rollcall (Meeting topic: kolla)" | 15:01 | |
| mgoddard | \o | 15:01 |
| yoctozepto | o/ | 15:01 |
| *** sorin-mihai__ has joined #openstack-kolla | 15:01 | |
| scottsol | o/ | 15:01 |
| yoctozepto | real crowds | 15:01 |
| osmanlicilegi | o/ | 15:01 |
| yoctozepto | mnasiadka | 15:02 |
| mnasiadka | o/ | 15:02 |
| priteau | \o | 15:02 |
| mgoddard | #topic agenda | 15:03 |
| *** openstack changes topic to "agenda (Meeting topic: kolla)" | 15:03 | |
| mgoddard | * Roll-call | 15:04 |
| mgoddard | * Announcements | 15:04 |
| mgoddard | * Review action items from last meeting | 15:04 |
| mgoddard | * CI status | 15:04 |
| mgoddard | * Kolla Klub https://etherpad.openstack.org/p/kolla-sig | 15:04 |
| mgoddard | * Ussuri release planning (kayobe) | 15:04 |
| mgoddard | * Ussuri release planning (kolla & kolla ansible) | 15:04 |
| mgoddard | * Virtual PTG | 15:04 |
| *** sorin-mihai_ has quit IRC | 15:04 | |
| mgoddard | * Train release planning (centos 8) | 15:04 |
| mgoddard | #topic announcements | 15:04 |
| *** openstack changes topic to "announcements (Meeting topic: kolla)" | 15:04 | |
| mgoddard | #info OpenStack now in feature freeze | 15:04 |
| mgoddard | #info Kolla feature freeze: Apr 27 - May 01 | 15:05 |
| mgoddard | so ~2 weeks to get those features in | 15:05 |
| mgoddard | Any others | 15:05 |
| mgoddard | ? | 15:05 |
| osmanlicilegi | nope | 15:06 |
| mgoddard | #topic Review action items from last meeting | 15:06 |
| *** openstack changes topic to "Review action items from last meeting (Meeting topic: kolla)" | 15:06 | |
| mgoddard | mgoddard to create a google doc for kolla klub meeting agenda and notes | 15:06 |
| mgoddard | mgoddard to send reminder to ML about tomorrow's meeting | 15:06 |
| mgoddard | Both done | 15:06 |
| yoctozepto | oh boy he did | 15:06 |
| mgoddard | #topic CI status | 15:06 |
| *** openstack changes topic to "CI status (Meeting topic: kolla)" | 15:06 | |
| mgoddard | I think we're pretty green | 15:06 |
| mgoddard | still the issue with horizon slowness, but no longer blocking CI | 15:07 |
| mgoddard | priteau: how is kayobe CI now? | 15:07 |
| priteau | Green like my lawn! | 15:07 |
| yoctozepto | does etherpad work for you? | 15:07 |
| hrw | ~o\ | 15:08 |
| yoctozepto | https://etherpad.opendev.org/p/KollaWhiteBoard | 15:08 |
| mgoddard | priteau: I thought you had decking? :) | 15:08 |
| yoctozepto | gets me only 'loading' | 15:08 |
| mgoddard | great | 15:08 |
| priteau | yoctozepto: nope, doesn't work for me | 15:08 |
| yoctozepto | priteau: ack, eh | 15:08 |
| *** erolguzoglu has joined #openstack-kolla | 15:08 | |
| mgoddard | seems I have a cached copy | 15:08 |
| osmanlicilegi | down for me too | 15:08 |
| yoctozepto | mgoddard: yeah, I refreshed, regretting now ;p | 15:08 |
| mgoddard | can't updated | 15:08 |
| mgoddard | ok, we're blind | 15:09 |
| mgoddard | #topic Kolla Klub https://etherpad.openstack.org/p/kolla-sig | 15:09 |
| *** openstack changes topic to "Kolla Klub https://etherpad.openstack.org/p/kolla-sig (Meeting topic: kolla)" | 15:09 | |
| mgoddard | Thanks to those who joined the call, hopefully it was useful | 15:09 |
| osmanlicilegi | It was a good start | 15:10 |
| mgoddard | #link https://docs.google.com/document/d/1EwQs2GXF-EvJZamEx9vQAOSDB5tCjsDCJyHQN5_4_Sw | 15:10 |
| mgoddard | #action mgoddard to prepare project update for kolla klub | 15:10 |
| born2bake | Hi, I ve re-deployed kolla, any reason why creation of vm instances are failing due to 2020-04-15T15:07:11.496343Z qemu-system-x86_64: failed to initialize KVM: Permission denied ? | 15:10 |
| yoctozepto | born2bake: in a meeting now :-) | 15:10 |
| hrw | born2bake: be back in ~1h | 15:11 |
| yoctozepto | born2bake: and looks like this https://bugs.launchpad.net/kolla/+bug/1681461 | 15:11 |
| openstack | Launchpad bug 1681461 in kolla "Instance fails to boot: Could not access KVM kernel module: Permission denied" [High,Confirmed] - Assigned to Chason Chan (chen-xing) | 15:11 |
| yoctozepto | mgoddard: let's move on | 15:11 |
| mgoddard | yoctozepto: if only we could | 15:11 |
| mgoddard | Kolla team will discuss ways to gather and maintain operator information, reporting back next time. | 15:11 |
| yoctozepto | mgoddard: ah, sorry | 15:12 |
| mgoddard | so, what do we track, and how do we track it? | 15:12 |
| mgoddard | it was not a rhetorical question :) | 15:13 |
| yoctozepto | mgoddard: haha | 15:13 |
| yoctozepto | mgoddard: the basics for sure | 15:14 |
| yoctozepto | mgoddard: are we doing this in doc? | 15:14 |
| mgoddard | not now | 15:14 |
| mgoddard | in IRC now | 15:14 |
| mgoddard | well what are our options for tracking? | 15:14 |
| mgoddard | etherpad | 15:14 |
| mgoddard | google doc | 15:15 |
| mgoddard | wiki | 15:15 |
| *** brault has joined #openstack-kolla | 15:15 | |
| mgoddard | .rst | 15:15 |
| yoctozepto | mgoddard: something doodle-like? | 15:15 |
| mgoddard | doodle? | 15:15 |
| mgoddard | isn't that for scheduling? | 15:15 |
| yoctozepto | mgoddard: yeah, that's why I added '-like' | 15:16 |
| mgoddard | I don't see how it's related | 15:16 |
| yoctozepto | something more organized one can tick they use binary/source centos/ubuntu | 15:16 |
| yoctozepto | which services | 15:16 |
| osmanlicilegi | maybe we should extend it like a poll to get technical specs for the clouds they operate. | 15:16 |
| yoctozepto | is ceph deployed too | 15:16 |
| yoctozepto | yes, I meant a poll | 15:17 |
| mnasiadka | ceph begone | 15:17 |
| yoctozepto | doodle is a poll for scheduling | 15:17 |
| osmanlicilegi | I now imagine a tool like debian's popularity-contest | 15:17 |
| osmanlicilegi | https://packages.debian.org/search?keywords=popularity-contest | 15:17 |
| yoctozepto | oh yeah | 15:17 |
| yoctozepto | but we don't need it automatic for the moment | 15:17 |
| osmanlicilegi | it should be optional | 15:18 |
| yoctozepto | there are some folks probably willing to spend a few minutes to do the poll | 15:18 |
| yoctozepto | sure, very good idea in the long run | 15:18 |
| yoctozepto | #action osmanlicilegi to adapt popularity-contest for kolla needs | 15:18 |
| yoctozepto | :-) | 15:18 |
| mgoddard | I see. I was thinking it would be more freeform, but maybe not | 15:18 |
| osmanlicilegi | yoctozepto: got it :] | 15:19 |
| yoctozepto | it should include something freeform sure | 15:19 |
| yoctozepto | but basics should be easily countable | 15:19 |
| mgoddard | https://www.surveymonkey.com/ | 15:19 |
| mgoddard | that kind of thing? | 15:19 |
| yoctozepto | mgoddard: YES! :D | 15:20 |
| mnasiadka | I'll probably be unpopular, but google forms? | 15:20 |
| mgoddard | at what point are we just duplicating the openstack user survey? | 15:20 |
| mnasiadka | at every | 15:20 |
| mnasiadka | :) | 15:20 |
| yoctozepto | they don't ask kolla specifics? | 15:20 |
| priteau | Etherpad is back up | 15:21 |
| yoctozepto | priteau: yay! :D | 15:21 |
| brault | HI There, could someone take a look at the following failure: https://zuul.opendev.org/t/openstack/build/69948769a99546d98845ec7995f912f1/log/kolla/build/000_FAILED_cinder-volume.log in relation to the following patch: https://review.opendev.org/#/c/717697/ ? | 15:22 |
| patchbot | patch 717697 - kolla (stable/train) - add dpdk_telemetry and logparser for C8, use cento... - 5 patch sets | 15:22 |
| mgoddard | hi brault, we're currently in a meeting | 15:22 |
| brault | sorry; | 15:23 |
| mgoddard | does anyone want to take an action to look into this? | 15:23 |
| yoctozepto | mgoddard: you mean dpdk telemetry? | 15:23 |
| mgoddard | no, the poll/survey/database | 15:24 |
| *** jimcrowleyibm has quit IRC | 15:24 | |
| yoctozepto | I wished I could but can't promise | 15:24 |
| yoctozepto | really nobody better? | 15:24 |
| osmanlicilegi | I will take it | 15:25 |
| yoctozepto | thanks! :D | 15:25 |
| *** erolguzoglu has quit IRC | 15:25 | |
| mgoddard | thanks osmanlicilegi | 15:25 |
| osmanlicilegi | I'm not well with polls but will try to create something for a start | 15:25 |
| mnasiadka | Maybe it would be possible to reuse the openstack user survey framework? to not duplicate things? and store data in weird places? | 15:25 |
| mgoddard | #action osmanlicilegi to look into kolla klub survey/poll | 15:25 |
| *** jimcrowleyibm has joined #openstack-kolla | 15:25 | |
| osmanlicilegi | we can improve later | 15:25 |
| mgoddard | mnasiadka: that's true | 15:25 |
| mgoddard | I would suggest for this week osmanlicilegi just looks into available tools, including that used for openstack survey | 15:26 |
| osmanlicilegi | got it | 15:26 |
| yoctozepto | go get them, osmanlicilegi! | 15:26 |
| priteau | The only surveys I enjoyed taking were implemented with Typeform | 15:27 |
| mgoddard | #topic Ussuri release planning (kayobe) | 15:27 |
| *** openstack changes topic to "Ussuri release planning (kayobe) (Meeting topic: kolla)" | 15:27 | |
| priteau | And I've created a couple of Typeform surveys, it's easy enough to use | 15:27 |
| mgoddard | good info pierre | 15:27 |
| osmanlicilegi | noted typeform | 15:27 |
| osmanlicilegi | thanks priteau | 15:27 |
| mgoddard | ok, kayobe | 15:28 |
| mgoddard | jovial[m] | 15:28 |
| mgoddard | we have a couple of weeks before freeze | 15:28 |
| mgoddard | https://review.opendev.org/#/q/project:openstack/kayobe+status:open+branch:master | 15:29 |
| jovial[m] | anything we can squeeze in? | 15:29 |
| priteau | Have we decided a freeze date for kayobe? | 15:29 |
| *** ykarel is now known as ykarel|away | 15:29 | |
| *** jimcrowleyibm has quit IRC | 15:29 | |
| mgoddard | same as kolla | 15:29 |
| priteau | Ah, I thought we were trailing from kolla | 15:30 |
| mgoddard | mnasiadka: you going to update https://review.opendev.org/#/c/709689/ ? | 15:30 |
| patchbot | patch 709689 - kayobe - Add framework to deploy user-defined containers on... - 11 patch sets | 15:30 |
| mgoddard | we have previously trailed, but have been trying to catch up | 15:30 |
| mnasiadka | mgoddard: yes, will do - had other errands to run | 15:30 |
| openstackgerrit | Merged openstack/kolla master: nova-compute: add ndctl to expose NVDIMMs to guests https://review.opendev.org/718081 | 15:30 |
| mgoddard | no doubt we'll trail a little but hopefully not much | 15:30 |
| *** jimcrowleyibm has joined #openstack-kolla | 15:30 | |
| *** sorin-mihai_ has joined #openstack-kolla | 15:31 | |
| priteau | We can probably merge a good chunk of these. | 15:32 |
| priteau | Lots of small features / bug fixes | 15:32 |
| mgoddard | quite a few patches open, would be nice to get them moving | 15:32 |
| mgoddard | yes | 15:32 |
| *** bline has joined #openstack-kolla | 15:32 | |
| mgoddard | I will go through and add RP+1 to some to focus attention | 15:32 |
| mgoddard | anything else for kayobe? | 15:33 |
| mgoddard | #topic Ussuri release planning (kolla & kolla ansible) | 15:33 |
| *** openstack changes topic to "Ussuri release planning (kolla & kolla ansible) (Meeting topic: kolla)" | 15:33 | |
| *** sorin-mihai__ has quit IRC | 15:34 | |
| priteau | mgoddard: Nothing else from me | 15:34 |
| mgoddard | ok, crunch time | 15:34 |
| mgoddard | looking at https://etherpad.opendev.org/p/KollaWhiteBoard, L187 | 15:34 |
| hrw | ;D | 15:34 |
| mgoddard | mnasiadka: contributor & PTL docs need an update, although it's not affected by feature freezr | 15:35 |
| mnasiadka | mgoddard: on my list | 15:35 |
| mgoddard | cool | 15:35 |
| mnasiadka | mgoddard: what about docker healthchecks for kolla-ansible - are we going to push it this time, or not? | 15:36 |
| hrw | mnasiadka: I thought that we merged that... | 15:36 |
| mgoddard | mnasiadka: I'm up for merging it if it gets reviews | 15:36 |
| mnasiadka | hrw: Kolla part - sure | 15:37 |
| osmanlicilegi | not yet but we should | 15:37 |
| mnasiadka | mgoddard: https://review.opendev.org/#/c/676389/ - I'll rebase it and do RP+1 - let's see if it gets reviews. | 15:37 |
| patchbot | patch 676389 - kolla-ansible - Use Docker healthchecks for core services - 49 patch sets | 15:37 |
| mgoddard | ok | 15:37 |
| yoctozepto | what do they give us apart from being there? :-) I'm for merging because kolla part merged but really... | 15:37 |
| mnasiadka | ah, and incorporate hrw's comments | 15:37 |
| mnasiadka | yoctozepto: well, that's only meaningful for an operator to understand if something is wrong :) | 15:38 |
| yoctozepto | mnasiadka: it's wrong when it's not worky! | 15:38 |
| mnasiadka | yoctozepto: and healthcheck says "not worky!" | 15:38 |
| mgoddard | I guess if you were monitoring the container state | 15:38 |
| mgoddard | you could alert on it | 15:38 |
| mnasiadka | yeah, another idea is to do monitoring and alerting | 15:38 |
| *** sorin-mihai has joined #openstack-kolla | 15:38 | |
| mnasiadka | I would say let's merge it - it doesn't hurt anybody - maybe we'll find some good way to integrate it with monasca alerting or something similar | 15:39 |
| mnasiadka | I mean of course after reviews :) | 15:39 |
| mnasiadka | will rebase soon and ask for reviews | 15:39 |
| yoctozepto | mnasiadka: yeah, I'm in | 15:39 |
| yoctozepto | first ovn though, more important, today, I remember | 15:40 |
| hrw | ussuri release... I want to get aarch64 CI for centos8/debian/ubuntu. which now is outside of kolla. for k-a still 2 patches for CI so can have debian/source train->ussuri checks | 15:40 |
| mgoddard | just going down the list... | 15:40 |
| mgoddard | [kolla ansible] performance improvements | 15:40 |
| mgoddard | we merged something for fact caching | 15:40 |
| mgoddard | it's a start | 15:40 |
| hrw | and then just some tweaks here and there as usual | 15:40 |
| yoctozepto | yay :D | 15:40 |
| mgoddard | probably needs some docs on ansible tuning | 15:40 |
| yoctozepto | we so performant <3 | 15:40 |
| mgoddard | [kolla ansible] Nova cells v2 | 15:40 |
| mgoddard | I can't see us getting to this, I'll mark it pushed to V | 15:41 |
| yoctozepto | yeah, it's been a tough cycle :/ | 15:41 |
| mgoddard | [kolla ansible] TLS everywhere | 15:41 |
| yoctozepto | ALMOST! | 15:41 |
| yoctozepto | :D | 15:41 |
| mgoddard | we merged keystone | 15:41 |
| mgoddard | there are patches for heat, glance, cinder | 15:42 |
| yoctozepto | I guess it's doable for the core | 15:42 |
| mgoddard | generalfuzz: if you have patches for other services, would be great to see them | 15:42 |
| *** sorin-mihai_ has quit IRC | 15:42 | |
| mgoddard | [kolla ansible] More host-level commands (day 2 ops) | 15:43 |
| mgoddard | Add a new compute host to /etc/hosts | 15:43 |
| mgoddard | seems we no longer need this as much with the live migration fix | 15:43 |
| osmanlicilegi | agree | 15:44 |
| scottsol | restart would still be nice to have | 15:44 |
| mgoddard | scottsol: true | 15:44 |
| mgoddard | [kolla] infra images | 15:45 |
| mgoddard | pushing to V | 15:45 |
| mgoddard | [kolla-ansible] Capture Docker logs | 15:45 |
| yoctozepto | mgoddard: unless you run tls because then you need up-to-date name resolution | 15:45 |
| yoctozepto | that said, then you probably should rely on dns... | 15:45 |
| mgoddard | right | 15:45 |
| mgoddard | [kolla ansible] OVN support | 15:46 |
| mgoddard | has 1x +2 | 15:46 |
| yoctozepto | OVN doing great | 15:46 |
| yoctozepto | I think it will get another one, it was close to mine before | 15:46 |
| mgoddard | cool | 15:46 |
| mgoddard | oven ready | 15:46 |
| mgoddard | [kolla ansible] Ansible lint | 15:46 |
| mgoddard | seems we got quite close to this? | 15:47 |
| yoctozepto | it lies somewhere | 15:47 |
| *** sorin-mihai_ has joined #openstack-kolla | 15:47 | |
| *** sorin-mihai_ has quit IRC | 15:49 | |
| *** sorin-mihai_ has joined #openstack-kolla | 15:49 | |
| mgoddard | most of the 'other things' didn't get done | 15:49 |
| *** sorin-mihai has quit IRC | 15:49 | |
| mgoddard | yoctozepto: any thoughts on https://review.opendev.org/#/c/699445/ ? | 15:50 |
| patchbot | patch 699445 - kolla-ansible - Add Ansible group check to prechecks - 4 patch sets | 15:50 |
| openstackgerrit | Merged openstack/kolla master: gnocchi: install missing python3-wheel package https://review.opendev.org/720159 | 15:50 |
| mnasiadka | mgoddard: for ansible-lint I'll check the status of required changes to make it pass and ping you & yoctozepto tomorrow | 15:50 |
| *** dking has joined #openstack-kolla | 15:50 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix Designate not to use etcd coordination backend https://review.opendev.org/719583 | 15:50 |
| mgoddard | mnasiadka: cool | 15:50 |
| mgoddard | it's not critical - not really a feature :) | 15:50 |
| yoctozepto | mgoddard: was it missing only full coverage afair? | 15:51 |
| yoctozepto | that precheck I mean | 15:51 |
| mgoddard | yoctozepto: I picked some bad examples in my testing that it couldn't handle | 15:51 |
| yoctozepto | mgoddard: oh well, but it's not making anything worse, right? | 15:52 |
| mgoddard | no | 15:52 |
| yoctozepto | send mnasiadka at it, and I will re-review later too | 15:52 |
| mgoddard | ok | 15:53 |
| mgoddard | still a couple of loose ends for CentOS 8 too, will try to tidy up soon | 15:53 |
| mgoddard | #topic Virtual PTG | 15:53 |
| *** openstack changes topic to "Virtual PTG (Meeting topic: kolla)" | 15:53 | |
| hrw | several teams started mentioning vptg | 15:54 |
| mgoddard | June 1 - 5 | 15:54 |
| mgoddard | I haven't read the latest email about it | 15:54 |
| mgoddard | but it will be nice to do our PTG at the same time as everyone else again | 15:55 |
| hrw | yep | 15:55 |
| hrw | and I have proper mic now ;D | 15:55 |
| osmanlicilegi | we will miss donald duck :] | 15:56 |
| hrw | but also know how to switch webcam one from 'Donald Duck' to 'quite normal, sometimes in a bucket' | 15:56 |
| mgoddard | I think we'll need to decide how many sessions we want | 15:56 |
| hrw | https://www.amazon.com/Trust-GXT-212-Mico-Microphone/dp/B07LC43PXH works good | 15:57 |
| hrw | mgoddard: kolla, kolla-ansible, kayobe, klub? | 15:57 |
| mgoddard | something like that | 15:57 |
| mgoddard | klub is probably separate? | 15:57 |
| hrw | I can provide zoom without time limit with potential recording | 15:57 |
| mgoddard | of course klub members welcome to join in | 15:58 |
| mgoddard | that would be good | 15:58 |
| mgoddard | although I think both previous PTGs we tried zoom then switched to meet | 15:58 |
| yoctozepto | osmanlicilegi, hrw: I hardly survived last donald duck session, it was too funny ;D | 15:58 |
| osmanlicilegi | meet is simple as it seems I think | 15:59 |
| hrw | I use zoom 3 times per week with 3-23 people at same time. | 15:59 |
| yoctozepto | mgoddard: what was the issue? | 15:59 |
| yoctozepto | mgoddard: with zoom | 15:59 |
| *** abdysn has quit IRC | 15:59 | |
| mgoddard | #action mgoddard to read PTG emails and request slots | 15:59 |
| hrw | osmanlicilegi: I am out if gmeet is in use | 15:59 |
| mgoddard | can't remember | 15:59 |
| hrw | osmanlicilegi: it does not work on my desktop | 15:59 |
| yoctozepto | well, except with hrw, gmeet works best, but it is not usable by Chinese folks | 16:00 |
| mgoddard | quick final topic | 16:00 |
| mgoddard | #topic Train release planning (centos 8) | 16:00 |
| *** openstack changes topic to "Train release planning (centos 8) (Meeting topic: kolla)" | 16:00 | |
| mgoddard | Who added it? | 16:00 |
| hrw | I did | 16:00 |
| yoctozepto | oh why | 16:00 |
| hrw | as we had ussuri release planning in there I would like to see some train release as we have it finally working with c8 | 16:00 |
| yoctozepto | makes sense | 16:01 |
| *** cah_link has quit IRC | 16:01 | |
| hrw | so we can say 'pip install kolla==9.train.version' and you are free to go | 16:01 |
| hrw | instead of 'fetch head of stable/train' | 16:02 |
| mgoddard | ok | 16:02 |
| hrw | we have kolla stable/* in stable situation so let release stein/train | 16:02 |
| mgoddard | acceptance criteria at the bottom of https://etherpad.opendev.org/p/kolla-centos8 | 16:02 |
| mgoddard | needs https://review.opendev.org/717807 | 16:03 |
| patchbot | patch 717807 - kolla (stable/train) - Add docs and release note for CentOS 8 - 1 patch set | 16:03 |
| mgoddard | and need to switch from delorean to centos-release-openstack-train | 16:03 |
| yoctozepto | ^ this | 16:03 |
| yoctozepto | also, binary seems brokish | 16:03 |
| yoctozepto | maybe due to delorean | 16:03 |
| hrw | +2+W | 16:04 |
| mgoddard | yeah | 16:04 |
| mgoddard | anyone want to pick up delorean? | 16:04 |
| hrw | can do | 16:04 |
| mgoddard | thanks | 16:04 |
| mgoddard | #action hrw to switch kolla centos 8 train from delorean to centos-release-openstack-train | 16:05 |
| mgoddard | (if it exists) | 16:05 |
| mgoddard | (if not, badger RDO) | 16:05 |
| mgoddard | ok, we're over time | 16:05 |
| mgoddard | thanks y'all | 16:06 |
| mgoddard | #endmeeting | 16:06 |
| *** openstack changes topic to "Please feel welcome to join Kolla SIG https://etherpad.openstack.org/p/kolla-sig | Kolla IRC meetings on Wednesdays @ 15:00 UTC - see agenda @ https://goo.gl/OXB0DL | Whiteboard: https://etherpad.openstack.org/p/KollaWhiteBoard | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b" | 16:06 | |
| openstack | Meeting ended Wed Apr 15 16:06:12 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:06 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-04-15-15.01.html | 16:06 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-04-15-15.01.txt | 16:06 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-04-15-15.01.log.html | 16:06 |
| osmanlicilegi | thanks folks | 16:07 |
| kplant | hey mgoddard, i just tried to run prechecks on stable/train and got an error message that "centos release core version 7.7 is not supported. supported release are: 8" | 16:07 |
| yoctozepto | thanks mgoddard | 16:07 |
| mgoddard | kplant: ooh | 16:07 |
| kplant | i'm totally cleaning this machine out and trying again | 16:07 |
| kplant | but that's scary | 16:07 |
| mgoddard | kplant: although I think that means you're using master | 16:07 |
| kplant | nah i specifically did a `git checkout stable/train` | 16:07 |
| yoctozepto | weeird | 16:08 |
| kplant | but i'm trying it again to eliminate layer 8 as a problem | 16:08 |
| yoctozepto | it's always layer nine | 16:08 |
| mgoddard | kplant: did you pip install? | 16:08 |
| kplant | that's true for me... charter | 16:08 |
| kplant | mgoddard: nah, git | 16:08 |
| yoctozepto | magical weave | 16:08 |
| mgoddard | kplant: pip install | 16:08 |
| yoctozepto | pip install . | 16:09 |
| mgoddard | ^ | 16:09 |
| yoctozepto | ^ | 16:09 |
| hrw | mgoddard: asked on #rdo for ETA | 16:09 |
| mgoddard | thanks r3ap3r | 16:09 |
| mgoddard | sorry, hrw | 16:09 |
| kplant | oh you mean pip install ./kolla and pip install ./kolla-ansible | 16:09 |
| kplant | yeah i do that | 16:09 |
| kplant | i thought you meant from pypi | 16:09 |
| *** priteau has quit IRC | 16:12 | |
| kplant | i just finished cleaning this machine out and rebuilding, going to bootstrap,prechecks again | 16:13 |
| openstackgerrit | Merged openstack/kolla stable/train: Add docs and release note for CentOS 8 https://review.opendev.org/717807 | 16:17 |
| kplant | ugh, now i just get yelled at about python sdk versions | 16:20 |
| kplant | but i think that's beyond the OS check | 16:20 |
| openstackgerrit | Merged openstack/kayobe stable/train: CentOS 8: Support DNF https://review.opendev.org/714173 | 16:23 |
| openstackgerrit | Merged openstack/kayobe master: Update documentation for release tasks https://review.opendev.org/709154 | 16:23 |
| openstackgerrit | Merged openstack/kayobe master: Docs: Configure firewall to allow testing of baremetal https://review.opendev.org/707678 | 16:23 |
| r3ap3r | mgoddard: ;D | 16:26 |
| openstackgerrit | Merged openstack/kayobe master: CentOS 8: seed VM & bifrost https://review.opendev.org/712133 | 16:28 |
| *** tonythomas has quit IRC | 16:30 | |
| *** rpittau is now known as rpittau|afk | 16:33 | |
| *** evrardjp has quit IRC | 16:37 | |
| *** evrardjp has joined #openstack-kolla | 16:37 | |
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Add support for encrypting glance/heat api https://review.opendev.org/707131 | 16:46 |
| kplant | mgoddard, yoctozepto: i'm going to call that a false alarm | 16:47 |
| kplant | i think i did something out of order to get bits of master in my machine | 16:47 |
| *** generalfuzz is now known as headphoneJames | 16:49 | |
| * hrw out | 16:53 | |
| openstackgerrit | Xing Zhang proposed openstack/kolla-ansible master: Remove task for adding octavia to admin project https://review.opendev.org/720243 | 16:55 |
| *** k_mouza has quit IRC | 16:55 | |
| *** priteau has joined #openstack-kolla | 17:05 | |
| openstackgerrit | Xing Zhang proposed openstack/kolla-ansible master: Remove task for adding octavia to admin project https://review.opendev.org/720243 | 17:05 |
| *** ricolin has quit IRC | 17:10 | |
| born2bake | :/ redeployed kolla twice and still getting 2020-04-15T17:20:42.843861Z qemu-system-x86_64: failed to initialize KVM: Permission denied :/ | 17:21 |
| *** bengates_ has joined #openstack-kolla | 17:21 | |
| *** jonaspaulo has joined #openstack-kolla | 17:22 | |
| yoctozepto | born2bake: well, that's good sign; it means we can debug! | 17:24 |
| *** bengates has quit IRC | 17:24 | |
| yoctozepto | what distro is that, which flavour binary/source, which release | 17:25 |
| yoctozepto | how base os was installed (e.g. maas, cloud image, ISO installation - which variant) | 17:25 |
| born2bake | a6b6b1939295 kolla/ubuntu-source-nova-libvirt:train | 17:25 |
| yoctozepto | for this we would need a lot of details as so far it was futile to reproduce | 17:25 |
| born2bake | ubuntu 18.04, bare metal host, aio, 64gb ram, 16cpu | 17:26 |
| *** seco has quit IRC | 17:26 | |
| *** priteau has quit IRC | 17:26 | |
| openstackgerrit | Merged openstack/kayobe master: Prevent openrc files from using wrong OS_CACERT value https://review.opendev.org/717314 | 17:27 |
| brault | Any chance someone could take a look at the following failure: https://zuul.opendev.org/t/openstack/build/69948769a99546d98845ec7995f912f1/log/kolla/build/000_FAILED_cinder-volume.log in relation to the following patch: https://review.opendev.org/#/c/717697/ ? | 17:27 |
| patchbot | patch 717697 - kolla (stable/train) - add dpdk_telemetry and logparser for C8, use cento... - 5 patch sets | 17:27 |
| yoctozepto | brault: | 17:28 |
| yoctozepto | INFO:kolla.common.utils.cinder-volume:error: db5 error(-30986) from dbcursor->c_put: BDB0075 DB_PAGE_NOTFOUND: Requested page not found | 17:28 |
| yoctozepto | INFO:kolla.common.utils.cinder-volume:error: error(-30986) adding header #366 record | 17:28 |
| yoctozepto | INFO:kolla.common.utils.cinder-volume:error: python3-pbr-5.4.3-2.el8.noarch: install failed | 17:28 |
| yoctozepto | so it's something wrong with the package | 17:28 |
| born2bake | restarted libvirt container, 1 vm started, 2nd vm failed with kvm perms | 17:28 |
| yoctozepto | either upstream (rdo) or mirror | 17:28 |
| yoctozepto | or bug in dnf/rpm, who knows | 17:29 |
| yoctozepto | brault: tried rerunning? | 17:29 |
| yoctozepto | issue recheck | 17:29 |
| yoctozepto | born2bake: oh, that is interesting | 17:29 |
| yoctozepto | so after the restart | 17:29 |
| yoctozepto | it got to boot one instance | 17:29 |
| born2bake | yes | 17:29 |
| yoctozepto | and then it failed | 17:29 |
| born2bake | just one | 17:29 |
| born2bake | yup | 17:29 |
| yoctozepto | without any change inbetween | 17:29 |
| born2bake | nope | 17:29 |
| yoctozepto | well, it's an art to make software block itself :-) | 17:30 |
| r3ap3r | mgoddard: Good afternoon, I was told by a coworker it is common courtesy that if you come across a bug submitted by a Dev and the fix suggested actually does work that you should comment on the bug report that it did resolve the issue. I did so for this bug: https://bugs.launchpad.net/kolla-ansible/+bug/1864181 but if you do not want or need that, feel free to let me know and I won't do it again. :-) | 17:30 |
| openstack | Launchpad bug 1864181 in kolla-ansible ussuri "Monasca deployment fails due to invalid variable monasca_log_dir" [Medium,In progress] - Assigned to Mark Goddard (mgoddard) | 17:30 |
| cosmicsound | TASK [gnocchi : Running gnocchi bootstrap container] | 17:30 |
| cosmicsound | anyone has any idea why it could hang on this always? | 17:31 |
| born2bake | I thought maybe it might cause I built amphora image on the same host machine and ran sudo chmod 0644 /boot/vmlinuz* | 17:31 |
| brault | yoctozepto, i'll have to defer to someone else here. @mrunge may be? | 17:31 |
| born2bake | following that guide https://docs.openstack.org/octavia/latest/admin/amphora-image-build.html | 17:31 |
| yoctozepto | r3ap3r: I don't know about others, but I like it; thanks | 17:32 |
| yoctozepto | brault: oh, you can just leave a comment with lone 'recheck' to make it happen | 17:33 |
| yoctozepto | born2bake: that does not sound related at all | 17:33 |
| r3ap3r | yoctozepto: yes sir. | 17:34 |
| born2bake | should I try other release | 17:36 |
| brault | yoctozepto, thx, looks like mgoddard did that this PM; Thx mgoddard | 17:37 |
| *** gfidente is now known as gfidente|afk | 17:37 | |
| cosmicsound | http://paste.openstack.org/show/792177/ this is why the gnocchi is not bootstrappign | 17:55 |
| cosmicsound | was working earlier | 17:55 |
| cosmicsound | no idea what could had changed meanwhile | 17:55 |
| yoctozepto | born2bake: is it repeatable? in that if you restart libvirt again you can spawn yet another instance? | 17:56 |
| *** scottsol has quit IRC | 17:56 | |
| yoctozepto | mnasiadka, mgoddard: ovn is going in; did a fairly thorough review, found some minor issues, see my comment, thanks for this; mnasiadka for doing, mgoddard for his thorough review ;-) | 17:57 |
| cosmicsound | ok there is a bug with external ceph and default value of gnocchi | 17:58 |
| cosmicsound | or not like bug, yet the statement is not valid in newer versions since ceph cannot be enabled by default. | 17:59 |
| cosmicsound | #gnocchi_backend_storage: "{{ 'ceph' if enable_ceph|bool else 'file' }}" | 17:59 |
| born2bake | yoctozepto not sure yet, i am deploying centos source now | 18:04 |
| yoctozepto | cosmicsound: on master? | 18:20 |
| *** e0ne has quit IRC | 18:31 | |
| born2bake | yoctozepto in centos:source I got - 2020-04-15 18:43:46.871+0000: 111446: error : virCapabilitiesDomainDataLookupInternal:750 : invalid argument: could not find capabilities for domaintype=kvm | 18:44 |
| *** hongbin has joined #openstack-kolla | 19:01 | |
| *** bengates_ has quit IRC | 19:06 | |
| *** factor has quit IRC | 19:14 | |
| *** factor has joined #openstack-kolla | 19:14 | |
| andrein | mgoddard: are you still around? | 19:17 |
| mgoddard | thanks for verifying the fix r3ap3r | 19:25 |
| mgoddard | yoctozepto: yvw :) | 19:26 |
| mgoddard | andrein: I'm around now, you? | 19:26 |
| r3ap3r | mgoddard: np. :-) | 19:29 |
| andrein | mgoddard: I barely had time to work on this the whole day, just started. I've removed the network stuff from globals and tried to add everything as hostvars for now, and it passed prechecks | 19:31 |
| andrein | this being the vars thinghy we talked about this morning | 19:31 |
| mgoddard | andrein: great, hopefully it will work for you now | 19:31 |
| andrein | I had another idea for handling this in kolla-ansible, inspired by kubespray. they ship the group vars with the sample inventory https://github.com/kubernetes-sigs/kubespray/tree/master/inventory/sample/group_varshttps://github.com/kubernetes-sigs/kubespray/tree/master/inventory/sample/group_vars | 19:33 |
| andrein | they don't use playbook group_vars/host_vars at all | 19:34 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe stable/train: CentOS 8: seed VM & bifrost https://review.opendev.org/720278 | 19:34 |
| andrein | (as a kubespray user, i have to say that it feels a lot more natural than what kolla-ansible is currently doing) | 19:35 |
| andrein | what do you think about this option? | 19:35 |
| mgoddard | andrein: that's possible. The problem is that they need to get updated with each release, or even more frequently as we make changes. Keeping them with code is easier | 19:36 |
| mgoddard | andrein: we already get problems when people forget to update their inventory file, and that rarely changes | 19:36 |
| *** factor has quit IRC | 19:36 | |
| andrein | I see | 19:37 |
| mgoddard | andrein: also, if you make changes to those variables and so do we upstream, merging the two will be a nightmare | 19:37 |
| andrein | well, i kinda agree on that one, most kubespray upgrades went smooth, but I remember a few when inventory changed significantly :) | 19:40 |
| *** factor has joined #openstack-kolla | 19:43 | |
| *** osmanlicilegi has quit IRC | 19:45 | |
| andrein | mgoddard: http://paste.openstack.org/show/792184/ | 19:46 |
| andrein | looks like it's passing prechecks, but failing on deplloy | 19:46 |
| andrein | i think api_interface isn't set properly for that host? | 19:48 |
| andrein | I'm no ansible guru, but I think api_interface is first set to network_interface (eth0) then network_interface is set to the value from inventory host_vars (vlan2119) and api_interface is still set to eth0 :( | 19:51 |
| *** vishalmanchanda has quit IRC | 19:54 | |
| andrein | mgoddard: if the assumption above is true and I'm reading the variable precedence docs correctly, I can't just override network_interface from host_vars/group_vars, I have to override all the extra vars that are inferred from network_interface | 20:01 |
| andrein | this is starting to look very clunky :-/ | 20:01 |
| andrein | tried overriding from group_vars, same error | 20:03 |
| andrein | doesn't seem to work even if I remove the variables from the playbook group_vars. caching issue? | 20:08 |
| andrein | hmm, reverted everything, cleared cache, it looks like it's working | 20:13 |
| *** e0ne has joined #openstack-kolla | 20:24 | |
| *** jimcrowleyibm has quit IRC | 20:37 | |
| *** dciabrin has quit IRC | 20:42 | |
| *** eliaswimmer has quit IRC | 20:48 | |
| openstackgerrit | Merged openstack/kolla-ansible master: OVN Support https://review.opendev.org/696841 | 20:51 |
| *** k_mouza has joined #openstack-kolla | 20:56 | |
| *** k_mouza has quit IRC | 21:00 | |
| *** e0ne has quit IRC | 21:02 | |
| *** e0ne has joined #openstack-kolla | 21:03 | |
| *** dciabrin has joined #openstack-kolla | 21:07 | |
| *** brault is now known as brault-away | 21:24 | |
| openstackgerrit | Pedro Henrique Pereira Martins proposed openstack/kolla-ansible master: Add suport to OpenID Connect Authentication flow https://review.opendev.org/695432 | 21:38 |
| *** e0ne has quit IRC | 21:42 | |
| *** jonaspaulo has quit IRC | 21:56 | |
| *** TrevorV has quit IRC | 22:18 | |
| *** negronjl has quit IRC | 22:27 | |
| *** negronjl has joined #openstack-kolla | 22:31 | |
| *** born2bake has quit IRC | 22:41 | |
| *** seco has joined #openstack-kolla | 22:48 | |
| *** hongbin has quit IRC | 22:53 | |
| *** hongbin has joined #openstack-kolla | 23:01 | |
| *** negronjl has quit IRC | 23:10 | |
| *** negronjl has joined #openstack-kolla | 23:17 | |
| *** gfidente|afk has quit IRC | 23:25 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!