| cloudxtiny | Anyone know if we can use openvswitch in the ironic_compute container? | 00:37 |
|---|---|---|
| cloudxtiny | seems to be failing to start due to "'/dev/hugepages': No such file or directory" for me | 00:38 |
| *** macz_ has joined #openstack-ansible | 00:39 | |
| *** tosky has quit IRC | 00:41 | |
| *** macz_ has quit IRC | 00:44 | |
| cloudxtiny | sorted it :-) | 00:52 |
| *** spatel has joined #openstack-ansible | 00:54 | |
| ThiagoCMC | Can someone help me with this error: http://paste.openstack.org/show/799811/ <- I reinstalled the Controllers but letf the Compute Nodes, now, they're all trowing this error here but, I can't see anything wrong from Horizon's Admin UI... How to fix this? | 00:56 |
| ThiagoCMC | cloudxtiny, cool! :-P | 00:56 |
| *** spatel has quit IRC | 01:23 | |
| *** macz_ has joined #openstack-ansible | 01:30 | |
| *** macz_ has quit IRC | 01:34 | |
| *** fresta_ has joined #openstack-ansible | 01:52 | |
| *** fresta has quit IRC | 01:53 | |
| *** cloudxtiny has quit IRC | 02:46 | |
| *** macz_ has joined #openstack-ansible | 03:54 | |
| *** macz_ has quit IRC | 03:58 | |
| *** pto has joined #openstack-ansible | 04:06 | |
| *** pto has quit IRC | 04:10 | |
| *** fresta_ has quit IRC | 04:24 | |
| *** fresta has joined #openstack-ansible | 04:32 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-ansible | 05:33 | |
| *** miloa has joined #openstack-ansible | 06:36 | |
| *** viks____ has joined #openstack-ansible | 06:57 | |
| noonedeadpunk | o/ | 07:34 |
| *** pto has joined #openstack-ansible | 07:46 | |
| *** pto has quit IRC | 07:47 | |
| *** pto has joined #openstack-ansible | 07:48 | |
| *** andrewbonney has joined #openstack-ansible | 08:10 | |
| *** cshen has joined #openstack-ansible | 08:16 | |
| *** rpittau|afk is now known as rpittau | 08:30 | |
| pto | Is there more configuration to murano other that defining murano-infra_hosts: *infrastructure_hosts? | 08:47 |
| pto | Getting an error with os-horizon: HORIZON_CONFIG['legacy_static_settings'] = LEGACY_STATIC_SETTINGS NameError: name 'HORIZON_CONFIG' is not defined | 08:51 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_glance stable/stein: Do not symlink ceph libraries for distro path https://review.opendev.org/760818 | 08:51 |
| noonedeadpunk | pto: um, not sure how it's related with murano | 08:53 |
| noonedeadpunk | ah | 08:54 |
| noonedeadpunk | it's murano dashboard | 08:54 |
| pto | Yep | 08:54 |
| noonedeadpunk | I think it's plugin issue. it needs `from openstack_dashboard.settings import HORIZON_CONFIG` | 08:55 |
| noonedeadpunk | eventually I think murano is pretty much unmaintained for several years... | 08:55 |
| pto | Ok. I was just testing it, im gonna skip that part | 08:56 |
| noonedeadpunk | so adding this import somewhere here https://opendev.org/openstack/murano-dashboard/src/branch/master/muranodashboard/local/local_settings.d/_50_murano.py#L49 or at the beginning of the file | 08:57 |
| pto | In the osa config file? | 08:59 |
| noonedeadpunk | no in murano dashboard plugin | 09:01 |
| noonedeadpunk | in horizon virtualenv | 09:01 |
| noonedeadpunk | it's not smth we can/should overwrite | 09:01 |
| noonedeadpunk | but needs patching in murano-dahsboard itself | 09:01 |
| jrosser | morning | 09:02 |
| noonedeadpunk | jrosser: hey! | 09:04 |
| jrosser | hello | 09:04 |
| jrosser | took some time away last week, needed a rest | 09:04 |
| noonedeadpunk | was wondering if you saw this http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2020-11-06.log.html#t2020-11-06T14:05:50 and if you have any thoughts | 09:04 |
| pto | noonedeadpunk: Is the muran dashboard in the horizon container? | 09:05 |
| noonedeadpunk | pto: yep | 09:05 |
| noonedeadpunk | jrosser: yeah, can totally understand this | 09:05 |
| jrosser | noonedeadpunk: yes i did see it, and i think the biggest thing would be getting good CI | 09:06 |
| jrosser | like currently we find all the wierd cases across debian/ubuntu/centos/.... where the repos all change randomly and stuff | 09:06 |
| noonedeadpunk | I also a bit afraid of that role will get in weird shape sooner or later | 09:06 |
| noonedeadpunk | in terms that they were looking to split it into small parts... | 09:07 |
| jrosser | humm yes and we just undid splitting up roles for galera because it was a massive pain | 09:08 |
| jrosser | i guess in a way it would be a bit like ceph-ansible has many roles | 09:09 |
| noonedeadpunk | yeah, but I\m not sure about that as having it that way feels reasonable as well, especially for our usecase. And not sure we have resources to adopt to this as well... | 09:10 |
| noonedeadpunk | and honestly... feels like pulling collections will be such a pain in some time... | 09:11 |
| noonedeadpunk | as their size is going to grow a lot with this approach | 09:11 |
| jrosser | maybe it's OK to take that role and use it for the basis of stuff in the collection | 09:12 |
| jrosser | if we can use it in OSA or not is another question | 09:12 |
| jrosser | or if anyone has time to work on that too | 09:12 |
| noonedeadpunk | well agree | 09:12 |
| noonedeadpunk | but doesn't make much sense to work on them separately afterwards as well... | 09:13 |
| noonedeadpunk | and according to how we strugled with new version of rabbit (because upstream patch wasn't been approved for a while) it felt like we were one of the biggest users.... | 09:14 |
| jrosser | i expect we push the rabbit version forward more aggressively than others | 09:14 |
| jrosser | just stick with the distro package and the version will be older | 09:15 |
| noonedeadpunk | well yes | 09:15 |
| jrosser | it is a difficult choice, if we don't use a role from the collection then we duplicate work but keep in control | 09:16 |
| jrosser | but if we do use it then things might turn out not so good, like ceph-ansible | 09:16 |
| noonedeadpunk | yeah, have super mixed fillings about that as well | 09:17 |
| noonedeadpunk | *feelings | 09:17 |
| pto | noonedeadpunk: I think i will skip the murano - i dont have a clue what needs to be fixed in the config file. | 09:24 |
| *** sshnaidm_ is now known as sshnaidm|rover | 09:24 | |
| pto | What is the status of Masakari? Is it stable in ussuri? | 09:24 |
| noonedeadpunk | pto: iirc it is | 09:25 |
| noonedeadpunk | well, using it from train | 09:25 |
| noonedeadpunk | except in U you will need to handle pacemaker installation on your own | 09:26 |
| noonedeadpunk | I have https://github.com/noonedeadpunk/ansible-pacemaker-corosync but I think it's not doing pacemaker-remote at the moment | 09:27 |
| pto | Is masakari like vitrage? | 09:30 |
| pto | Auto healing of a dead compute? Or did I misundertand something? | 09:31 |
| noonedeadpunk | masakari auto evacuates instances from failed compute | 09:34 |
| pto | noonedeadpunk: Is it complicated to get working? | 09:42 |
| noonedeadpunk | no, not at all | 09:42 |
| noonedeadpunk | in case you have pacemaker+corosync cluster working:) | 09:42 |
| noonedeadpunk | the role I referenced did it great, but has limitation in 16 nodes, as otherwise you need to use pacemaker-remote | 09:43 |
| pto | noonedeadpunk: I have never triged to deploy a pacemaker cluster | 09:43 |
| pto | Have you ever looked at the Vitrage project? | 09:43 |
| noonedeadpunk | ah, is it project held by Nokia? | 09:45 |
| noonedeadpunk | I think it's pretty different | 09:45 |
| noonedeadpunk | as what masakari does - it checks pacemnaker cluster status and in case one compute host is down it tries to issue nova host-evacuate command. also it has processmonitor to check for the ps and restarting service if smth is down and instancemonitor, to ensure that VM with specific tag is always on;ine and boot it in case it goes down | 09:47 |
| noonedeadpunk | eventually pacemaker cluster is needed only for hostmonitor | 09:47 |
| pto | Correct, the vitrage project is backed by Nokia. I have used their commercial openstack for some time in my previous job. It uses vitrage combined with zabbix, and if a host goes down, zabbix informs vitrage and it will evacuate the host | 09:53 |
| * noonedeadpunk loves zabbix | 09:53 | |
| noonedeadpunk | yeah, but it kind of the way more complicated flow then with masakari imo) | 09:54 |
| pto | noonedeadpunk: tbh, i think masakari looks a little complicated to begin with. | 09:55 |
| pto | Not much usable documentation | 09:55 |
| noonedeadpunk | also nice thing I was using with masakari is reserved host. So we were having a disabled host, that was with 0 VMs and set in reserved state in masakari. And in case of host failure host is enabled in nova, unmarked from reservation and all vms moves to it | 09:55 |
| noonedeadpunk | so you kind of sure you will have enough resources to spawn everything | 09:56 |
| noonedeadpunk | while agree on poor documentation, it's working pretty reliable and straightforward | 09:56 |
| pto | That makes good sense. I spend allot of planning compute resources in my previous job (Telco). Almost everything depended on a SRVIOV and hard pinned CPU cores | 09:57 |
| pto | noonedeadpunk: The os cluster will have 50+ computes, will this fly with pacemaker+ | 10:07 |
| noonedeadpunk | yes, but as I said, it needs pacemaker-remote. also you can split up pacemaker clusters into groups and add them as different segmentation groups in masakari as well | 10:11 |
| pto | noonedeadpunk: Cool. Thanks for helping. I think i will skip this part for now, as I have plenty of other stuff to look at. But it will be on the todo with high priority | 10:13 |
| fanfi | Guys, could somebody help me please. When I trying to create a new image I got following error and I cant find any error in log files. :( ...Image creation failed: Unable to establish connection to http://172.16.1.91:9292/v2/images/b333b5df-29bb-40a7-adba-0fbbcc10d759/file : ('Connection aborted.', BrokenPipeError(32, 'Broken pipe')) | 10:18 |
| fanfi | Glace api works | 10:18 |
| fanfi | there is commands debug https://pastebin.com/7tVeXWSN | 10:18 |
| noonedeadpunk | I'm not sure if it's the case, but you can try setting glance_use_uwsgi: false in user-variables and re-run os-glance-install.yml playbook | 10:21 |
| jrosser | would also be worthwhile checking the glance service logs, as those errors are just from the client | 10:21 |
| noonedeadpunk | but I think it was some different issue here... | 10:21 |
| noonedeadpunk | fanfi: btw, logs are in journald | 10:22 |
| fanfi | thx, I will try it. | 10:22 |
| noonedeadpunk | as it seems you don;t use import out of the logs, which will fail with uwsgi, so yeah, seems really idfferent | 10:25 |
| pto | noonedeadpunk: Does the pacemaker run on ubuntu? | 10:52 |
| noonedeadpunk | well I run it on ubuntu | 10:52 |
| *** tosky has joined #openstack-ansible | 11:09 | |
| pto | noonedeadpunk: I have removed the murano setting, but the os-horizon keeps failing with the same error. I have removed the container using lxc-containers-destroy.yml. What else need to be removed? ansible_facts? | 11:16 |
| noonedeadpunk | um, no, host from inventory | 11:43 |
| noonedeadpunk | we have /scripts/inventory-manage.py | 11:43 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Use a consolidated gate queue for integrated jobs https://review.opendev.org/660751 | 11:48 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Use a consolidated gate queue for integrated jobs https://review.opendev.org/660751 | 11:50 |
| fanfi | @noonedeadpunk @noonedeadpunk it's works. I added the setting to user-variables than I saw wrong authentication in log :) ...and that was easy to fixed :) thanks | 11:51 |
| noonedeadpunk | jrosser: have you ever seen https://review.opendev.org/#/c/584857/5 ?:) | 11:54 |
| pto | noonedeadpunk: I have removed the horizon container including files, and the facts files and this time removed from inventory too. Still failing on horizin. The configs are working from the run before murano was activated | 12:04 |
| *** rfolco has joined #openstack-ansible | 12:04 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Fix upgrade jobs for bind-to-mgmt https://review.opendev.org/758461 | 12:08 |
| *** cloudxtiny has joined #openstack-ansible | 12:14 | |
| cloudxtiny | Hello | 12:14 |
| cloudxtiny | it seems gnocchi_service_setup_host variable is getting duplicated in /etc/ansible/roles/os_gnocchi/defaults/main.yml so database setup fails | 12:15 |
| cloudxtiny | is that file created dynamically so I must have set something up wrong | 12:19 |
| *** yasemind has quit IRC | 12:21 | |
| *** ericzolf has joined #openstack-ansible | 12:24 | |
| noonedeadpunk | I don't see it being dublicated..... | 12:25 |
| noonedeadpunk | what version arte you using? | 12:25 |
| cloudxtiny | noonedeadpunk how do I tell the version? I think i pulled the lastest | 12:26 |
| noonedeadpunk | when you've pulled openstack/openstack-ansible you probably did checkout to some version (it's not required but most likely) | 12:28 |
| noonedeadpunk | and /etc/ansible/roles/os_gnocchi/defaults/main.yml is not created dynamically, it's taken from https://opendev.org/openstack/openstack-ansible-os_gnocchi/src/branch/master/defaults/main.yml | 12:29 |
| cloudxtiny | I cloned master :-( | 12:36 |
| cloudxtiny | git clone -b master https://opendev.org/openstack/openstack-ansible /opt/openstack-ansible | 12:36 |
| cloudxtiny | I was following this tutorial -----> https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/deploymenthost.html | 12:37 |
| noonedeadpunk | ok, I don't see this option being dublicated in master | 12:37 |
| cloudxtiny | humm. I should have selected the latest release version. | 12:37 |
| cloudxtiny | thanks | 12:37 |
| noonedeadpunk | can you comment out no_log statement in /etc/ansible/roles/os_gnocchi/tasks/db_setup.yml and try running role one more time? (if it's the task where it failed) | 12:38 |
| cloudxtiny | yeah I commented out line 75 and it worked | 12:39 |
| cloudxtiny | #gnocchi_db_setup_host: "{{ ('galera_all' in groups) | ternary(groups['galera_all'][0], 'localhost') }}" | 12:40 |
| cloudxtiny | seems my version was trying to use the galera node as the utility node for setting up the database | 12:41 |
| noonedeadpunk | oh my | 12:41 |
| noonedeadpunk | you're right and I'm blind | 12:41 |
| cloudxtiny | :-) . Just happy to help out :-) | 12:42 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_gnocchi master: Remove dublicated gnocchi_db_setup_host https://review.opendev.org/761910 | 12:42 |
| noonedeadpunk | that's super useful | 12:42 |
| *** cloudxtiny has quit IRC | 12:43 | |
| noonedeadpunk | merging https://review.opendev.org/#/c/760818/ would be super cool | 12:43 |
| *** cloudxtiny has joined #openstack-ansible | 12:44 | |
| jrosser | done | 12:46 |
| jrosser | noonedeadpunk: i've never seen that actually needed in a cert, the IP: {{ internal_vip }} SAN | 12:48 |
| jrosser | but then again i've never seriously tried to make the selfsigned stuff actually be trusted | 12:48 |
| jrosser | the patch would suggest that there wasnt even internal DNS pointing to the VIP | 12:49 |
| noonedeadpunk | yeah, me neither. | 12:49 |
| openstackgerrit | Georgina Shippey proposed openstack/openstack-ansible-galera_server master: Ability to take mariadb backups using mariabackup https://review.opendev.org/755261 | 12:50 |
| jrosser | that pattern is used in some of the ops repo stuff though | 12:51 |
| noonedeadpunk | actually patch is almost 3 years old, but it still feels interesting... maybe worth taking into consideration during ssl redesign | 12:52 |
| jrosser | yes, it would certainly be good if when a selfsigned cert was made it used a CA | 12:53 |
| jrosser | that patch also does create a CA cert which the commit message doesnt really talk about | 12:53 |
| openstackgerrit | Merged openstack/openstack-ansible master: Bump ansible version to 2.10.3 https://review.opendev.org/761443 | 12:53 |
| jrosser | but in a way thats really the most useful thing it does | 12:53 |
| *** spatel has joined #openstack-ansible | 12:55 | |
| *** spatel has quit IRC | 12:59 | |
| cloudxtiny | Are the ceilometer APIs not exposed via haproxy? | 13:00 |
| cloudxtiny | can't see them here: https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/haproxy/haproxy.yml | 13:01 |
| cloudxtiny | Nevermind. | 13:03 |
| noonedeadpunk | yeah, ceilometer does not have api.... | 13:08 |
| cloudxtiny | yeah, just realised that what gnocchi is for now :-) | 13:11 |
| cloudxtiny | noonedeadpunk similar issue for Aobh as well ----> "/etc/ansible/roles/os_aodh/defaults/main.yml" [Modified] line 58 | 13:41 |
| *** d34dh0r53 has quit IRC | 13:46 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_aodh master: Remove dublicated aodh_db_setup_host https://review.opendev.org/761922 | 13:46 |
| noonedeadpunk | /o\ | 13:46 |
| cloudxtiny | :-) | 13:47 |
| noonedeadpunk | well, good that you've picked up master :) | 13:47 |
| cloudxtiny | happy to help | 13:47 |
| *** d34dh0r53 has joined #openstack-ansible | 13:52 | |
| *** rh-jelabarre has joined #openstack-ansible | 13:54 | |
| *** spatel has joined #openstack-ansible | 13:59 | |
| pto | I know its a little OT, but have anyone here tested OpenStack Migrate (https://os-migrate.github.io/os-migrate/index.html)? | 14:02 |
| noonedeadpunk | we do upgrades, so have no idea why this even exist, except you stick with newton and want ussuri in 1 step | 14:04 |
| noonedeadpunk | as you kind of need a lot of extra hardware here | 14:04 |
| pto | I know, but there is no plausible upgrade path from queens to ussuri | 14:05 |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance stable/stein: Do not symlink ceph libraries for distro path https://review.opendev.org/760818 | 14:05 |
| *** cloudnull has quit IRC | 14:11 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible stable/stein: Bump stable/stein for last release https://review.opendev.org/761931 | 14:13 |
| ThiagoCMC | Morning guys! How can I make one of my haproxy nodes, the "main one"? | 14:14 |
| noonedeadpunk | well it's up to keepalived to decide where ip will be spawned | 14:16 |
| noonedeadpunk | it has weights | 14:16 |
| *** cloudnull has joined #openstack-ansible | 14:18 | |
| noonedeadpunk | so haproxy is main where vip is spawned | 14:18 |
| ThiagoCMC | I know... I tried to "systemctl status haproxy.service" but it came back | 14:19 |
| ThiagoCMC | But, no worries... Thanks! | 14:20 |
| ThiagoCMC | Currently, I have a bigger problem... lol | 14:21 |
| noonedeadpunk | keepalived has haproxy script which returns haproxy when keepalived is active | 14:21 |
| ThiagoCMC | Oh, ok | 14:21 |
| noonedeadpunk | and keepalived will tend to return to the same master as it has higher weight set explicitly | 14:21 |
| ThiagoCMC | Got it! | 14:21 |
| ThiagoCMC | Nice! | 14:21 |
| ThiagoCMC | I forgot about keepalived... | 14:22 |
| noonedeadpunk | we have some variables for that I think but can't instanly recall | 14:22 |
| ThiagoCMC | No problem | 14:22 |
| ThiagoCMC | So, another issue... I reinstalled all controllers (fresh deployment) but I kept the compute nodes, now, they're all trowing the following error: http://paste.openstack.org/show/799811/ | 14:23 |
| ThiagoCMC | Any idea about how to clean it up? | 14:23 |
| spatel | ThiagoCMC: just adjust priority and you be able to decide which will be primary all the time. | 14:23 |
| -spatel- [root@infra-lxb-1 conf.d]# cat /etc/keepalived/keepalived.conf | grep priority | 14:23 | |
| -spatel- priority 100 | 14:23 | |
| ThiagoCMC | spatel, awesome!!! :-D | 14:23 |
| ThiagoCMC | I tried to "nova-manage cell_v2 delete_host --cell_uuid <CELL_UUID> --host <HOST>" but it didn't solved the above issue... | 14:25 |
| spatel | ThiagoCMC: look like nova-placement related issue. | 14:25 |
| ThiagoCMC | yep | 14:25 |
| ThiagoCMC | I'm wondering if there is a way to clean it up without re-deployibng the whole thing, agian. | 14:25 |
| ThiagoCMC | *again | 14:25 |
| spatel | you need to delete your compute nodes from placement and let them re-create new UUID | 14:26 |
| ThiagoCMC | Any docs about how to achieve that? | 14:26 |
| spatel | http://paste.openstack.org/show/799827/ | 14:27 |
| spatel | try delete <compute_node> do it on 1 node and restart nova-compute on compute node and it should re-create new UUID | 14:28 |
| ThiagoCMC | Trying it now | 14:30 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible stable/stein: Switch to stable/stein for EM https://review.opendev.org/761937 | 14:37 |
| noonedeadpunk | jrosser: when you have enought time would be great to take a look at https://review.opendev.org/#/c/756313/ | 14:39 |
| noonedeadpunk | I can set reversed backend, as the integrated reo fails | 14:39 |
| noonedeadpunk | but I checked that it was passing on patchset 9 | 14:39 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Added Openstack Adjutant role deployment https://review.opendev.org/756310 | 14:41 |
| *** gshippey has joined #openstack-ansible | 14:49 | |
| fanfi | folks, could you help me please again :) ...on the compute node when I would like to start new instacnce i getting following error https://pastebin.com/1r4hjbwH aborted: Unable to update attachment.(Bad or unexpected response from the storage volume backend API: Driver initialize connection failed (error: Unexpected error while running command | 14:51 |
| fanfi | if I run the commad manulaly..works | 14:52 |
| fanfi | and the package python3-ceph-argparse is already installed. | 14:53 |
| noonedeadpunk | that;s interesting | 14:54 |
| jrosser | distro install :( | 14:54 |
| fanfi | yes, it's distro | 14:55 |
| noonedeadpunk | ah.... | 14:55 |
| noonedeadpunk | than it explains a lot | 14:55 |
| fanfi | :( ...but how I can fixed ? | 14:56 |
| noonedeadpunk | do you have these patches? https://review.opendev.org/#/q/status:merged+topic:ceph_client_distro | 14:56 |
| noonedeadpunk | I'm wondering also what python nova is using... | 14:58 |
| jrosser | fanfi: i'm interested to know why you prefer the distro install? | 14:58 |
| jrosser | noonedeadpunk: this is kind of zero test coverage really, centos + distro + ceph | 14:59 |
| noonedeadpunk | oh, it's centos | 14:59 |
| jrosser | i *think* so | 15:00 |
| noonedeadpunk | meh | 15:00 |
| jrosser | fanfi: can you confirm which OS you are using? | 15:00 |
| noonedeadpunk | it' liekly that nova is py2 then at all | 15:00 |
| fanfi | jrosser I do not have any prefernces...I was think it was more quick and better :( but it was wrong idea probably | 15:01 |
| fanfi | yes centos8 | 15:01 |
| noonedeadpunk | on centos 8 it should be py3... | 15:01 |
| fanfi | yes | 15:01 |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-os_tempest master: Allow deployer to skip default resource creation https://review.opendev.org/733892 | 15:02 |
| fanfi | it's better to reinstall OSA and use source insall mettod | 15:04 |
| fanfi | ? | 15:04 |
| noonedeadpunk | well, it's better to use source method, yes:) | 15:04 |
| noonedeadpunk | I really not sure why this error happens. It's worth looking at what python nova is actually usng, as it might be that it's different one from system python, and so missing libraries... | 15:06 |
| noonedeadpunk | this shouldn't be actually the case but I don't really see why this issue might happen so needs deeper look | 15:06 |
| spatel | fanfi: yes source is best way to go, i am running pretty big cloud on CentOS using source and so far so good | 15:06 |
| fanfi | ah....ok :) i will go to reinstall the environment | 15:08 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Fix upgrade jobs for bind-to-mgmt https://review.opendev.org/758461 | 15:09 |
| noonedeadpunk | at least source is better tested and more reliable for sure | 15:10 |
| ThiagoCMC | Just curious about something... Have you guys ever tried to run the `neutron-*-agens` inside of a LXD/LXC Container? The command: `ip netns exec qdhcp-BLAH bash -i` returns: "mount of /sys failed: Operation not permitted" and I'm wondering what exactly do I have to allow at the LXD host... (just for fun) :-P | 15:10 |
| noonedeadpunk | ThiagoCMC: mgariepy did I think | 15:11 |
| ThiagoCMC | Cool! | 15:11 |
| noonedeadpunk | but iirc he said it's broken now :( | 15:11 |
| ThiagoCMC | I have all my Ceph OSD, and KVM hosts, inside of LXD containres and now, time to do the same with Network Nodes! lol | 15:12 |
| ThiagoCMC | Oh, okay... | 15:12 |
| noonedeadpunk | well there was solid performance penalty and some things were weird so we decided to move networking out of containers by default a while ago | 15:12 |
| ThiagoCMC | Yep, I know! | 15:13 |
| noonedeadpunk | but I think it's technically possible and I guess worth fixing anyway | 15:14 |
| ThiagoCMC | I'll give this another shot! | 15:14 |
| mgariepy | dmsimard, https://ara.recordsansible.org/ cert expired ? | 15:14 |
| dmsimard | :( | 15:18 |
| jamesdenton | jrosser Any thoughts on this requirements failure? https://review.opendev.org/#/c/588372/ | 15:18 |
| mgariepy | the let's encrypt bot needs some monitoring ;p | 15:19 |
| noonedeadpunk | I think GitPython is not on openstack available requirements file | 15:19 |
| dmsimard | mgariepy: fixed ty | 15:19 |
| mgariepy | dmsimard, thanks | 15:20 |
| noonedeadpunk | we can use what is in https://opendev.org/openstack/requirements/src/branch/master/global-requirements.txt | 15:20 |
| noonedeadpunk | oh, hm, it's in it | 15:20 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Use parallel git clone https://review.opendev.org/588372 | 15:22 |
| *** nurdie has joined #openstack-ansible | 15:22 | |
| jamesdenton | noonedeadpunk thx | 15:23 |
| noonedeadpunk | what I don't like here is location of the library.... | 15:25 |
| *** yann-kaelig has joined #openstack-ansible | 15:28 | |
| ThiagoCMC | noonedeadpunk, the "security.nesting=true" did the trick with the "ip netns"! Now, checking other problems... :-D | 15:33 |
| noonedeadpunk | would be great to know full list of steps to recover that path:) | 15:34 |
| spatel | jamesdenton: i have successfully deploy ovs+dpdk using OSA (i have some draft patch which i am going to submit for CentOS-8 support). | 15:36 |
| spatel | noonedeadpunk: do we have OVS+DPDK CI job for validation? | 15:37 |
| openstackgerrit | Siavash Sardari proposed openstack/openstack-ansible-os_neutron master: Remove securitygoup section due to duplication in agents config file https://review.opendev.org/761954 | 15:37 |
| *** nurdie has quit IRC | 15:41 | |
| jamesdenton | spatel that's great to hear! | 15:42 |
| openstackgerrit | Siavash Sardari proposed openstack/openstack-ansible-os_neutron master: Remove securitygoup section due to duplication in agents config file https://review.opendev.org/761815 | 15:42 |
| spatel | jamesdenton: Now i am testing SR-IOV + DPDK combine deployment. (where i will add VF as a DPDK interface to solve my bonding issue) | 15:43 |
| *** nurdie has joined #openstack-ansible | 15:45 | |
| jamesdenton | was it not enough to create an openvswitch bond? | 15:47 |
| *** rpittau is now known as rpittau|bbl | 15:52 | |
| jrosser | jamesdenton: i wonder if it is becasue there is no constraint supplied? | 15:53 |
| spatel | jamesdenton: in my case i have only 2x10G nic and if i assigned both nic to dpdk then i don't have any mgmt nic to ssh into host. | 15:53 |
| jamesdenton | spatel gotcha. | 15:54 |
| spatel | Best practice is having multiple nic and dedicated nic for dpdk but in my case i have blade center and i can't add NIC in it. | 15:54 |
| *** cloudnull has quit IRC | 15:55 | |
| spatel | I am thinking i am going to create multiple VF using SR-IOV and assigned those VF from both pysical nic to br-mgmt and ovs to bond them. for br-mgmt i will use LinuxBridge and for br-vlan i will use OVS+DPDK | 15:56 |
| spatel | each VF has own PCI BUS ID so i will use that to build bond inside OVS | 15:57 |
| spatel | i need to create very smart PXE boot to handle all this stuff without human intervene | 15:58 |
| jamesdenton | indeed | 15:58 |
| spatel | jamesdenton: do we have zuul CI job to validate ovs+dpdk code? | 16:00 |
| spatel | I don't think so but still asking :) | 16:00 |
| jamesdenton | we do not. | 16:00 |
| spatel | no worry. | 16:00 |
| *** cloudnull has joined #openstack-ansible | 16:02 | |
| ThiagoCMC | Any idea about this error: "neutron-linuxbridge-agent ..... ERROR neutron.agent.linux.ip_lib [req-...] Device brq3cdcc787-c4 cannot be used as it has no MAC address" ? | 16:06 |
| ThiagoCMC | Safe to ignore? lol | 16:07 |
| *** jgwentworth is now known as melwitt | 16:12 | |
| cloudxtiny | hummm... ionic install seems to be dependent on swift. I am not setting up swift. Is there anyway around that? | 16:16 |
| jrosser | cloudxtiny: https://github.com/openstack/openstack-ansible-os_ironic/blob/master/defaults/main.yml#L107-L116 | 16:17 |
| jrosser | the defaults/main.yml of all of these ansible roles is the bits you can tweak with | 16:18 |
| cloudxtiny | sweet thanks | 16:18 |
| cloudxtiny | For this "This requires http_root and | 16:22 |
| cloudxtiny | can I just use the repo container? | 16:22 |
| jrosser | as far as i can see you don't need to do anything, this is all automatic https://github.com/openstack/openstack-ansible-os_ironic/blob/master/templates/ironic.conf.j2#L66-L69 | 16:24 |
| jrosser | i think it will make a web server in the ironic container itself, but i've never deployed this | 16:25 |
| jrosser | jamesdenton: ^ you've done this I think? | 16:25 |
| *** pto has quit IRC | 16:25 | |
| jamesdenton | looking | 16:25 |
| jamesdenton | with ironic_enable_web_server_for_images it creates a local http server, yes. no need for swift. i am not sure i have tested it with multi-node | 16:27 |
| jrosser | not sure if that uses the loadbalanber or if the local dhcp server will just point next-server to it's local http server.... | 16:28 |
| jrosser | i.e each container does it's own thing and the one which does the dhcp wins | 16:28 |
| jamesdenton | right, that's what i would want to verify. | 16:29 |
| *** klamath_atx has joined #openstack-ansible | 16:30 | |
| jrosser | ewwww https://github.com/openstack/openstack-ansible-os_ironic/blob/master/files/dhcpd.conf#L39 | 16:32 |
| jamesdenton | when ironic_standalone | 16:34 |
| jamesdenton | but yeah, that needs some work | 16:35 |
| ThiagoCMC | jamesdenton, hey buddy! Sorry to ask again... lol - Do you know if it would be ok to ignore that "Device brq3cdcc787-c4 cannot be used as it has no MAC address" error? | 16:53 |
| jamesdenton | what are the circumstances of the error? | 16:53 |
| ThiagoCMC | When I create an L3 Router, it shows that. AND, it's running inside of a LXD container (it's actually working - has connectivity). Just curious if you see this message ofter too, or not. | 16:55 |
| ThiagoCMC | Sorry, ERROR is actually from neutron-linuxbridge-agent and neutron-dhcp-agent, not l3. | 16:56 |
| jamesdenton | can you send a paste of 'ip link shwo brq3cdcc787-c4' ? | 16:56 |
| jamesdenton | *show | 16:56 |
| ThiagoCMC | Here: http://paste.openstack.org/show/799834/ | 16:57 |
| *** klamath_atx has quit IRC | 17:02 | |
| jamesdenton | https://opendev.org/openstack/neutron/src/branch/master/neutron/agent/linux/ip_lib.py#L955-L967 | 17:04 |
| jamesdenton | might be safe to ignore if its then going back and setting it up | 17:04 |
| ThiagoCMC | Hmm... Cool, thanks! | 17:11 |
| *** rpittau|bbl is now known as rpittau | 17:22 | |
| *** viks____ has quit IRC | 17:25 | |
| *** cloudxtiny has quit IRC | 17:32 | |
| openstackgerrit | James Denton proposed openstack/openstack-ansible-os_ironic master: Update Inspector listener address and network https://review.opendev.org/760660 | 17:35 |
| ThiagoCMC | BTW, OSA Ussuri is also affected by: https://bugs.launchpad.net/neutron/+bug/1887281 | 17:50 |
| openstack | Launchpad bug 1887281 in neutron "[linuxbridge] ebtables delete arp protect chain fails" [Medium,In progress] - Assigned to Lukas Steiner (steinerlukas) | 17:50 |
| ThiagoCMC | I manually applied the patch, ERROR gone! :-P | 17:50 |
| jamesdenton | Is this CentOS specific | 17:55 |
| jamesdenton | noonedeadpunk re: uwsgi for neutron. I am not able to replicate the failure locally with centos distro install | 17:58 |
| *** ericzolf has quit IRC | 17:59 | |
| ThiagoCMC | It was happening with Ubuntu 20.04 as well. | 18:01 |
| ThiagoCMC | I've never used CentOS in my life! :-P | 18:02 |
| jamesdenton | good to know, thank you | 18:02 |
| jamesdenton | hah | 18:02 |
| ThiagoCMC | NP! | 18:02 |
| *** andrewbonney has quit IRC | 18:09 | |
| ThiagoCMC | Wheee! My latest OSA deployment is finally working! It's unique! 1- Controllers are QEMU VMs; 2- Ceph OSDs are LXD Containers; 3- Compute Nodes are LXD; 4- Network Nodes are also LXD Containers! | 18:27 |
| ThiagoCMC | I'm not seeing any performance issue with the Neutron Agents as LXD containers! :-D | 18:27 |
| *** gyee has joined #openstack-ansible | 18:30 | |
| *** miloa has quit IRC | 18:33 | |
| ThiagoCMC | Check it out! https://imgur.com/a/D5JckcD <- This Ubuntu QEMU/KBM Instance (OSA Ussuri) is running inside of a bare-metal LXD Container! LOL | 18:35 |
| ThiagoCMC | Also it's L3 Router! | 18:35 |
| mgariepy | nice | 18:38 |
| mgariepy | ThiagoCMC, where is your blog ? | 18:38 |
| mgariepy | ;p | 18:38 |
| * jrosser still has WIP lxd roles for OSA.... | 18:42 | |
| *** nsmeds has joined #openstack-ansible | 18:43 | |
| jrosser | ThiagoCMC: did you get anywhere with qdrouterd? | 18:43 |
| *** rpittau is now known as rpittau|afk | 18:45 | |
| *** yann-kaelig has quit IRC | 18:47 | |
| ThiagoCMC | mgariepy, I don't have any blog! (facepalm) LOL | 18:55 |
| ThiagoCMC | jrosser, it failed but, I'm willing to try again. The OS Services could not authenticate against qdrouterd (auth failed) | 18:56 |
| jrosser | maybe something good to poke at in an AIO build | 18:59 |
| ThiagoCMC | Hmmm... That sounds like a good idea! | 19:00 |
| ThiagoCMC | I still have to work to enable Erasure Code in my Ceph pools for OSA, also better understand Cinder integration with Ceph and then, I'll give qdrouterd another shot. | 19:02 |
| ThiagoCMC | This is a Cloud in my basement and I only have 48TB, with replica 3, I'm down to 16TB, which sucks | 19:03 |
| ThiagoCMC | I might enable EC and Compression lol | 19:04 |
| mgariepy | ThiagoCMC, for EC, i can give you some hint if you want. | 19:11 |
| ThiagoCMC | Awesome!!! | 19:12 |
| ThiagoCMC | I have a few questions before getting there =P | 19:12 |
| ThiagoCMC | BTW! I have a couple of questions for the Ceph masters here! Under "admin/hypervisors", I can see Ceph as a "Local Storage" and it shows up 48T BUT, because of replica 3, I don't actually have 48T available. Is there any way to tell OpenStack what "ceph df" shows? | 19:12 |
| ThiagoCMC | And, when with Ceph, is there any way to still use the local storage of each Compute Node? | 19:13 |
| mgariepy | you can do different things. like having nova use ephemeral storage on your computes, and use ceph in cinder via volume | 19:13 |
| ThiagoCMC | Great! | 19:14 |
| mgariepy | but you won't be able to use the same hdd/ssd as both in the same time ;P | 19:14 |
| ThiagoCMC | Sure :-D | 19:14 |
| mgariepy | as for the df stuff i'm not sure haha | 19:14 |
| ThiagoCMC | I have good 1T NVMe on each compute node, doing nothing... It would make sense to use them but, they don't show up anywhere! | 19:15 |
| mgariepy | ThiagoCMC, https://github.com/openstack/openstack-ansible-os_nova/blob/master/templates/nova.conf.j2#L237-L240 | 19:20 |
| mgariepy | tldr, if you don't have nova_libvirt_images_rbd_pool your local storage will be used. | 19:21 |
| mgariepy | if formated and mounted at the correct place. | 19:21 |
| mgariepy | like: /var/lib/nova | 19:21 |
| openstackgerrit | Merged openstack/openstack-ansible-os_manila master: Start using uWSGI role https://review.opendev.org/704935 | 19:31 |
| ThiagoCMC | mgariepy, so, do I have to upload the same image, twice, to Glance? One RAW for Ceph and another QCOW2 for "regular"? | 19:32 |
| mgariepy | it's possible to config cinder caching for the images | 19:41 |
| mgariepy | ThiagoCMC, https://docs.openstack.org/cinder/latest/admin/blockstorage-image-volume-cache.html | 19:42 |
| mgariepy | it works pretty well . i didn't had issue with that. | 19:43 |
| ThiagoCMC | Thanks! | 19:58 |
| ThiagoCMC | About EC pools, what do you think of this post: https://themeanti.me/technology/2018/08/23/ceph_erasure_openstack.html ? | 19:59 |
| *** tosky has quit IRC | 20:17 | |
| *** tosky has joined #openstack-ansible | 20:28 | |
| *** cshen has quit IRC | 20:29 | |
| *** cshen has joined #openstack-ansible | 20:37 | |
| djhankb | Hey folks - is anyone able to help me figure out my cloudkitty installation? I was able to get containers built with VENVs, but UWSGI is bombing out trying to find the 'encodings' module, which I thought was a builtin... http://paste.openstack.org/show/799846/ | 20:53 |
| *** klamath_atx has joined #openstack-ansible | 20:55 | |
| *** klamath_atx has quit IRC | 21:01 | |
| *** rfolco has quit IRC | 21:28 | |
| *** klamath_atx has joined #openstack-ansible | 22:10 | |
| *** klamath_atx has quit IRC | 22:15 | |
| *** nurdie has quit IRC | 22:36 | |
| *** klamath_atx has joined #openstack-ansible | 22:38 | |
| ThiagoCMC | admin0, hey buddy! I'm curious about something, in your FAQ: "https://www.openstackfaq.com/openstack-ansible-ceph/" - you configured the "cinder_backends:" 3 times (for example, at conf.d/cinder.yml) but, at the documentation "https://docs.openstack.org/openstack-ansible/latest/user/ceph/full-deploy.html" it's actually at the user_variables.yml declared just once. Why you did it 3 times, is there any | 22:45 |
| ThiagoCMC | difference from the two ways? | 22:45 |
| *** klamath_atx has quit IRC | 22:48 | |
| *** spatel has quit IRC | 22:49 | |
| ThiagoCMC | In my environment, I ended up with the two! But, from what I'm seeing, only the values from "user_variables-ceph.yml" are going to cinder_volumes' cinder.conf file. So, I'm not sure if the "container_vars:" under "conf.d/cinder.yml" is being ignored, or not. :-P | 23:05 |
| *** cshen has quit IRC | 23:06 | |
| *** tosky has quit IRC | 23:33 | |
| *** nurdie has joined #openstack-ansible | 23:47 | |
| *** nurdie has quit IRC | 23:51 | |
| *** cshen has joined #openstack-ansible | 23:55 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!