| *** cshen has joined #openstack-ansible | 00:43 | |
| *** cshen has quit IRC | 00:48 | |
| *** yolanda has quit IRC | 01:26 | |
| *** yolanda has joined #openstack-ansible | 01:26 | |
| *** yolanda has quit IRC | 01:27 | |
| *** yolanda has joined #openstack-ansible | 01:27 | |
| *** spatel has joined #openstack-ansible | 01:31 | |
| spatel | folks, Just deployed k8s cluster using magnum and now trying to tun kubectl but asking for password and not sure what is this password, in process it never ask for password to enter | 01:32 |
|---|---|---|
| -spatel- [root@aio1-utility-container-3cfca9fd my_k8s]# kubectl get pods | 01:32 | |
| -spatel- Please enter Username: | 01:32 | |
| *** cshen has joined #openstack-ansible | 02:43 | |
| *** dave-mccowan has quit IRC | 02:44 | |
| *** cshen has quit IRC | 02:48 | |
| *** spatel has quit IRC | 03:38 | |
| *** raukadah is now known as chkumar|rover | 04:20 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-ansible | 04:33 | |
| *** cshen has joined #openstack-ansible | 04:43 | |
| *** cshen has quit IRC | 04:48 | |
| *** jbadiapa has joined #openstack-ansible | 05:28 | |
| *** cshen has joined #openstack-ansible | 05:32 | |
| *** cshen has quit IRC | 05:37 | |
| *** miloa has joined #openstack-ansible | 06:08 | |
| *** sep has joined #openstack-ansible | 07:02 | |
| *** cshen has joined #openstack-ansible | 07:10 | |
| *** miloa has quit IRC | 07:12 | |
| jrosser | noonedeadpunk: can you review this https://review.opendev.org/#/c/746366/ | 07:18 |
| noonedeadpunk | I always forget to vote on backport patches I made :( | 07:18 |
| noonedeadpunk | Like I see myself as Owner and just go futher | 07:18 |
| noonedeadpunk | Like I opened that patch several times during weekends ;( | 07:19 |
| jrosser | doh :) | 07:20 |
| jrosser | quite some things broken with the proposal bot patches | 07:20 |
| jrosser | centos/distro fails for magnum in a quite odd way https://zuul.opendev.org/t/openstack/build/0ffdff31c5e34b7bb68732794bbbdd24/log/logs/openstack/aio1-utility/tempest_run.log.txt#70 | 07:23 |
| jrosser | and something bad happens with ceilometer-upgrade on all the telemetry jobs | 07:25 |
| *** miloa has joined #openstack-ansible | 07:25 | |
| noonedeadpunk | yeah, saw both. for magnum I think maybe there's some extar package that needs to be installed for centos 8.... | 07:33 |
| *** tosky has joined #openstack-ansible | 07:56 | |
| admin0 | morning | 08:06 |
| *** oyrogerg has joined #openstack-ansible | 08:18 | |
| *** pcaruana has joined #openstack-ansible | 08:22 | |
| admin0 | hi all .. do you know of any quick thoughts on top of your head why on a ceph based cluster the instance stay at a Spawning state | 08:23 |
| admin0 | debug of both neutron and nova does not produce anything with errors .. i can see in ceph that the disk is created | 08:24 |
| masterpe | I was looking into the following: https://review.opendev.org/#/c/739146/9 | 09:15 |
| masterpe | and was wondering how the test where working. But if I open the logs of the test of "openstack-ansible-deploy-aio_lxc-ubuntu-bionic" then it never teste the masakari changes. | 09:17 |
| jrosser | it won't test masakari with that patch, becasue it is a patch to the openstack-ansible repo itself rather than the os_masakari role | 09:18 |
| jrosser | there would need to be a patch in the os_masakari repo which depends-on https://review.opendev.org/#/c/739146 in order to test it with masakari being deployed | 09:19 |
| jrosser | masterpe: here is such a patch actually https://review.opendev.org/#/c/744673/ | 09:21 |
| masterpe | but that patch does not test the corosync changes. | 09:22 |
| masterpe | but that was ofcourse a different question | 09:23 |
| jrosser | masterpe: regarding your comment here https://review.opendev.org/#/c/739146/9/playbooks/os-masakari-install.yml shouldnt the previous play gather the facts? | 09:25 |
| noonedeadpunk | +1 | 09:25 |
| masterpe | I was playing with this and I got the failure fatal: The error was: error while evaluating conditional (ansible_os_family | lower == 'redhat'): 'ansible_os_family' is undefined when I played os-masakari-install.yml with the corosync patch | 09:28 |
| *** shyamb has joined #openstack-ansible | 09:30 | |
| masterpe | Ah I see my problem, the task "- name: Gather masakari facts" is missing at my os-masakari-install.ym | 09:30 |
| noonedeadpunk | yeah, that is essential part) I think that if you run checkout command from Download dropdown, you'd get playbook as needed | 09:31 |
| masterpe | I'm running 19.1.0 and do not use master in the production :) | 09:33 |
| *** SmearedBeard has joined #openstack-ansible | 09:35 | |
| noonedeadpunk | ok, yes:) | 09:37 |
| noonedeadpunk | thqat's fair | 09:37 |
| *** shyamb has quit IRC | 09:41 | |
| *** shyamb has joined #openstack-ansible | 09:45 | |
| noonedeadpunk | jrosser: is Andrew in IRC? So we could ping him for reviews :p | 09:45 |
| jrosser | I think today he is taking holiday | 09:46 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_barbican master: Updated from OpenStack Ansible Tests https://review.opendev.org/745840 | 09:47 |
| * noonedeadpunk jealous | 09:48 | |
| *** shyamb has quit IRC | 09:53 | |
| *** shyamb has joined #openstack-ansible | 09:53 | |
| *** maharg101 has joined #openstack-ansible | 10:02 | |
| *** pcaruana has quit IRC | 10:15 | |
| admin0 | how do I troubleshoot (all) new instance stuck in build state .. this is a snippet from nova and neutron that I get: https://gist.githubusercontent.com/a1git/98bc43d6eda318f8ce5386fcc9dc6e62/raw/cb7b93d432f690c30223ee4452a0cd7a39b3fc00/gistfile1.txt .. the UUID is 35c2c021-5c1c-47ee-970d-52ffe62bceb2 | 10:18 |
| admin0 | [instance: 35c2c021-5c1c-47ee-970d-52ffe62bceb2] Claim successful on node h2 | 10:18 |
| admin0 | this log is from h2 | 10:19 |
| *** pcaruana has joined #openstack-ansible | 10:19 | |
| admin0 | the task i see is spawning .. virsh list --all on the hypervisor generates no output | 10:20 |
| noonedeadpunk | probably check nova-scheduler or placement logs | 10:22 |
| noonedeadpunk | oh, hm, seems like placement has chosen candidate and resources are claimed | 10:23 |
| admin0 | yes .. the logs are from h2 ( the hypervisor) | 10:26 |
| admin0 | and that is all the logs i have | 10:26 |
| admin0 | the disk is also claimed in ceph , which i can see from rados -p vms ls | grep $uuid | 10:27 |
| admin0 | also how is this journalctl supposed to work .. i have specified a log server and i see a log container .. but inside the log container,there are no logs from anywhere else .. journalctl --since "30 minutes ago" generates logs just from itself ? | 10:31 |
| admin0 | or do more stuff needs to be used in variables/config to have this logging | 10:32 |
| admin0 | after which i am hoping i can just do journalctl -u nova* and get all nova logs | 10:32 |
| noonedeadpunk | so journalctl responsible for logs collection only on host/container where it's launched | 10:33 |
| noonedeadpunk | so like journalctl -u nova* should show you all nova logs from that host/container out of the box | 10:34 |
| admin0 | in older tags, i could specifiy a log container and all logs where automatically appearing there | 10:34 |
| admin0 | log_host i meant | 10:34 |
| noonedeadpunk | I guess it was rsyslog? | 10:35 |
| admin0 | yep .. but it did not neeed anything in the user_vars i mean | 10:35 |
| admin0 | either way, what is the best way in the current tag to have all logs go in the logs_container | 10:35 |
| noonedeadpunk | eventually if you have graylog or elk - you can easily forward logs there | 10:35 |
| admin0 | it can be rsyslog, or journalctl | 10:35 |
| jrosser | the trouble with rsyslog is that over time stuff is logging more and more to systemd journal, so it' wouldnt necessarily end up on your log host anyway | 10:36 |
| jrosser | there is a feature in journald to forward logs over http from one host to another, but thats currently disabled https://opendev.org/openstack/openstack-ansible/commit/1bd21d80e47a66be145911d6be80992dbe0cc7c2 | 10:37 |
| noonedeadpunk | the best way is to have graylog or elk and forward logs there :) | 10:37 |
| jrosser | yes | 10:37 |
| jrosser | really the log hosts / containers today are not really doing anything unless you happen to follow the ELK setup guide from the ops repo | 10:39 |
| jrosser | personally for a prod deployment i'd not put the ELK stack on the controllers either | 10:40 |
| noonedeadpunk | ++ | 10:40 |
| admin0 | i will setup graylo g | 10:43 |
| admin0 | from the logs i have, what else i can do to figure out why its stuck in spawning state ? | 10:43 |
| openstackgerrit | amolkahat proposed openstack/openstack-ansible-os_tempest master: Added tempest ironic resources setup. https://review.opendev.org/720705 | 10:49 |
| admin0 | hi .. where can i find logs for graylog integration ? | 11:08 |
| admin0 | https://docs.openstack.org/openstack-ansible-rsyslog_server/latest/ is for rsyslog onluy | 11:08 |
| noonedeadpunk | admin0: https://opendev.org/openstack/openstack-ansible-ops/src/branch/master/graylog | 11:09 |
| admin0 | i already have external graylog | 11:10 |
| admin0 | just that target line in user_graylog.yml is enough ? | 11:11 |
| admin0 | nah | 11:11 |
| noonedeadpunk | then you can just use that playbook https://opendev.org/openstack/openstack-ansible-ops/src/branch/master/graylog/graylog-forward-logs.yml | 11:11 |
| admin0 | i have to replace graylog_target with a.b.c.d:1514 port format ? | 11:13 |
| noonedeadpunk | it will install journal2gelf and will forward logs in gelf format to your graylog server | 11:13 |
| masterpe | I have set to rsyslog_client_enabled to true and have set rsyslog_client_user_defined_targets | 11:13 |
| noonedeadpunk | you should set graylog_targets list | 11:13 |
| noonedeadpunk | in that format yes | 11:13 |
| masterpe | https://docs.openstack.org/openstack-ansible-rsyslog_client/stein/ is what I used as an example | 11:14 |
| admin0 | will it be a gelf-tcp or a gelf-udp input in the graylog ? | 11:18 |
| admin0 | in the playbook in the playbook, i see both https://opendev.org/openstack/openstack-ansible-ops/src/branch/master/graylog/graylog-forward-logs.yml, i see graylog_targets .. and graylog_target .. i am assuming i need to change graylog_tareget to "a.b.c.d:15151" which is the input udp port for gelf/syslog ? and again what do i change the graylog_targets to ? if i am running only that playbook ? | 11:21 |
| admin0 | masterpe, in your case, is the input syslog input ? | 11:22 |
| admin0 | syslog input ? | 11:22 |
| admin0 | syslog udp | 11:22 |
| noonedeadpunk | regarding tcp vs udp that's really awesome question. From what I see in journal2gelf is using GELFHandler, which is not valid anymore for graypy module... | 11:25 |
| noonedeadpunk | or, I was looking at wrong repo lol | 11:26 |
| noonedeadpunk | it's UDP | 11:27 |
| admin0 | :) | 11:27 |
| noonedeadpunk | As I said, you should set graylog_targets as in doc https://opendev.org/openstack/openstack-ansible-ops/src/branch/master/graylog#user-content-sending-logs-to-graylog2 | 11:28 |
| admin0 | noonedeadpunk, what do I replace graylog_targets in the playbook with ? | 11:30 |
| admin0 | i see i have to override 2 variabels .. graylog_target ( which i think is the graylog input:port) , but graylog_targets is not clear | 11:30 |
| *** cshen_ has joined #openstack-ansible | 11:31 | |
| admin0 | if i use masterpe method, is this enough? https://gist.github.com/a1git/5dc4912a43e7c5a6907cee84db8deab6 | 11:31 |
| noonedeadpunk | you shouldn't set graylog_target - it's loop var | 11:32 |
| noonedeadpunk | graylog_targets is a list of graylog server, where each element is ip:port format | 11:32 |
| admin0 | got it | 11:33 |
| *** cshen has quit IRC | 11:33 | |
| admin0 | noonedeadpunk, so i just run this 1 playbook: https://gist.github.com/a1git/21e6cf223f698b56dd319e8e1730e310 (my change in Line 25) | 11:34 |
| admin0 | that port is gelf udp input | 11:35 |
| admin0 | its running .. that yaml .. | 11:37 |
| admin0 | will know soon | 11:37 |
| admin0 | all containers failed with "Unable to find any of pip2, pip to use. pip needs to be installed." | 11:37 |
| admin0 | i guess ansible all -m shell -a "apt install python-pip" is sufficient ? | 11:38 |
| admin0 | or python -m ensurepip --default-pip ? | 11:39 |
| *** dave-mccowan has joined #openstack-ansible | 11:39 | |
| admin0 | using apt method | 11:40 |
| masterpe | admin0: yes that is enough. | 11:42 |
| masterpe | the syslog playbook is run at every setup-openstack .... role at the end. | 11:42 |
| openstackgerrit | Merged openstack/openstack-ansible-os_designate master: Updated from OpenStack Ansible Tests https://review.opendev.org/745845 | 11:45 |
| jrosser | seems like some kind of systematic failure on lxc centos8 jobs https://zuul.opendev.org/t/openstack/build/ebf6a61a58e3472480e48ca524352b59/log/logs/host/lxc-cache-prep-commands.log.txt#53-54 | 11:54 |
| *** rh-jelabarre has joined #openstack-ansible | 11:58 | |
| *** miloa has quit IRC | 11:58 | |
| masterpe | hi noonedeadpunk about https://github.com/noonedeadpunk/ansible-pacemaker-corosync/blame/master/tasks/pacemaker.yml | 12:09 |
| masterpe | souldn't "when: pacemaker_corosync_use_logfile | bool is true" be "when: pacemaker_corosync_use_logfile == true" | 12:10 |
| jrosser | a non empty string will evaulate as true | 12:12 |
| jrosser | which is quite likley not what the intention is | 12:12 |
| masterpe | ah, with this line I get error was: template error while templating string: no test named 'true' | 12:14 |
| masterpe | But I use ansible 2.7.16, I don't if this behavior has been changed. | 12:15 |
| noonedeadpunk | yep, I'm pretty sure it is:( | 12:16 |
| admin0 | graylog is working \o/ | 12:17 |
| admin0 | but i am still not getting any logs or indications as why my instances are stuck in spawn state :) | 12:17 |
| noonedeadpunk | masterpe: but this can be made better | 12:18 |
| *** rh-jelabarre has quit IRC | 12:18 | |
| noonedeadpunk | masterpe: I think that should do the trick https://github.com/noonedeadpunk/ansible-pacemaker-corosync/commit/7ae17fbd39885e0b37383e6d192d25e54d8554c8 | 12:20 |
| masterpe | noonedeadpunk: beter | 12:22 |
| *** rh-jelabarre has joined #openstack-ansible | 12:25 | |
| noonedeadpunk | jrosser: we have telemetry failing vbecause of this... https://review.opendev.org/#/c/746491/ | 12:29 |
| *** dave-mccowan has quit IRC | 12:32 | |
| *** shyamb has quit IRC | 12:39 | |
| *** dave-mccowan has joined #openstack-ansible | 12:41 | |
| *** SmearedBeard has quit IRC | 12:43 | |
| jrosser | noonedeadpunk: should we pin gnocchi back to something before that change? | 12:49 |
| noonedeadpunk | eventually it's the fix | 12:52 |
| noonedeadpunk | neither gnocchi nor ceilometer changed | 12:53 |
| noonedeadpunk | I think we bumped upper-connstraints version | 12:53 |
| noonedeadpunk | and some library update made things broken | 12:53 |
| noonedeadpunk | I hope telemetry will merge it asap, as they goit their CI broken as well | 12:53 |
| *** cshen_ has quit IRC | 13:04 | |
| *** cshen has joined #openstack-ansible | 13:06 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_neutron master: Make has_offline_migrations output predictable https://review.opendev.org/746496 | 13:07 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_neutron master: Make has_offline_migrations output predictable https://review.opendev.org/746496 | 13:07 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: Remove deprecated scenario image option https://review.opendev.org/720242 | 13:07 |
| *** cshen has quit IRC | 13:11 | |
| *** spatel has joined #openstack-ansible | 13:22 | |
| *** nurdie has joined #openstack-ansible | 13:27 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_ceilometer master: Updated from OpenStack Ansible Tests https://review.opendev.org/745842 | 13:56 |
| *** d34dh0r53 has joined #openstack-ansible | 13:57 | |
| noonedeadpunk | jrosser: sorry, I kind of missed what've decided regarding https://review.opendev.org/#/c/733443 | 13:59 |
| *** cshen has joined #openstack-ansible | 14:00 | |
| noonedeadpunk | should I make patches for the rest of the roles and are we good with this concept? | 14:00 |
| *** sshnaidm is now known as sshnaidm|MYG | 14:27 | |
| *** sshnaidm|MYG is now known as sshnaidm|mtg | 14:27 | |
| *** chkumar|rover is now known as raukadah | 14:45 | |
| *** gyee has joined #openstack-ansible | 14:58 | |
| *** cshen has quit IRC | 15:02 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Remove nova_api_placement from inventory https://review.opendev.org/746538 | 15:03 |
| *** cshen has joined #openstack-ansible | 15:05 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Remove nova_api_placement from inventory https://review.opendev.org/746538 | 15:05 |
| *** cshen has quit IRC | 15:09 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Return nova services reload https://review.opendev.org/746544 | 15:13 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Fix neutron-server default serial https://review.opendev.org/746546 | 15:28 |
| *** cshen has joined #openstack-ansible | 15:41 | |
| *** cshen has quit IRC | 15:47 | |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible master: Fix neutron-server default serial https://review.opendev.org/746546 | 16:01 |
| *** sshnaidm|mtg is now known as sshnaidm | 16:08 | |
| admin0 | so .. qemu-kvm is not installed by default my the playbooks :) | 16:24 |
| admin0 | by* | 16:24 |
| noonedeadpunk | on.. compute hosts? | 16:33 |
| openstackgerrit | Merged openstack/openstack-ansible-os_barbican master: Remove py2 packages https://review.opendev.org/746355 | 17:15 |
| *** cshen has joined #openstack-ansible | 17:42 | |
| *** cshen has quit IRC | 17:46 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible-os_barbican stable/ussuri: Remove py2 packages https://review.opendev.org/746574 | 17:48 |
| *** cshen has joined #openstack-ansible | 17:54 | |
| openstackgerrit | Charles Farquhar proposed openstack/openstack-ansible stable/rocky: Ensure that global-requirement-pins.txt is applied https://review.opendev.org/746580 | 18:05 |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone master: Use the utility host for db setup tasks https://review.opendev.org/733443 | 18:16 |
| *** jopdorp has joined #openstack-ansible | 18:27 | |
| jopdorp | Unregistered again? | 18:28 |
| jopdorp | This irc app is a bit unclear, but I think this is openstack-ansible right? | 18:29 |
| *** cshen has quit IRC | 18:32 | |
| openstackgerrit | amolkahat proposed openstack/openstack-ansible-os_tempest master: Added tempest ironic resources setup. https://review.opendev.org/720705 | 18:38 |
| *** miloa has joined #openstack-ansible | 18:52 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Remove mapping of gnocchi_dentity_only to swift https://review.opendev.org/739347 | 19:11 |
| *** spatel has quit IRC | 19:15 | |
| *** jbadiapa has quit IRC | 19:21 | |
| jrosser | jopdorp: yes, this is openstack-ansible | 19:23 |
| openstackgerrit | Merged openstack/openstack-ansible master: Fix UPGRADE_SOURCE_BRANCH https://review.opendev.org/746004 | 20:00 |
| openstackgerrit | Merged openstack/openstack-ansible-os_blazar master: Updated from OpenStack Ansible Tests https://review.opendev.org/745841 | 20:05 |
| *** miloa has quit IRC | 20:20 | |
| *** cshen has joined #openstack-ansible | 20:27 | |
| *** cshen has quit IRC | 20:31 | |
| *** cshen has joined #openstack-ansible | 22:27 | |
| *** cshen has quit IRC | 22:32 | |
| *** tosky has quit IRC | 22:57 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!