Wednesday, 2021-02-24

*** tosky has quit IRC00:10
*** dmsimard0 has joined #openstack-ansible00:50
*** mgariepy has quit IRC00:51
*** dmsimard has quit IRC00:51
*** dmsimard0 is now known as dmsimard00:51
*** mgariepy has joined #openstack-ansible00:52
*** maharg101 has joined #openstack-ansible01:22
*** maharg101 has quit IRC01:26
*** spatel has joined #openstack-ansible01:29
*** LowKey has quit IRC01:44
*** fridtjof[m] has quit IRC01:54
*** fridtjof[m] has joined #openstack-ansible01:54
*** macz_ has quit IRC02:04
*** zul has quit IRC02:08
*** rh-jelabarre has quit IRC03:01
*** macz_ has joined #openstack-ansible03:05
*** macz_ has quit IRC03:10
*** maharg101 has joined #openstack-ansible03:22
*** macz_ has joined #openstack-ansible03:26
*** maharg101 has quit IRC03:27
*** macz_ has quit IRC03:31
*** macz_ has joined #openstack-ansible04:06
*** macz_ has quit IRC04:10
*** macz_ has joined #openstack-ansible04:27
*** macz_ has quit IRC04:32
*** macz_ has joined #openstack-ansible04:48
*** macz_ has quit IRC04:52
openstackgerritSatish Patel proposed openstack/openstack-ansible-os_senlin master: Allow Enable/Disable feature of uWSGI support  https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/77729604:55
*** macz_ has joined #openstack-ansible05:09
*** macz_ has quit IRC05:13
*** maharg101 has joined #openstack-ansible05:23
*** maharg101 has quit IRC05:28
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-ansible05:33
*** gyee has quit IRC05:34
*** spatel has quit IRC05:42
*** djhankb has quit IRC06:26
*** jbadiapa has quit IRC06:45
*** macz_ has joined #openstack-ansible06:47
*** miloa has joined #openstack-ansible06:49
*** miloa has quit IRC06:51
*** macz_ has quit IRC06:52
*** ioni has joined #openstack-ansible06:55
*** SiavashSardari has joined #openstack-ansible07:06
*** macz_ has joined #openstack-ansible07:08
*** macz_ has quit IRC07:12
*** sshnaidm|afk is now known as sshnaidm07:38
*** ahsen has joined #openstack-ansible07:41
*** luksky has joined #openstack-ansible07:54
*** pto has joined #openstack-ansible08:03
*** ahsen has quit IRC08:04
*** pto has quit IRC08:08
*** rpittau|afk is now known as rpittau08:11
*** pto has joined #openstack-ansible08:13
openstackgerritSlawek Kaplonski proposed openstack/openstack-ansible-os_tempest stable/stein: Add provider_physical_network option to private network  https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/77730908:14
openstackgerritSlawek Kaplonski proposed openstack/openstack-ansible-os_tempest stable/rocky: Add provider_physical_network option to private network  https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/77712908:15
openstackgerritSlawek Kaplonski proposed openstack/openstack-ansible-os_tempest stable/queens: Add provider_physical_network option to private network  https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/77731108:18
*** pto has quit IRC08:18
*** maharg101 has joined #openstack-ansible08:20
*** andrewbonney has joined #openstack-ansible08:27
zigojrosser: You will *not* be able to run wallaby on Buster, I would strongly suggest switching it to Bullseye at this point.08:29
zigoEven for Victoria, I have my own set of backports for Qemu and so on.08:30
zigoI will not do the work for Wallaby, as I strongly believe it should be running on Bullseye.08:30
jrosserzigo: are there infra CI images for bullseye?08:30
zigojrosser: I don't think so, but it shouldn't be hard to add one.08:31
zigoIf you didn't know, Bullseye is already in soft freeze (meaning, no new upstream version will get in...).08:31
*** pto has joined #openstack-ansible08:35
zigojrosser: Can you do the infra patch to add bullseye ? I can review it...08:36
*** pto has quit IRC08:39
jrosserzigo: possibly - though likley not able to look at that till next week08:41
zigojrosser: Just ping me if you need then.08:41
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_senlin master: Use global service_install_method for senlin  https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/77731508:44
*** macz_ has joined #openstack-ansible08:44
openstackgerritAndrew Bonney proposed openstack/openstack-ansible-os_neutron master: Combined patch to unblock CI  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/77523908:48
*** macz_ has quit IRC08:49
openstackgerritAndrew Bonney proposed openstack/openstack-ansible-os_neutron master: Combined patch to unblock CI  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/77523908:51
*** miloa has joined #openstack-ansible08:51
*** miloa has quit IRC09:12
*** tosky has joined #openstack-ansible09:18
admin0morning \o09:37
openstackgerritMerged openstack/openstack-ansible-apt_package_pinning master: Use ansible_facts[] instead of fact variables  https://review.opendev.org/c/openstack/openstack-ansible-apt_package_pinning/+/77695709:47
openstackgerritMerged openstack/openstack-ansible-haproxy_server master: Use ansible_facts[] instead of fact variables  https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/77706509:49
openstackgerritJonathan Rosser proposed openstack/ansible-role-systemd_networkd master: Use ansible_facts[] instead of fact variables  https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/77695509:53
*** macz_ has joined #openstack-ansible10:07
*** macz_ has quit IRC10:12
openstackgerritEbbex proposed openstack/openstack-ansible-os_horizon master: Add hide_create_volume option to defaults  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/77735010:17
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Use ansible_facts[] instead of fact variables  https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/77696010:21
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-rabbitmq_server master: Use ansible_facts[] instead of fact variables  https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/77707110:25
openstackgerritMerged openstack/openstack-ansible-lxc_hosts master: Remove per-backend container creation tasks files  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/77623910:35
*** lemko has quit IRC10:44
*** lemko has joined #openstack-ansible10:48
*** lemko has quit IRC10:50
noonedeadpunkok, so we've got problem with devbian upgrades along with centos ones as well...11:08
openstackgerritMerged openstack/openstack-ansible-lxc_hosts master: Update Centos-8 LXC image to 8.3  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/77577811:13
*** macz_ has joined #openstack-ansible11:13
openstackgerritMerged openstack/ansible-role-systemd_networkd master: Add reference to gpg key location to EPEL repo config  https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/77622511:18
*** macz_ has quit IRC11:18
openstackgerritMerged openstack/ansible-role-systemd_mount master: Use ansible_facts[] instead of fact variables  https://review.opendev.org/c/openstack/ansible-role-systemd_mount/+/77696211:20
*** akahat has quit IRC12:05
*** akahat has joined #openstack-ansible12:05
*** dirtwash has left #openstack-ansible12:13
*** SiavashSardari has quit IRC12:27
*** miloa has joined #openstack-ansible12:43
*** miloa has quit IRC12:44
*** mgariepy has quit IRC13:00
*** macz_ has joined #openstack-ansible13:01
*** macz_ has quit IRC13:06
*** rh-jelabarre has joined #openstack-ansible13:20
*** mgariepy has joined #openstack-ansible13:28
*** zul has joined #openstack-ansible13:41
*** spatel has joined #openstack-ansible13:56
spatelnoonedeadpunk does this looks good to you? - https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/77729613:58
*** macz_ has joined #openstack-ansible13:58
noonedeadpunkyep, except part that senlin does not really test distro jobs atm14:02
*** macz_ has quit IRC14:02
*** rpittau is now known as rpittau|afk14:04
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible master: Use proper distro path for senlin  https://review.opendev.org/c/openstack/openstack-ansible/+/77738414:09
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_senlin master: Use global service_install_method for senlin  https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/77731514:10
*** janno_ has quit IRC14:12
*** janno has joined #openstack-ansible14:13
*** LowKey has joined #openstack-ansible14:17
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_container_create master: Use ansible_facts[] instead of fact variables  https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/77695814:19
spatelnoonedeadpunk what do you mean by distro jobs?14:19
spateljust noticed senlin_install_method: distro.14:20
spatelRPM/APT get based deployment right?14:20
noonedeadpunkyeah14:20
noonedeadpunkas eventually these jobsare failing atm14:21
spatelyeah!14:21
openstackgerritJonathan Rosser proposed openstack/ansible-role-uwsgi master: Use ansible_facts[] instead of fact variables  https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/77738614:23
openstackgerritJonathan Rosser proposed openstack/ansible-role-systemd_networkd master: Use ansible_facts[] instead of fact variables  https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/77695514:26
*** spatel has quit IRC14:36
*** luksky has quit IRC14:36
*** juanoterocas has joined #openstack-ansible14:41
*** LowKey has quit IRC14:41
*** LowKey has joined #openstack-ansible14:41
*** lemko has joined #openstack-ansible14:46
*** macz_ has joined #openstack-ansible14:53
*** macz_ has quit IRC14:57
*** chandankumar has quit IRC14:58
*** admin0 has quit IRC15:10
*** spatel has joined #openstack-ansible15:18
*** jbadiapa has joined #openstack-ansible15:35
*** spatel has quit IRC15:39
*** admin0 has joined #openstack-ansible15:47
*** macz_ has joined #openstack-ansible16:15
*** juanoterocas has quit IRC16:19
*** gyee has joined #openstack-ansible16:41
*** luksky has joined #openstack-ansible17:02
*** gshippey has joined #openstack-ansible17:24
*** mgariepy has quit IRC17:33
*** ierdem has joined #openstack-ansible17:39
ierdemHi everyone, I have OSA-Ussuri cluster with 2 infra nodes. Actually it was 3 but one of them has corrupted. After this corruption rabbitmq services started to give errors then removed all rabbitmq containers, I edit openstack_user_config.yaml file and I installed rabbitmq to just only second infra node, so it works standalone now. After this17:47
ierdeminstallation I tested my cluster, my old instances work fine but I can not create new instance. When I check nova-api-container I see these errors in nova-conductor journal http://paste.openstack.org/show/802972/. In rabbitmq logs everything seems fine http://paste.openstack.org/show/802973/. Can you help me about this problem, please?17:47
jrosserierdem: have you re-run all the playbooks for the openstack services?17:51
ierdemI can ping rabbitmq container from other containers, and also I can telnet on 5671 port17:54
jrosserthis is an example of how the config files for each service are set up https://github.com/openstack/openstack-ansible-os_keystone/blob/master/templates/keystone.conf.j2#L2717:58
jrosserwhatever hosts are defined in the ansible inventory for rabbitmq are templated into the .conf files for everything17:58
jrosserso if you still have the two hosts that no longer run rabbit in the inventory you are going to get two bad IP put in there17:59
*** maharg101 has quit IRC18:00
jrosseri think that this can be overridden globally for your deployment by setting `oslomsg_rpc_servers`in user_variables if you don't want to edit the broken rabbit hosts out of the inventory18:01
*** mgariepy has joined #openstack-ansible18:11
*** spatel has joined #openstack-ansible18:31
spatelnoonedeadpunk assuming OSA has some short of OVN support18:32
spatelI may start playing with OVN + DRV deployment18:32
*** ioni has quit IRC18:35
noonedeadpunkwell. we have some ort of it:)18:36
spatelnoonedeadpunk cool, lets try to make it stable :)18:39
spatelI am going to start playing to see how things work out.18:39
spatelI am looking for DPDK+DVR+OVN solution18:40
*** andrewbonney has quit IRC18:42
mgariepyany issue of upgrading U > V ?18:48
mgariepyovn and dvr it's kinda included by design ;)18:48
noonedeadpunkwell I upgraded 2 regions by far and things were pretty smooth for second one18:49
noonedeadpunkwas going to release 22.1.0 with https://review.opendev.org/c/openstack/openstack-ansible/+/77552118:49
noonedeadpunkoh, well, if you have barbican , see https://review.opendev.org/c/openstack/openstack-ansible/+/775856/118:49
mgariepyno i don't18:50
ierdemjrosser I set ansible inventory and I checked services' config files, there is correct rabbitmq parameters in these files. The weird thing is just only nova services throws error18:50
noonedeadpunkoh, yes, and rabbitmq hosts file ofc18:50
*** maharg101 has joined #openstack-ansible18:53
ierdemjrosser btw, yes I ran all playbooks18:59
*** gregwork has joined #openstack-ansible19:06
ierdemHow can I sure the nova-api services try to reach correct rabbitmq?19:26
*** rh-jelabarre has quit IRC19:27
*** rh-jelabarre has joined #openstack-ansible19:30
mgariepynoonedeadpunk, rabbitmq host file?19:45
noonedeadpunkyeah, inside of the rabbitmq containers, after running setup-hosts and before setup-infrastructrue check out for records for 127.0.1.1 in /etc/hosts19:46
noonedeadpunkthey could go super weird making cluster to break19:46
jrosserierdem: you would check the nova.conf  to make sure it only lists the working rabbit host19:47
mgariepyok19:47
ierdemjrosser I checked the nova.conf files, they have correct rabbit host but still throws 'EHOSTUNREACH" error and it does not appear which host is unreachable, there is no IP or hostname in journals about this error19:51
jrossertcpdump might be your next step19:52
jrosseryou can also put debug=true in the nova.conf for more verbose logging19:53
*** pto has joined #openstack-ansible20:02
*** pto has quit IRC20:06
*** sshnaidm is now known as sshnaidm|afk20:28
ierdemjrosser I set debug=true but still does not appear IP or hostname in journals, how can I try tcpdump? I didn't use it before, is there any command to check traffic that you can tell me?20:30
admin0ierdem..  use only 1 rabbit at a time, restart.. see it connectes20:35
admin0you have just 320:35
admin0so either all will not work .. or one of them might not work20:36
admin0and you will get your answer20:36
*** sahilA has joined #openstack-ansible20:40
sahilAhi20:41
*** LowKey has quit IRC20:41
ierdemadmin0 I just 1 rabbit on my cluster, I have 2 nodes now, infra1 and infra3. Rabbit container is running on infra3 node and it seems work fine20:41
*** LowKey has joined #openstack-ansible20:41
admin0so you are saying you are getting "they have correct rabbit host but still throws 'EHOSTUNREACH" error "   out of the blue and it can connect fine ?20:42
*** juanoterocas has joined #openstack-ansible20:42
admin0did you run the full playbooks ?20:43
admin0try running openstack-ansible setup-everything and see if it runs without issues20:43
ierdemyes, exactly. I ran all of them step by step20:43
ierdemother services are not throwing errors except nova-*20:44
admin0see, you had 3 nodes .. so unless you remove the rabbitmq entry from inventory and config, it will still be 3 entries for nova20:44
ierdemI can connect rabbit container from nova containers, I can telnet,ping etc..20:45
admin0and maybe its tries to connect, gives an error on the non working one and connects to the right one20:45
admin0it could just be an  error message only20:45
admin0and nova actually works fine20:45
sahilAcan anyone help me with openstack-ansible aio? i installed everything and launched instances via horizon just fine but then i lost power and when i try to restart i keep getting haproxy backend service has no server available error20:45
admin0sahilA, aio  means you can redo it again20:45
admin0unfortunately20:46
admin0if you shut it down and try to bring it up, it will give issues20:46
admin0its meant to be used once and throw away kind20:46
sahilAoh damn20:46
admin0because during aio .. there are loopback devices created and made for ceph, cinder etc  which does not do well even if there was no power loss, but you tried to run the full playbooks again20:46
ierdemadmin0 nop, I have 2 nodes, I cleaned and reconfigure inventory file and nova is not running fine because I cannot create instance. When I try to launch an instance, it stucks on scheduling step20:47
admin0ierdem, can u backup the nova.conf, edit it ,  try to use only 1 connection to rabbitmq and restart the process/container and see if it connects20:48
sahilAadmin0 thanks. what else is the easiest way to setup openstack that isnt crash and burn?20:48
admin0if it cannot, and you are 100% sure its the correct ip, user,pass .. then its not a nova issue .. it could be firewall, mtu, or something else20:49
admin0can u tcpdump in the host with rabbitmq and see the connection attempt from the container ?20:49
admin0sahilA, i would suggst taking a vps with a cloud provider or a small dedi server20:49
admin0or if you have a production cloud, make one vps for yourself named aio20:49
*** spatel has quit IRC20:52
sahilAi have a dedicated server and want to setup something i can provision vms with. I thought i would give openstack a try.20:53
jrosserthe AIO is just really a test fixture20:53
admin0sahilA, in that case, do not use an AIO .. based on how big the server is, the server itself can be a controller and a qemu/kvm can be compute .. or the other way around20:54
admin0my test bed is usually a big server with multile virtual kvm machines acting as controllers, ceph, compute etc20:54
admin0and i use the same way as openstack .. boot the vms using focal as a backup file ( like openstack does) so bringing up a new instance is less than 30 seconds20:54
*** maharg101 has quit IRC20:54
openstackgerritMerged openstack/openstack-ansible-os_zun master: Reinstate voting for upgrade jobs  https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/77556320:55
admin0sahilA, are the VMs for production/users ?20:55
jrosseryou could easily make something like the AIO in a single server but using real storage rather than loop devices20:55
sahilAno right now its just for me and my friends but ideally i would like to make it production ready20:56
jrosserthe AIO networking is also strange as it’s all nat behind a single IP, which is not normal at all for a real cloud20:56
sahilAi would have to do something similar tho? as i would only have 1 static ip20:57
admin0yep ..20:57
sahilAdo any of the guides/docs cover a single node setup?20:58
ierdemadmin0 I ran "tcpdump -i eth1 port 5672" command both nova and rabbit containers, rabbitmq output is http://paste.openstack.org/show/802985/, nova is http://paste.openstack.org/show/802986/21:08
ierdemIt seems they can communicate with each other21:08
*** poopcat1 is now known as poopcat21:11
admin0so looks like they are communicating fine21:16
admin0can u login to rabbit and see the nova client listed on any queue ?21:16
*** sahilA has quit IRC21:16
admin0you have to do some port forwarding etc stuff to reach the admin port , login as admin to see the queues and the clients listening to the queue21:16
admin0or do some nginx reverse proxy thing to expose rabbitmq admin port to what you can access21:17
admin0that is the next step .. if you don't see nova client at all , then you may have to nuke all your rabbit container and re-run the playbooks so that its re-created from scratch21:17
ierdemI can see /nova when I run rabbitmqctl list_vhosts21:20
*** NewJorg has quit IRC21:49
*** NewJorg has joined #openstack-ansible21:55
*** devtolu1__ has joined #openstack-ansible21:56
*** juanoterocas has quit IRC21:58
ierdemadmin0 There are some logs, can you check that please http://paste.openstack.org/show/802987/. These IPs are not used, so they cannot be reached from somewhere. When I search these IPs in containers, it is empty set but on infra01 node these are consist in /etc/openstack_deploy/ansible_facts/ dir. Does it cause this problem?21:59
*** gshippey has quit IRC22:06
*** rh-jelabarre has quit IRC22:08
openstackgerritBjoern Teipel proposed openstack/openstack-ansible-os_ironic stable/ussuri: Fixing minor issue when no inspector is deployed  https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/77747522:13
*** devtolu1__ has quit IRC22:19
*** juanoterocas has joined #openstack-ansible22:19
openstackgerritBjoern Teipel proposed openstack/openstack-ansible-os_ironic stable/ussuri: Fixing minor issue when no inspector is deployed  https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/77747522:24
CeeMacierdem: you'd probably be better off running tcpdump against the rabbit port then trying to perform an action in nova and see what destination ip is being targeted22:30
CeeMac*port number22:30
CeeMacThen you can track back to see if that is going to the correct node22:31
*** djhankb has joined #openstack-ansible22:31
*** luksky has quit IRC22:39
ierdemCeeMac like what admin0 said, I am running setup-eveything.yml now, after this I will try that, thank you22:43
CeeMacSure, good luck :)22:44
*** maharg101 has joined #openstack-ansible22:51
*** maharg101 has quit IRC22:56
*** juanoterocas has quit IRC23:13
*** ierdem has quit IRC23:47

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!