Wednesday, 2021-06-09

*** odyssey4me has quit IRC00:03
*** odyssey4me has joined #openstack-ansible00:12
*** spatel has joined #openstack-ansible01:42
*** spatel has quit IRC03:00
*** odyssey4me has quit IRC04:31
*** b1tsh1ft3r has joined #openstack-ansible05:42
*** b1tsh1ft3r has quit IRC05:50
*** luksky has joined #openstack-ansible05:56
noonedeadpunkand do we really need 2 managers?06:13
jrosseri was wondering if that means its setup to bind to two ports, 127.0.0.1:6640 and 0.0.0.0:664006:16
noonedeadpunkFrom what I got from the log - it probably is. But do we need to change ports or just make it listen only on one specific address we need rather then 0.0.0.006:21
jrossermore likley change the binding i think06:21
jrosserseems we're missing a diagram of all the parts here06:21
noonedeadpunkyep :(06:22
jrosserwell theres one on here https://blog.russellbryant.net/2016/09/29/ovs-2-6-and-the-first-release-of-ovn/06:23
jrosserwhat made me wonder about needing different ports is that there seem to be 3 instances of ovsdb-server which will all be on the same host for metal06:24
jrosseri'm not sure if they are listening servers or clients though06:24
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Use PKI role to install CA certificates  https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/79043106:28
noonedeadpunkoh, so we can actually omit defining transport_url for OVN?06:30
noonedeadpunkbut I don't really see how we come up with 3 ovsdb on the same host from diagrams...06:32
noonedeadpunkat least with server parts06:32
jrosserdont we have the northbound, southbound and local db for an AIO metal?06:33
jrosserthough i've never looked at any detail of this06:33
noonedeadpunkI guess you're right...06:33
noonedeadpunkI actually also wasn't looking into ovn at all :(06:34
noonedeadpunkthough it's super interesting and worth doing one day...06:34
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Add OS compatability matrix  https://review.opendev.org/c/openstack/openstack-ansible/+/78937606:36
jrosseroh well actually its here https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/providers/setup_ovs_ovn.yml#L23-L3606:38
noonedeadpunkoh, yeah... but we deploy it on the same hosts anyway?06:38
noonedeadpunkboth Southbound and Northbound.. and local..06:39
jrosserright so specifically this line https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/providers/setup_ovs_ovn.yml#L2406:41
jrosserthats binding to 0.0.0.0 which collide with the haproxy?06:41
jrosseryes https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/haproxy/haproxy.yml#L42206:42
jrosserthis code is 3 years old so is from before bind-to-mgmt and also from before we add haproxy to metal deployments06:46
*** odyssey4me has joined #openstack-ansible06:55
noonedeadpunkI wonder if we can set it to the ``ptcp:6641:{{ openstack_service_bind_address }} as well...07:01
noonedeadpunkAnd wonder why it hasn't been covered with https://github.com/openstack/openstack-ansible-os_neutron/commit/69845d63a023f58beef172a03e48400ed367174207:02
noonedeadpunkoh, btw...07:02
noonedeadpunkI wonder if spatel might be missing this patch07:03
noonedeadpunkas I think https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/providers/setup_ovs_ovn.yml#L24 might be binded to 127.0.0.107:03
noonedeadpunk(not sure at all)07:04
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Implement uWSGI for neutron-api  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/48615607:04
*** andrewbonney has joined #openstack-ansible07:16
*** tosky has joined #openstack-ansible07:53
opendevreviewJonathan Herlin proposed openstack/openstack-ansible-os_cloudkitty master: Cloudkitty role cleanup and config updates  https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/79419908:05
jonher^ added back examples/playbook.yml08:08
noonedeadpunkawesome, thanks!08:08
*** rpittau|afk is now known as rpittau09:22
*** b1tsh1ft3r has joined #openstack-ansible09:44
*** b1tsh1ft3r has quit IRC09:52
mgariepyspatel, jrosser : https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/providers/setup_ovs_ovn.yml#L2412:10
mgariepythis binds ovsdb to 0.0.0.0:664012:10
jrosseryeah, is that deliberate?12:10
mgariepyfound it yesterday but had an emergency come up so i didn't had time to post it haha.12:10
jrosserthe lines further down are binding to the mgmt ip12:11
mgariepyyep the deamon bind to both.12:11
jrosseroh well i mean that for the north/south stuff the IP is specified12:11
jrosserrather than unspecified12:11
mgariepyhttp://paste.openstack.org/show/806487/12:12
mgariepythat's from `cat /etc/openvswitch/conf.db12:13
mgariepy`12:13
*** b1tsh1ft3r has joined #openstack-ansible12:15
*** b1tsh1ft3r has quit IRC12:15
jrossermgariepy: do you think we should change that to bind to the mgmt network ip?12:15
jrosserit collides with haproxy for a metal deplotment12:15
mgariepyit binds to .100 also. so i think we should just not bind to 0.0.0.012:16
mgariepyi'll do a quick test.12:16
*** raildo has joined #openstack-ansible12:18
mgariepythe set-manager stuff is changed by spatel's patch.12:23
opendevreviewMarc Gariépy proposed openstack/openstack-ansible-os_neutron master: Add ovn clustering support  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/79426612:30
opendevreviewMarc Gariépy proposed openstack/openstack-ansible-os_neutron master: quicktest  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/79552812:30
mgariepyhttps://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/providers/setup_ovs_ovn.yml#L68 << this one is creating the bind to 127.0.0.112:34
admin1hi guys .. what would be the best way to do osa upgrade for a platform that is behind a lot of versions .. do  the .0 ( start) and .end  of each tag ?12:45
admin1each version i mean12:45
*** b1tsh1ft3r has joined #openstack-ansible12:45
mgariepyi would to the lastest for each.12:46
noonedeadpunkYou should jump only between major version12:46
noonedeadpunkand pick latest of each, yeah12:46
*** spatel has joined #openstack-ansible12:46
mgariepy.0 are somewhat always a bit buggy, so i would assume that the latest should work a bit better.12:47
admin1so if i am on say 19.x , i first go to end of 19.x .. then 20.0 .. and then 20.latest .. or can i jump from the last of each tag .. like 18.last, 19.last, 20.last, 21.last etc ?12:47
mgariepyif you are on 19.x, i would go to 20.last then 21.last etc..12:48
admin1ok12:49
*** b1tsh1ft3r has quit IRC12:54
*** b1tsh1ft3r has joined #openstack-ansible13:01
*** raildo has quit IRC13:06
*** gilou__ has quit IRC13:17
*** b1tsh1ft3r has quit IRC13:17
noonedeadpunkjrosser: hm, pki patches started failing after integrated one landed during upgrade13:18
noonedeadpunkhttps://zuul.opendev.org/t/openstack/build/01cbed83ca2f4075af83a39751684f08/log/job-output.txt#1838413:18
noonedeadpunkwhat's interesting is that /etc/openstack_deploy/pki/roots/ even doesn't exist13:19
noonedeadpunkOh, Create CA has been skipped because no hosts matched13:22
noonedeadpunkhttps://zuul.opendev.org/t/openstack/build/01cbed83ca2f4075af83a39751684f08/log/job-output.txt#1557413:24
noonedeadpunkwe don't have localhost?13:25
jrosseroooh - localhost becomes aio1 maybe?13:31
jnamdarit's called aio1 in ansible iirc13:32
noonedeadpunkbut isn't localhost smth really default that is always can be used...13:32
jrosseri thought that localhost == deploy host13:32
noonedeadpunkyeah, we intended to do that way13:32
jrosserhow is this working for the non upgrade cases them13:32
noonedeadpunkhttps://docs.ansible.com/ansible/latest/inventory/implicit_localhost.html13:34
jrosserhttps://github.com/openstack/openstack-ansible/blob/master/playbooks/certificate-authority.yml#L1613:34
noonedeadpunkyeah, I saw that and that `openstack_pki_setup_host` is not defined anywhere13:34
noonedeadpunkI just wonder if that's some bug from 2.10.10....13:36
noonedeadpunkbut again - it's working for non-upgrade...13:37
* noonedeadpunk tries to reproduce locally13:40
*** Gues_____ is now known as jamesdenton13:40
*** b1tsh1ft3r has joined #openstack-ansible13:48
*** b1tsh1ft3r has quit IRC13:56
*** b1tsh1ft3r has joined #openstack-ansible14:07
*** b1tsh1ft3r has quit IRC14:08
spatelI am trying to delete vm and not able to delete it look like not action happening, this is just 1 bad VM14:14
spatellook like a bug or something14:14
*** tosky has quit IRC14:16
*** tosky has joined #openstack-ansible14:18
jnamdarI switched to stable/victoria to install masakari and senlin in AIO. I think there are some missing deps for masakari python dep builds14:22
jnamdarI had to install pkg-config and libvirt-devel because libvirt-python wasn't building14:22
jnamdar(I run debian 10.9)14:22
jonher^ yes, i think there is already a launchpad issues about that14:26
jnamdarah yes indeed https://bugs.launchpad.net/openstack-ansible/+bug/188671914:27
opendevmeetLaunchpad bug 1886719 in openstack-ansible "masakari-monitors wheel building failed for libvirt-python" [Undecided,New]14:27
jnamdaris there any way I can tell I met the bug too or something on launchpad? I guess adding those apt dependencies would be fine to fix14:34
jnamdarI think I saw an old release note mentioning libvirt-python as well14:34
jonherwhen i brought it up in here last time i believe targeting was in questions and that on LXC AIO it tries to build in repo and not on host, in metal it should have the deps (but possibly from other roles)14:36
jonheri don't have enough scrollback, but iirc that was the conclusion at the time14:36
jrosserit would be nice to update the bug with this stuff rather than lose the scrollback14:39
jrosseri don't know if it's right or not to be trying to build libvirt-python from source or if that package should be symlinked into the venv from the host python environment14:40
jrosserwith libraries like that which have tightly coupled C bindings to an underlying thing (like libvirt) it needs to be handled carefully14:41
jrossersame happens with the ceph python bindings14:41
jrosserbuilding from source in the venv risks there being drift between the installed (and maybe upgraded) libvirt and the python binding14:42
jrosserif both come from distro packages then they should stay in lockstep14:43
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Fix upgrade jobs out of CI  https://review.opendev.org/c/openstack/openstack-ansible/+/79556314:44
noonedeadpunkjrosser: um... I was able to reproduce that... but... only when I'm inside run_upgrade.sh script14:48
noonedeadpunkI guess because of the limits https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/run-upgrade.sh#L175-L17914:49
noonedeadpunkhttp://paste.openstack.org/show/806492/14:50
noonedeadpunksounds familiar with https://github.com/ansible/ansible/issues/3822114:51
noonedeadpunkbut not really...14:51
jonherhere's the irclog from the masakari issue: https://meetings.opendev.org/irclogs/%23openstack-ansible/%23openstack-ansible.2021-03-03.log.html#t2021-03-03T11:30:1614:52
jonherthe log i pastebin'ed included the playbook run + some info from the wheels build log in repo container but that has expired now14:54
noonedeadpunkwell, this doesn't look related to what we see, but actually also an issue and good that you've reminded about it :)14:55
noonedeadpunkbut eventually masakari-monitors must be isntalled on the computes hosts, not LXC, so libvirt-python should be there at least for nova14:56
noonedeadpunkbut yeah, I think I saw smth like this on my own some time ago...14:57
noonedeadpunkWill try to build another aio and check things out14:57
jrosserwe should already be installing the distro version of that i think https://github.com/openstack/openstack-ansible-os_nova/blob/master/vars/debian.yml#L7314:58
jrosserand then for nova they are symlinked into the venv https://github.com/openstack/openstack-ansible-os_nova/blob/master/vars/debian.yml#L83-L8514:58
noonedeadpunkand so for masakari https://opendev.org/openstack/openstack-ansible-os_masakari/src/branch/master/vars/debian.yml#L3214:59
jrossermaybe something goes wrong here https://opendev.org/openstack/openstack-ansible-os_masakari/src/branch/master/tasks/main.yml#L7915:00
noonedeadpunkmight be...15:01
noonedeadpunkbut actually now I'm more concerned with localhost not being part of the limit15:01
noonedeadpunkunless, we put localhost inside dynamic_inventory I guess15:02
jrosseris it possible to add localhost to that limit?15:09
noonedeadpunknope15:09
noonedeadpunkhttp://paste.openstack.org/show/806494/15:09
*** miloa has joined #openstack-ansible15:10
noonedeadpunkand I even added localhost to dynamic_inventory...15:10
jrosserhow about calling the certificate_authority playbook seperatly in RUN_TASKS15:10
noonedeadpunkyeah, that's possible. But I can imagine this to fail in case of regular haproxy run?15:11
noonedeadpunkwhen you want to run it against single haproxy?15:11
*** ierdem has joined #openstack-ansible15:11
ierdemHello everyone, I have Centos7 Cloud image on my OSA-Victoria env. When I try to launch VM with this image by using flavor which has 6TB disk, it only mount 2 TB to root partition. Is there a limit on Centos images? I tried to launch Ubuntu image with same flavor, it mounted all of 6TB to VM. can you help me please? Thanks15:12
opendevreviewMarc Gariépy proposed openstack/openstack-ansible-tests master: add openvswitch to the log collect  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/79556815:12
mgariepythis might be useful for debug ^^15:12
mgariepyovn/ovs stuff.15:12
noonedeadpunkierdem: In case FS is in ext3 - it would be 2Tb max15:13
noonedeadpunkhttps://wiki.centos.org/About/Product15:13
noonedeadpunk(I guess I misread this)15:14
noonedeadpunkmgariepy: we collect logs with https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/log-collect.sh15:14
noonedeadpunkfor integrated jobs15:15
mgariepylol15:15
mgariepyarf what is the other used for ?15:15
noonedeadpunkfor functional ones15:16
noonedeadpunkbut almost none of them left15:16
opendevreviewMarc Gariépy proposed openstack/openstack-ansible master: add openvswitch to the log collect  https://review.opendev.org/c/openstack/openstack-ansible/+/79557215:17
mgariepyabandoning the openstack-ansible-tests one .15:18
jrossernoonedeadpunk: can you try putting RUN_TASKS+=("certificate-authority.yml") here https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/run-upgrade.sh#L17415:21
jrosserwith no --limit15:21
noonedeadpunkyes, it will worlk15:21
miloanoonedeadpunk quick feedback about our yesterday problem on multiregion, adding region_name in section neutron of nova.conf.j2 and in nova section of neutron.conf.j2 solves all the problem we had, this "field" was present in rocky but disapear in train.15:21
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova stable/train: config: add region_name to neutron  https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/79546715:24
noonedeadpunkmiloa: uh, and it has been fixed in U...15:25
miloanoonedeadpunk : ok thanks :) We are still in train :)15:26
noonedeadpunkand you had to add it to neutron as well?15:26
miloayes15:26
noonedeadpunkor you added as pre-cautious step?15:26
ierdemnoonedeadpunk, how can I set/change initial FS of Centos cloud image? Do you have any idea?15:27
jrosserierdem: it's an image of a block device, the filesystem is baked in from the moment the image is created by the centos people15:28
noonedeadpunkum, iirc there's some file inside fs that's resposible of that... like /etc/mke2fs.conf15:28
jrossersurely that cant be changed?15:28
miloain nova.conf.j2 it resolves an error of not finding "default security group" and on neutron.conf.j2 it solves a problem on cconecting instance to networks which was unable to find.15:29
noonedeadpunkI think I was changing it for CentOS 7 by removing `extra_isize` for ext4 at least as it was failing otherwise....15:29
noonedeadpunkBut looking into it now, I think I changed default FS some other way...15:29
noonedeadpunkBut I was repacking images, so yeah15:30
jonher7.1907 does xfs, might not be on older 7.x cloud images15:31
noonedeadpunkoh, I even was removing `64bit` :)15:32
noonedeadpunkand it was centos 6.5, so disregard :)15:33
jonher7.5 1804  is the oldest image i have of centos7, and that too did xfs15:33
noonedeadpunkyeah, I have no idea, just assumed what it could be...15:34
jonherierdem how did you verify the block devices size inside the OS? what does lsblk or parted show?15:35
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron stable/train: config: add region_name to nova  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/79546815:35
ierdemjonher, lsblk shows /dev/vda as 6TB and under this /dev/vda1 is 2 TB15:37
noonedeadpunkwonder if that might be some growpart thing15:37
noonedeadpunk*growroot15:37
jonhermount | grep '/ '   to check what fs is used, likely xfs then15:38
jonheryeah cloud-init might not have handled grow correctly, possibly some trace of that in logs15:38
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Run certificate-authority independently during upgrade  https://review.opendev.org/c/openstack/openstack-ansible/+/79557915:44
jrosseri wonder if we will come across other certificate related trouble with --limit as we start to use it more in the deployment15:51
*** ierdem has quit IRC15:53
noonedeadpunkyeah, that what I was afraid about15:54
jrossereven if we make the deploy host actually be in the inventory to avoid relying on implicit localhost i think that wont help?15:56
jnamdarzz I got an endpoint type problem again lol15:56
jnamdarsenlin seems to want to use keystone's public endpoint at all costs15:57
jrosserthough the tasks we care about are delegated to the deploy host, which is a little different to how things work in the certificate-authority.yml playbook15:57
jrosserjnamdar: senlin needs fixing, unfortunately :(15:57
jnamdarwhich doesn't work for some reason. i had to modify keystone's public endpoint with the internal url15:57
jonherierdem https://bugs.launchpad.net/ubuntu/+source/cloud-utils/+bug/1259703 it seems to be a bug, i tried latest growpart from centos7 package and that says "partition 1 is size 4294965248. it cannot be grown" so looks like you need to do some manual work there15:58
opendevmeetLaunchpad bug 1259703 in cloud-utils (Ubuntu Saucy) "growpart on disk larger than 2TB fails" [Medium,Fix released] - Assigned to Scott Moser (smoser)15:58
*** ierdem has joined #openstack-ansible15:58
jonherhttps://github.com/openstack/senlin/blob/3b0f21972be0bb067e8c4391a6b77aa8815a0ca2/senlin/tests/unit/drivers/test_keystone_v3.py#L190 looks like senlin overrides in tests16:00
jonherhttps://review.opendev.org/c/openstack/senlin/+/74987416:02
ierdemjonher, it's xfs16:07
jonheryeah i started a machine on my end and tested, the link to the bug is posted above with your name mentioned16:08
*** ioni has quit IRC16:13
noonedeadpunkjrosser: yeah, so issues might be if somebody decide to run setup-hosts.yml with limit, but once CA is there, it won't matter16:13
*** rpittau is now known as rpittau|afk16:16
*** miloa has quit IRC16:39
opendevreviewMerged openstack/openstack-ansible master: Add Debian Bullseye support  https://review.opendev.org/c/openstack/openstack-ansible/+/78360617:02
*** andrewbonney has quit IRC17:05
opendevreviewMarc Gariépy proposed openstack/openstack-ansible-os_neutron master: quicktest  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/79552817:51
*** jnamdar has quit IRC18:15
noonedeadpunkjrosser: mgariepy: what do you think about https://review.opendev.org/c/openstack/openstack-ansible/+/795579 ? To fix pki upgrades at least for CI?18:43
noonedeadpunkwe also need https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/795282 and https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/795283 for stable branches18:46
jrosserI think it’s reasonable - do we need to add it to the manual upgrade documentation too?18:46
noonedeadpunkgood point18:47
jrosserthat can be a separate patch of course so we don’t have to run the other one again18:49
* noonedeadpunk writing18:50
noonedeadpunkmight be follow-up will be good indeed so re-check could be issued faster for rabbit and haproxy roles18:51
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Document CA generation upgrade step  https://review.opendev.org/c/openstack/openstack-ansible/+/79566519:02
noonedeadpunksummoning spotz for checking my awful language ^ :)19:03
noonedeadpunkand all previous patches as we ll :)19:03
noonedeadpunkbtw I believe https://review.opendev.org/c/openstack/openstack-ansible/+/789376 is also good to go...19:04
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Update distro mirrors to match release  https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/79567019:17
spotznoonedeadpunk: OOn it19:30
opendevreviewAmy Marrich proposed openstack/openstack-ansible master: [doc] Add OS compatability matrix  https://review.opendev.org/c/openstack/openstack-ansible/+/78937619:33
spotzOm https://review.opendev.org/c/openstack/openstack-ansible/+/789376 I just moved libvirt example up a line jrosser had already +2'd so if no one disagrees I'll speed it through if it passes the gate19:34
opendevreviewAmy Marrich proposed openstack/openstack-ansible master: [doc] Document CA generation upgrade step  https://review.opendev.org/c/openstack/openstack-ansible/+/79566519:38
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible stable/ocata: Switch Ocata to EOL  https://review.opendev.org/c/openstack/openstack-ansible/+/79569020:15
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible stable/ocata: Switch Ocata to EOL  https://review.opendev.org/c/openstack/openstack-ansible/+/79569020:21
opendevreviewMerged openstack/openstack-ansible-lxc_hosts stable/victoria: Add CentOS 8.4 support  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/79528220:22
opendevreviewMerged openstack/openstack-ansible-lxc_hosts stable/ussuri: Add CentOS 8.4 support  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/79528320:22
*** spatel has quit IRC20:42
*** ierdem has quit IRC20:53
*** luksky has quit IRC21:58
*** luksky has joined #openstack-ansible22:00
*** djhankb has quit IRC22:00
*** djhankb has joined #openstack-ansible22:00
opendevreviewMerged openstack/openstack-ansible master: Run certificate-authority independently during upgrade  https://review.opendev.org/c/openstack/openstack-ansible/+/79557922:02
*** tosky has quit IRC22:47
*** luksky has quit IRC23:43

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!