Friday, 2024-03-15

opendevreviewMerged openstack/openstack-ansible-os_nova master: Ensure nova_device_spec is templated as JSON string  https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/91324810:24
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron stable/2023.2: Use ansible_facts['processor_vcpus'] instead of fact variable  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/91275310:39
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron stable/2023.1: Use ansible_facts['processor_vcpus'] instead of fact variable  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/91275410:41
gebz_New to IRC, how do i connect to this channel via hexchat?10:49
gebz97Anybody here?10:52
noonedeadpunko/11:01
noonedeadpunkgebz: hey11:02
gebzwhat's up11:02
gebznew here11:02
noonedeadpunksure - feel free to fire your questions :) ppl are mostly around here during EU business hours though. But we can be around quite late to catch up with EST as well :)11:03
gebzIt's ok, I'm GMT+211:03
gebzI mostly have a general idea what openstack's architecture looks like, but i'm really struggling to get wrap my head around neutron11:05
noonedeadpunkoh, well :D you're not the only one for sure11:05
gebz:D 11:05
noonedeadpunkimo, networking is really the hardest topic for me as well11:05
gebzSimply put, do the neutron networks only exist as a tunneled network inside openstack? (i.e. they require BGP peering with data center core network to be visible?)11:06
gebzor can they exist as vlans on the DC network?11:07
noonedeadpunkI guess it depends on the type of network11:08
gebzspine-leaf 11:08
noonedeadpunkIf we're talking about tenant private networks (east-west traffic) - they exist only as vxlan or geneve tunnels (depending on the driver)11:09
jrosser(you can also do those as vlans if you want to)11:09
noonedeadpunkOr well. They can be also vlans in theory, though vlans don't scale in a good way, so usually they're not used for that11:09
gebzwhat about the openstack infrastructure hosts?11:10
noonedeadpunkbut no, there's no requirement of having BGP visible for any networks11:10
noonedeadpunkLike you can do just fine with vlans for north-south just fine11:10
gebzbut the switch ports that the physical hosts are connected to need to be trunked correct?11:10
noonedeadpunkwell, yes, that's how vlans work?11:11
noonedeadpunkthis is not required for vxlan/geneve though11:11
noonedeadpunkas it's tunneled traffic 11:12
gebzsame as kubernetes pod network/service network right?11:12
noonedeadpunkor well, again I guess depends11:12
noonedeadpunkI think pretty much yes.11:12
noonedeadpunkSo you basically set IP/network which will be used for the tunneled traffic11:13
noonedeadpunkand neutron ensures to create vxlan/geneve on top of that11:13
noonedeadpunkbut that's about east-west more or less11:13
jrosserthose tunneled networks will go just fine over your spine/leaf between computes and infra (specifically neutron l3 agent or OVN gateway nodes)11:13
noonedeadpunkfor north-south you'd need either just vlan, or indeed go with bgp solution11:13
jrosseryou need to consider separately what is going to happen for external “provider”11:14
jrossernetworks that might be l2 constructs and won’t natively get across that spine/leaf11:14
noonedeadpunkwell, there're ways to make them....11:14
jrosserindeed, but this is an important decision11:15
noonedeadpunk++11:15
jrosserlike you can dedicate a leaf to the control plane11:15
gebza leaf switch for the openstack infra?11:15
jrosseror you can use vxlan on the network hardware to span these things where you need them11:15
* noonedeadpunk had a partial fiasco now with ovn-bgp-agent11:15
gebzlike a top-of-rack?11:15
jrossergebz: so for example, in my largest deployment I have a pair of switches dedicated to the control plane11:16
jrosserbut that’s just because the design I have needed that number of ports11:16
opendevreviewMerged openstack/openstack-ansible master: Bump ansible version to 2.15.9  https://review.opendev.org/c/openstack/openstack-ansible/+/90561911:16
gebzbut those 2 switches is where for example the management network would live, correct?11:17
gebzalso the storage network11:18
jrosserthe management network has to go to all the nodes11:18
gebzOk, the network team's gonna be more involved in this than I originally thought :'D11:18
jrosserso your choice would be to have it be l3 on your leaf/spine11:18
jrosseror you do it l2 with evpn or something11:19
jrosseror of course l3 inside evpn is also possible11:19
jrosserI think that what I’m saying is the logical architecture is important to tie down rather than just looking at the pile of hardware and deciding how to plug it up11:19
gebzRight now I'm sort of in a PoC phase.. trying to pitch openstack to our CTO, doing the deployment inside of proxmox11:20
gebzWe're 97% VMWare atm11:20
jrosserobviously everything I have said is entirely dependant on what scale you might want to achieve11:20
jrossersmaller things can be much much simpler11:21
gebz3 physical hosts with proxmox11:21
gebz120 vCPUs, 0.5 TiB ram, ~200 TiB storage (40TiB NVME/160TiB HDD)11:22
gebztotal11:22
gebzThose are the boxes i'm working with rn11:22
jrosserwell “I want to build a small openstack and already have a spine/leaf network” is not the same as “I want to build an openstack so large that a spine/leaf network is the only way”11:23
gebzall are on the same vlan, connected to the same top of rack11:23
gebzno, i meant there was already a spine/leaf in our data center11:23
gebzMust have misunderstood :D11:23
gebzIs there any real difference if I have the infra hosts as VMs instead of on bare metal?11:29
noonedeadpunkno, not really, except performance conciderations11:31
noonedeadpunkeventually, you can also split each component to it's own VM easily11:31
gebznot too big an issue since it would be a PoC11:31
noonedeadpunkyou can actually just spawn everything in a 1 single VM including ceph, through AIO build11:32
gebzThat sounds interesting actually..11:33
gebzIn that case, does that chunky boi needs to have 6x VNICs?11:34
gebzwith 6 IP addresses?11:34
noonedeadpunkhttps://docs.openstack.org/openstack-ansible/latest/user/aio/quickstart.html11:34
noonedeadpunkNo, just 1 VM with 16gb of ram, 4-8 vcpus and 100gb of disk11:34
noonedeadpunk1 nic is fine11:34
noonedeadpunkrest will be "dummied"11:34
noonedeadpunkactually, there's even eaasier way of doing All-In-One then in the doc...11:35
gebzTell me11:35
noonedeadpunkbasically - git clone https://opendev.org/openstack/openstack-ansible; cd openstack-ansible; ./scripts/gate-check-commit.sh aio_lxc_ceph`11:35
gebzlol that simple?11:36
noonedeadpunkyou can add any service to the list of `aio_lxc_ceph`, ie `aio_lxc_ceph_magnum_trove_ceilometer`11:36
noonedeadpunkyou can also replace `lxc` with `metal` if you don't wanna have LXC containers11:36
gebzI would be lying if I said I understand LXC containers either, mostly worked with docker and k8s11:37
noonedeadpunkthey're just system containers11:37
noonedeadpunknot application ones like docker11:37
noonedeadpunkso consider them as a very lightweight VM with a shared kernel11:38
noonedeadpunkLike OpenVZ :D11:38
gebzF**k it imma do it right now11:38
noonedeadpunkwe don't do docker or k8s here, but there're deployment projects who do11:39
gebzI see11:40
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron stable/2023.1: Use ansible_facts['processor_vcpus'] instead of fact variable  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/91275411:40
gebzOk, created a rhel vm with 16 vcpu, 32G ram, and 100G nvme, let's see what this bad boy can do :D11:49
andrewbonneynoonedeadpunk: have you seen any issues with reply queues when taking down and rebuilding rmq nodes?11:56
andrewbonneyThese only live on one node in a set due to https://github.com/openstack/openstack-ansible-rabbitmq_server/commit/52ad552129afc715dc978c61edf881090fcf48c0 and we've seen issues with them not getting re-created during maintenance11:56
noonedeadpunkandrewbonney: frankly speaking, I'm not absolutely sure here. Like we totally had an issue with heat-engine - each time it was loosing connection to rabbitmq it needed a restart more or less12:00
noonedeadpunkgebz: I'm not 100% sure about health of ceph deploy on rhel today... bit worth trying/fixing. we jsut don't have that job in our CI12:01
gebzdebian/ubuntu/suse?12:02
noonedeadpunkubuntu works 100%12:03
gebzrhel is the simplest to manage because im on a restricted network12:03
noonedeadpunkdebian should work, so that should rhel as well. 12:03
noonedeadpunkeventually, we should catch/fix that if it doesn't, but we just don't run any EL in productions, so I don't really know if today it works12:04
noonedeadpunkworth trying :)12:04
gebzi forked stable/2023.212:04
gebzcloned*12:04
noonedeadpunkbtw, we have whole section about limited connectivity...12:04
gebznoticed :D12:05
gebzspent 3 hours yesterday spinning in circles till i found it12:05
gebzDocumentation team ought to have that at the beginning12:05
noonedeadpunkcontributions are always welcome :D12:05
gebzas "special considerations" section in the introduction or something12:06
gebzis there a repo for the documentation?12:06
noonedeadpunkwell, we have it a bit spreaded. So we have docs for the project, and it's stored here: https://opendev.org/openstack/openstack-ansible/src/branch/master/doc/source12:07
noonedeadpunkThen there's a deploy guide, which is a bit different instance: https://opendev.org/openstack/openstack-ansible/src/branch/master/deploy-guide/source12:08
noonedeadpunkand then we also have document pages for each role separately, descriping possible options/usecases12:08
noonedeadpunkie - for neutron: https://docs.openstack.org/openstack-ansible-os_neutron/latest/12:08
gebzok, my finger is on the trigger, you said ./scripts/gate-check-commit.sh aio_lxc_ and what else can i stuff in there? :D12:09
noonedeadpunkwhich is basically here: https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/doc/source12:09
noonedeadpunkgebz: pretty much wahtever project you want to see installed12:09
gebzhow may are there?12:09
gebz:DD12:09
noonedeadpunkaio - stands for just basics - keystone, horizon, placement, nova, cinder, glance, neutron12:09
gebzdoes aio include swift?12:11
noonedeadpunkso basically anything from this list, cutting `os_` preffix: https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/configure.html#openstack-service-roles12:11
noonedeadpunkno, it does not12:11
noonedeadpunkbut.12:11
noonedeadpunkceph does12:11
noonedeadpunkor well12:11
gebzoh yea12:12
noonedeadpunkI guess there's a choice you'd need to make, if you wanna to have swift-swift or RGW offering Swift-compatible API12:12
gebzCTO was more interested in ceph, so lets do ceph12:12
gebzI pressed the big red button :D12:16
gebzIt failed in the middle because it tried to ssh to itself using the new IPs but they weren't in the known_hosts12:33
gebzfixed that, reran, now it failed at bootstrap-host create LVM LV12:34
gebzI guess it doesn't like the fact that a ceph cluster was already created12:34
gebzis there a way to clean it up and rerun?12:35
gebzI see what you mean why it might fail on rhel, no repo called crb..12:46
noonedeadpunkhuh13:04
noonedeadpunksorry, was in a meeting13:04
noonedeadpunkso, it would be really interesting to see an output of the first failure.13:04
noonedeadpunkas it's super weird, given that in ansible settings it should ignore known_hosts for that kind of build13:05
noonedeadpunkhttps://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/openstack-ansible.rc#L4813:06
noonedeadpunkgebz: regarding crb repo - can you please paste output of the task that failed?13:06
noonedeadpunklike - thorugh https://paste.openstack.org/ or smth?13:07
gebzI could rerun it again later13:15
noonedeadpunkI wonder if that's smth available for centos/rocky but not for RHEL directly13:15
noonedeadpunkas we don't have RHEL anywhere handy to test things against it13:15
gebzThing is, Im using satellite to manage local repos13:16
gebzmight be a reason, hold on i will rerun it13:16
gebzmachine won't boot :'D13:19
gebzmight be some systemd service that's stuck13:19
gebzah.. networkmanager got disabled13:20
gebzrhel9 uses networkmanager by default13:20
jrossergebz: this is known to work on rocky9 so it should be very similar13:21
jrosserand the all-in-one uses systemd-networkd to manage its networking13:21
gebzIs rocky 9 officially supported by the project?13:23
jrosseryes, because it is possible to test that in our CI13:23
jrosserwhich is not the case for RHEL13:23
gebzPerfect13:23
gebzgonna tear it down and rebuild :D13:23
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Switch SHAs to EOM  https://review.opendev.org/c/openstack/openstack-ansible/+/91341315:28
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Switch SHAs to EOM  https://review.opendev.org/c/openstack/openstack-ansible/+/91341415:33
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Switch SHAs to EOM  https://review.opendev.org/c/openstack/openstack-ansible/+/91341815:36
noonedeadpunkjrosser: have you seen that there's presumably an OAUTH support now in Skyline?15:37
noonedeadpunkhttps://review.opendev.org/c/openstack/kolla-ansible/+/90586015:38
jrossernoonedeadpunk: i saw there was something but there was basically no docs15:38
noonedeadpunkI've seen they're doing some l18n attempts as well15:39
jrosseryou see the thing is that openid != openid-connect15:39
noonedeadpunk(which is nice for non-EN I guess :D)15:39
noonedeadpunkwell.... that is true....15:39
gebzretried building an AIO stack on rocky 915:40
jrosserand unfortunately sometimes these get used interchangeably15:40
jrosserbut everything i see in skyline only refers to openid15:40
gebzThis time it failed at LXC installation, proxy flagged the repo as suspicious :'D15:41
jrosserand so i never see any place to configure client-id / client-secret like you'd have with google auth or whatever15:41
noonedeadpunkgebz: ah, well... true... I think we're using a NeilHanlon's one at the moment specifically for templates?15:48
gebzyup15:48
gebzI looked at it, only 40 downloads :'D seems risky even to me15:49
gebzIm redoing it on ubuntu, passed the LXC step15:53
gebzbeen on the "build the base image using CLI tool" for ~20 mins now15:53
jrosser^ this should be quick, just a minute or two15:54
noonedeadpunkgebz: you could try just metal, without LXC15:54
gebzdoes it download anything external at this step?15:54
noonedeadpunkit does, yes15:54
gebzah.. I see...15:54
gebzmust be the damn proxy again15:54
noonedeadpunkit's building the image using debootrstrap15:54
gebz^ I'll give that a try after I've had something to eat, been at this for hours :'D15:56
jrosserif you have a complicated environment with proxies and limited connectivity, it's going to take some time and effort to get things working15:56
jrosserthis is not a "shrink wrap" type installation15:57
noonedeadpunkyeah, so that's a repo build by NeilHanlon for us, so that we could have lxc-templates-extra package15:57
gebz@jrosser tell me about it :'D15:57
noonedeadpunkI can recall smth was changed in epel regarding LXC, so maybe it's not needed anymore... but not sure15:57
noonedeadpunkhuh, just found https://github.com/ganto/copr-lxc4/tree/master15:58
noonedeadpunkso we can be able now to replace that one....15:58
jrossergebz: the idea is that all the hooks are there, so if you have a local repo mirror, you can point to it15:58
jrosseror if you locally cache pypi packages, you can point to that too15:59
noonedeadpunkand seems they've also builded incus15:59
jrosserbut we don't provide tooling to create any of that, as every place has pretty much their own requirements15:59
noonedeadpunkbut that's only for fedora /o\16:00
jrosseralso http proxies are pretty well supported during the deployment, but you need to understand and take care with what you are doing 16:00
gebzOnly got satellite on-prem, and that's for rhel.. no repo mirrors for other distros16:03
jrosserwell, if you have an http proxy it should work16:04
noonedeadpunkI wonder if we still need lxc-templates-extras.....16:05
ThiagoCMCjrosser, nobody is replying to the ceph-ansible GitHub issue (https://github.com/ceph/ceph-ansible/issues/7496)... I'm thinking about sending an e-mail to the guy from IBM that's changing the `stable-8.0` branch. Do you think worth contacting him? If yes, want me to including any message from you?16:14
jrosserThiagoCMC: well they did put back the config overrides bit in a later patch16:17
jrosserso you might want to edit that a bit16:17
jrosserand it's also likley that the data that was taken out specifically for openstack can be moved instead into osa group_vars16:19
ThiagoCMCOh, nice! Thanks for the info! The OpenStack-related bits you're talking about are these: https://github.com/ceph/ceph-ansible/commit/9c467e41b39af63dec9a0b237b550edb9c57da85 - Right?16:21
jrosserThiagoCMC: i have not checked in detail16:25
ThiagoCMCOk16:26
opendevreviewJonathan Rosser proposed openstack/openstack-ansible master: Use container setup role from plugins repo  https://review.opendev.org/c/openstack/openstack-ansible/+/90500416:38
noonedeadpunkjrosser: huh, seems you did 2 patches?:) https://review.opendev.org/c/openstack/openstack-ansible/+/905004 vs https://review.opendev.org/c/openstack/openstack-ansible/+/90898416:40
noonedeadpunkwhich you wanna leave? :LD16:40
noonedeadpunkI'd vote for one not in merge conflict :D16:41
jrosseroh hah17:01
opendevreviewMerged openstack/openstack-ansible-os_neutron stable/2023.1: Use ansible_facts['processor_vcpus'] instead of fact variable  https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/91275417:31
NeilHanlonnoonedeadpunk, gebz: yeah .. the lxc-templates stuff is (as I understand) deprecated and shouldn't be used. I don't even know that we need it any longer as I think we changed how we are creating the containers17:43
noonedeadpunkyeah...17:43
noonedeadpunkworth checking that...17:43
gebz:D17:43
gebzI'M HELPING17:43
NeilHanlonand apologies for being suspicious :D 17:43
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Add support for deploying Horizon with Skyline  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/91343917:54
noonedeadpunkcrap18:35
noonedeadpunkseems skyline does not support running on sub-urls18:35
noonedeadpunkie /skyline18:35
NeilHanlonbut... why18:40
NeilHanloni guess we can rewrite the urls in haproxy but..... why18:40
noonedeadpunkNeilHanlon: I'm sooooo long ago was dealing with web server config18:49
noonedeadpunkso help is appreciated18:49
noonedeadpunkso the thing is, they do have this in their static file: https://opendev.org/openstack/skyline-console/src/branch/master/skyline_console/static/index.html#L1118:49
noonedeadpunkso we need to kinda rewrite all uris to contain /skyline?18:49
NeilHanlonew..18:49
noonedeadpunkhm. maybe you're right, rewrite is exactly what I need... 18:50
NeilHanlonpossible, but also.. those files shouldn't be in git..18:51
noonedeadpunklol18:54
noonedeadpunkyeah. static files are really static :D18:54
noonedeadpunkfwiw, this repo contains _only_ such static files. Which are installable through python :D18:55
NeilHanlonyep.. so i think they actually get overwritten18:56
NeilHanlonhttps://opendev.org/openstack/skyline-console/src/branch/master/config/webpack.prod.js18:56
noonedeadpunkhm18:59
NeilHanloni *think* that the index.html in there needs to be modified to support webPack putting in the right info.. https://github.com/jantimon/html-webpack-plugin#options19:01
NeilHanlonbut I'm also not a web guy anymore lol19:01
NeilHanlonare we rebuilding ? or just installing from pip19:01
NeilHanloni was assuming we were running the build in repo_container but I guess that is a bad assumption19:02
noonedeadpunkI think we just installing19:02
noonedeadpunkwell19:02
noonedeadpunkbuilding wheels is not helpful to adjust it I assume19:02
noonedeadpunkbut yes, we build first kinda19:03
noonedeadpunkthough we don't trigger npm or anything like that I assume19:03
NeilHanloni think not, as they bundle the static content before building the wheel https://opendev.org/openstack/skyline-console/src/branch/master/Makefile#L7719:03
NeilHanlonotoh, if the url in their template wasn't using `/` at the start, this would just work, i think19:04
noonedeadpunkI guess so as well, yes19:11
NeilHanlonsed the file? 😂19:13
noonedeadpunkrewrite actually results in cycle redirect....19:13
noonedeadpunkyeah, so dunno19:13
NeilHanloni'll try to poke at your change this weekend19:26
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: [Feature] Add skyline deployment capability  https://review.opendev.org/c/openstack/openstack-ansible/+/85944619:30
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: [Feature] Add skyline deployment capability  https://review.opendev.org/c/openstack/openstack-ansible/+/85944619:31
noonedeadpunkI think we can consider this OK as long as we're OK with Skyline being default, when it's installed together with Horizon19:32
noonedeadpunkWhile Horizon will be under `/horizon`19:32
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Reflect horizon_webroot setting in Apache vhost configuration  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/91344219:36
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Add support for deploying Horizon with Skyline  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/91343919:38
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Add support for deploying Horizon with Skyline  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/91343919:38
jrossernoonedeadpunk: NeilHanlon if you look at the very first version of my os_skyline role (perhaps the one in my github early commits) i did the whole npm build from scratch23:32
jrosserit was very slow and needed a gigantic amount of RAM, many GB, so would have never worked in a CI job23:33
NeilHanlonyeah.. that sounds like webpack, honestly23:58
* NeilHanlon grumbles about javascript23:59

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!