Friday, 2023-09-08

noonedeadpunkwould be sweet to land that to fix adjutant on master: https://review.opendev.org/c/openstack/openstack-ansible/+/89383708:20
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_container_create master: Properly apply tags for include_tasks  https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/89429411:01
farbodhello again11:21
farbodi have a question11:21
farboddo we need a range of vlan IDs for linux bridge implementation of OSA and just one VLAN for OVS and OVN?11:22
farbod?11:28
noonedeadpunkum, it depends?11:33
noonedeadpunkyou're not required to have vlan networks at all in any scenario11:33
farbodso what is this trunc scenario with ranged VLANs11:34
farbodi am really confused11:34
farbodare they physical VLAN IDs on my switch?11:34
farbodor they are virtual inside the neutron service?11:35
farbodwhy we assign a whole interface to vlan network?11:36
farbodand whatis the difference between the br-vlan and br-vxlan network?11:36
farbodi am reading these concept about three days in a row and i cant understand it at all!11:37
farbodI would be very grateful if someone could explain these concepts to me completely from the basics.11:37
noonedeadpunkThese should be configred on switches for sure11:37
noonedeadpunkIt depends on usecases. For example, you might want to provide a separate external or internal networks for one of your customers.11:39
noonedeadpunkWith having only flat - you're limited with only 1 external network11:39
farbodi am not talking about external right now11:40
farbodi mean11:40
farbodas i understanded11:40
noonedeadpunkor well, depending on amount of flat networks ofc. But each flat requires to have a physical interface kind of11:40
mgariepybr-vlan can be used if you want to use vlan provider network.It can be used for a variety of things, like exposing some vlan to projects, or using it for external network.11:40
noonedeadpunkThough, you can use vlan tagged interfaces for flat networks as well11:40
farbodyes i know11:40
farbodbut imagine this scenario11:40
jamesdenton"provider" networks are VLAN networks, and these need to be configured by someone with the admin role. "tenant" networks can also be VLAN networks, and these are automatically assigned by Neutron. That's what the range is for; the allocatable range of VLAN IDs. "Tenant" networks can also be GRE, VXLAN or GENEVE, and there are ranges for those, too11:40
farbodso we are limited to VLAN range for the number of networks made by users?11:41
noonedeadpunkyeah, it's pretty much depends on your scenario11:41
noonedeadpunkfarbod: usually, ppl use vxlans for user private networks11:41
noonedeadpunknot vlans11:42
farbodyou mean one physical  VLAN is enough for vxlan network and many projects?11:42
noonedeadpunkyes11:42
jamesdentonVLANs are used for infratructure networks, like the overlay network encapsulates the dozens or hundreds of vxlan networks11:43
farbodso you mean packet tagging will be done inside the neutron and not on physical?11:43
jamesdentonVLANs can also be used for VM workloads, "tenant networks"11:43
jamesdentonyes, Neutron is expected to handle the tagging of packets11:43
farbodok11:43
farbodbut11:43
jamesdentontagging of frames, rather. the vlan tagging. The switchports need to be configured to allow that. And the VLANs need to exist on the switches11:44
farbodwhats the difference between overlay and tenant network?11:44
jamesdentonwell, an overlay network is a network type that "encapsulates" traffic. Meaning, you can run all of the VXLAN-based tenant networks inside a single 'overlay' network. So VLAN 100 might carry traffic for VXLAN 5,10,1511:45
farbodfor cross node traffic?11:45
jamesdentonthe tenant network 'web' is vxlan id 5, and tenant network 'db' is vxlan ID 1011:45
jamesdentonbetween nodes, yes11:45
jamesdentonthe nodes build a vxlan mesh11:45
farbodand why br-vlan uses multiple VLANs, i mean a range of VLANs for overlay network?11:46
jamesdentonthat's a mix of concepts11:46
jamesdentonfor vlan networks you might support vlan IDs 100-99911:47
jamesdentonfor vxlan networks you might support vxlan ids 8999-1133411:47
farbodon the switch?11:47
jamesdentonvlan on the switch, vxlan inside an overlay network (there's only 1) - the overlay network between hosts might be vlan 5011:48
jamesdentonvxlan traffic gets tunneled through it. 11:48
farbodaha11:48
farbodso we have one tunnel network connected to a physical VLAN on the switch and there are lots of vxlan networks which are tenant network and are virtually inside the neutron?11:49
farbodam i right?11:49
jamesdentonexactly11:49
farbodso11:49
farbodok11:49
farbodif we leave this aside11:49
jamesdentonand those aren't directly reachble, thus the need for a neutron router - it connects the tenant network (vxlan) to a provider network (vlan) - and through the user of a float IP (nat) allows someone to connect to the vm11:49
farbodok 11:50
jamesdentoni've got to run - be back in a few hrs11:50
farbodthanks11:50
farbodanyone else to answer my questions?11:50
farbodsorry about that :(11:50
farbod?11:51
noonedeadpunkyou throw a question and the we'll see :D11:55
farbodoh OK11:56
farbodsorry11:56
farbodi mean11:56
farbodi know only one VLAN is enough for overlay network11:57
farbodbut if it's enough11:58
farbodwhy11:58
farbodwe use br-vlan11:58
farbodwith a range of VLAN IDs?11:58
noonedeadpunkas I said - you're not obliged to use it11:58
farbodwhy some ppl use it?\11:58
noonedeadpunkWe use it to bring owned by customer networks to the region11:58
noonedeadpunkSo folks came and say - we have an ipv4 network we own - we want to use it in your cloud11:59
noonedeadpunksecond usecase I personally have - to connect Octavia lbaas network with controllers11:59
farbodyou mean this br-vlan is only for external networks?12:00
noonedeadpunkAs octavia-api needs to reach amphoras. One way could be to route that, but another just to use vlan network12:00
farbodand not internal ones?12:00
noonedeadpunkit can be used for internals as well12:00
noonedeadpunkbut it can for externals12:00
farbodinternal in physical space? or inside neutron?12:00
noonedeadpunkLike I know folks who don't do flat networks at all - they jsut use vlan for external12:00
noonedeadpunkWell. with vlan it's more how you configure it on switches :D12:01
opendevreviewMerged openstack/openstack-ansible stable/2023.1: Update Adjutant and Neutron SHAs  https://review.opendev.org/c/openstack/openstack-ansible/+/89383712:01
noonedeadpunkwhat neutron do is basically bridge tagged vlan with VM interface12:01
noonedeadpunkand creates that tag12:01
farbodlet me ask my question in another way12:02
noonedeadpunkand then it's up to switch and your networking how "internal" it is12:02
farbodif i don't configure a br-vlan, what will i miss?12:02
noonedeadpunkdepending on your usecases I guess12:03
noonedeadpunkLike I can imagine how to use just bunch of flat networks instead of vlan12:03
noonedeadpunkmatter of taste imo12:03
farbodso let me explain my scenario12:03
farbodi have a bunch of servers with only one physical interface12:04
farbodand limited to only 5 VLANs o n physical switch12:04
noonedeadpunkif you want to have range of vlans pre-configured on switches that you can use anytime, or create a flat network and run a role each time you need extra one12:04
farbodi want to make a private cloud12:04
noonedeadpunkyeah, then you don't need that12:04
farbodwith different projects12:04
farbodwith their own networks12:04
farbodmultiple networks12:04
noonedeadpunkif you have limitation on amount of vlans on switch12:04
farbodand maybe some routing between networks12:05
noonedeadpunkjust make a flat network as a vlan12:05
noonedeadpunkfor external connectivity12:05
noonedeadpunkand you will be fine12:05
farbodbut on other hand i want to assign public IPs which are available on one this VLANs12:05
farbodactually i can bridge on this vlan 12:05
noonedeadpunkyou can add eth1.100 as a flat network12:05
noonedeadpunkno problems with that as long as interface name is consistent12:06
farbodOK12:06
noonedeadpunkand then eth1.200 as vxlan12:06
noonedeadpunkand eth1.300 as mgmt12:06
noonedeadpunkand eth1.400 as storage12:06
farbodwill it make difference if i use LXB OVS OVN?12:06
noonedeadpunkI guess you got the idea :)12:06
farbodyes i got it12:06
farbodand one VLAN for external network yes?12:07
noonedeadpunkyes, you just add that eth1.100 as flat network to neutron12:07
farbodoh OK12:07
noonedeadpunkit can be tagged interface as well12:07
farbodsorry for  taking your time but do you have a some time to guide me in configuration? 12:08
farbodi have my configs ready and just want to check them out12:08
noonedeadpunkAnd I wouldn't add just eth1 as flat network, as then neutron will manage it and will add to the bridge12:08
noonedeadpunkso you won't be able to create another vlans on it12:08
farbodyes i tested that out and made some problems 12:08
noonedeadpunkRegarding lxb/ovs/ovn - well. I really love lxb but it's barely maintained nowadays. It's even marked as experimental, so it may be gone anytime soon due to limited interest. But I like it most by far, as it's super simple to debug and understand wtf is going on12:09
farbodwiil you guide me./12:09
farbod?12:10
noonedeadpunkI can try, but I don't have plenty of time to be frank12:10
farbodOK 12:10
farbodso lets look at this12:10
farboduser config yml file: https://paste.opendev.org/show/bzehqFuNZ5GS24011ukP/12:11
noonedeadpunkovs is kinda evolution of lxb, which I didn't get - jsut more troublesome version that has same drawbacks as lxb basically. OVN is a step forward and kind of future. Has it's own issues, like it's super complex to debug and understand what's wrong it's there12:11
farbodwhat changes shoulf i make to make it compatible with my scenario?12:11
noonedeadpunkIt's also relatively new, but it solves issues of ovs/lxb with networks namespaces12:12
farbodaha 12:12
farbodinfra node network config: https://paste.opendev.org/show/b8UbL8rvYPwML7h8WR85/12:13
farbodand compute and storage node network config: https://paste.opendev.org/show/bbvYa26PW0jVmep6rHYC/12:13
farbodthat enp8s0.4040 is the VLAN that have access to providing public IPs12:14
farbodwhich is placed in infra node and its config file12:14
noonedeadpunkaha, and you're going with... ovn/ovs/lxb?12:15
noonedeadpunkas config will depend on that12:15
farbodwith lxb12:15
farbodi want to be as simple as possible12:15
farbodat least for now which my knowledge is limited :) 12:16
noonedeadpunkok, then it's easier as I know how to do that lol12:16
farbodnice :D12:16
noonedeadpunkgive me couple of mins12:19
farbodOK 12:19
noonedeadpunkfarbod: is interface naming consistent across all hosts? All of them are enp8s0?12:23
farbodno12:23
farbodthe second one is enp41s012:23
farbodthe compute and storage one12:23
noonedeadpunkand it's ubuntu 20.04? 12:24
farboddebian 1212:24
noonedeadpunkdoesn't it come with netplan?12:25
farbodno 12:25
noonedeadpunkAlso I don't think we support debian 12 yet12:25
farbodi edit12:25
farbodoh soory12:26
farboddebian 1112:26
farbodyes12:26
farbodi tried the 12 one12:26
mgariepysystemd-networkd can rename the interface as well.12:26
noonedeadpunkaha, ok, as 12 comes with netplan12:26
farbodi got error in deployment12:26
farbodyes12:26
noonedeadpunkfarbod: so, you'll need either to rename interfaces, or create a bridge for enp8s0.404012:28
farbodi prefer the bridge one12:28
noonedeadpunkyou actually had quite fair config with 2 small nits12:33
noonedeadpunkhere's edited one: https://paste.openstack.org/show/bkPQyY8sy4JcftArFaTa/12:33
noonedeadpunkI assumed you will create br-ext enp8s0.4040 in it12:33
farbodcan you guide me thriugh that ? :) i think i will miss configure it.12:34
mgariepynoonedeadpunk, do you have some pci passthrouth with zed somewhere?12:34
mgariepymy config looks ok but the scheduler do not want to cooperate lol12:35
noonedeadpunknah, I don't12:35
noonedeadpunkI do have gpu's but they're vgpus12:35
noonedeadpunkand they're on Xena still12:36
mgariepyok12:36
noonedeadpunkwill upgrade to 2023.1 in 2 weeks though12:36
mgariepyi'm not quite sure if it's an issue with zed since it seems to be a half release  or with something else.12:37
mgariepyThe DB do have the pci device available.12:37
noonedeadpunkbut it's placement then, not scheduler you mean?12:37
mgariepyi'ts not in placement12:38
noonedeadpunkAs nova asks placement for pci device and then scheduler ofc decides if there's a match from available resources12:38
noonedeadpunkso placement gives resources where it's possible to schedule?12:38
noonedeadpunkas pci devies are kinda among resource providers12:39
mgariepyit passes until pcipassthroughfilter.12:39
noonedeadpunkaha12:39
noonedeadpunkok, we don't use that at all12:39
farbodnoonedeadpunk: is this ok for infra node network config file?: https://paste.opendev.org/show/bLWVt6kVFYkcObRbAJFY/12:41
farbodand should i have this br-ex bridg on compute nodes?12:41
noonedeadpunkyes, you should12:41
noonedeadpunkor well12:41
noonedeadpunkagain depending on your scenario :)12:42
farbodi mean12:42
noonedeadpunkif you want to have VMs directly connected to external network - then yes12:42
farbodthe public IP won't route from network which here is infra node?12:42
noonedeadpunkThough there's another way around, by having only internal (vxlan) network on VM, then have neutron router, basicaly ip network namespace on infra, which connects internal and external networks with nart12:43
noonedeadpunk*nat12:43
farbodoh12:43
farbodi get it12:43
noonedeadpunkand then use floating ip to access from external network to vm through public IPs12:43
farbodyes12:43
farboddid you check the infra network config file which i sent above?12:44
noonedeadpunklooks okeyish12:44
farbodwhat about this storage compute node config? : https://paste.opendev.org/show/bjwsAFI3I5F5sRmQVAnY/12:45
noonedeadpunkI honestly not sure about storage and how LVM works with scsi, as I never really used that outside of AIO setups12:46
noonedeadpunkBut excluding this part - looks fair12:46
farbodok12:46
farbodshould i do additional configs for this storage or its enough?12:46
noonedeadpunk(and if glance needs storage network at all, as I believe it will jsut store locally as files12:47
farbodis there any simplest way to config this storage node?12:49
farbodi mean an automated approach12:51
farbodi just want a storage for running vms12:51
mgariepyany issue to upgrade to 2023.1 from zed right now ?12:52
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Re-order OVN diagrams in networking guide  https://review.opendev.org/c/openstack/openstack-ansible/+/89429712:52
noonedeadpunkfarbod: I believe your usecase if described well enough by this usecase: https://docs.openstack.org/openstack-ansible/latest/user/network-arch/example.html#single-interface-or-bond12:53
noonedeadpunkmgariepy: if you upgrade to stable/2023.112:54
farbodnoonedeadpunk: what is the simplest way to make an storage?12:55
farbodin my config there are LVM backends that i don't want12:55
noonedeadpunkprobably simplest is .... NFS?12:55
noonedeadpunkI hate it though12:55
farbodshould i do configs on the host?12:56
noonedeadpunkbut at least that is shared storage that you technically can scale....12:56
noonedeadpunkwe don't provide means to install nfs, and yes, you'll need to apply some configs to make use of it12:56
noonedeadpunkbut it's pretty straighforward I'd say12:57
farbodany docs for it?12:57
noonedeadpunkthat;'s for cinder: https://docs.openstack.org/openstack-ansible-os_cinder/latest/configure-cinder.html#nfs-backend12:57
noonedeadpunkfor glance it was matter of configuring mount iirc that can be done with role12:58
noonedeadpunklike that: https://opendev.org/openstack/openstack-ansible-os_glance/src/branch/master/defaults/main.yml#L247-L25212:58
farbod👍️12:59
admin1mgariepy, for me upgrade broke the nova-console and terraform also broke :D 13:00
admin1rest look fine 13:00
noonedeadpunkFor us everything worked quite nicely. 2 small bugs wrt custom features we don't test in gates13:01
noonedeadpunkone regarding vpnaas and another for wiping nova db periodically13:01
mgariepywhat broke in terraform ?13:03
mgariepyadmin1, ?13:03
mgariepystuff not tested in gate is always a bit more tricky lol.13:03
mgariepyit's always there that issue occurs :) also DB with more history sometimes causes issue.13:04
mgariepynova: pci.report_in_placement = False in zed,. and to be frank i'm not even sure placement support pci correcly on zed. 13:29
noonedeadpunkyeah, it should13:30
noonedeadpunkas we report gpu as pci in xena13:30
mgariepyit doesnt support the traits stuff in the device.13:30
mgariepyhmm do you set report_in_placement to true ?13:30
mgariepynot even an option there :s13:31
noonedeadpunkhm, no... though we do see gpu PCI devices as resources that are part of compute node13:36
noonedeadpunkthough we use `enabled_mdev_types` - so it might be different13:37
noonedeadpunkbut it;s like that: https://paste.openstack.org/show/bMmC9aHqRPx3HsdxpCWd/13:37
noonedeadpunkbug I guess while it's look like PCI, it's not in fact the same13:37
opendevreviewChristian Rohmann proposed openstack/openstack-ansible-haproxy_server stable/yoga: Make use of haproxy_rise and haproxy_fall variables  https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/89429913:38
mgariepyin the placement db you have these ressource ?13:40
mgariepyit's probably a bit different for these..13:40
noonedeadpunkI do13:46
mgariepyit's not type 3 is it?13:47
mgariepytype: PCI_DEVICE13:47
mgariepyprobably like 10 ? VGPU or something13:47
noonedeadpunkhttps://paste.openstack.org/show/bGSM4H5BBIxgwObJYTYg/13:48
noonedeadpunkhow to check type?13:48
mgariepymysql 13:49
noonedeadpunkyeah, resource_class VGPU13:49
noonedeadpunkchecked with `openstack resource provider inventory list`13:49
mgariepyso it is indeed different13:49
noonedeadpunkdespite it's `local_pci_0000_25_01_1`13:50
mgariepyyeah13:50
noonedeadpunkwell, sorry for confusing you :)13:50
mgariepyzed really seems to be half release on this part lol.13:50
mgariepyno worries13:50
opendevreviewMerged openstack/openstack-ansible master: Bump SHA for openstack-ansible-plugins collection  https://review.opendev.org/c/openstack/openstack-ansible/+/89383513:59
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_adjutant master: Stop reffering _member_ role  https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/89146214:04
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_adjutant master: Use proper galera port in configuration  https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/89009214:04
noonedeadpunkI hope adjutant should be fixed now14:04
farbodhi again :)14:28
farbodwhen i try to make volumes i get this message ->  schedule allocate volume:Could not find any available weighted backend. 14:29
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Add example network architectures for OVN  https://review.opendev.org/c/openstack/openstack-ansible/+/89438416:23
noonedeadpunkjamesdenton: it would be awesome if you could take a look at this thing, as I'm not 100% sure how true is that ^16:24
noonedeadpunkFWIW, I will be mostly away during next week16:24
noonedeadpunkso glhf :D16:25
mgariepyhave fun also .16:30
mgariepyas a matter of preference i do prefer not having lxb in the way of ovn 16:30
mgariepyonly uses lxb on ctrl for lxc containers 16:32
noonedeadpunkI played with ovs for lxc and it worked nicely as well just in case16:32
noonedeadpunknot sure why, but it works :)16:32
mgariepylol16:32
mgariepyyeah indeed it should.16:32
mgariepyi prefer lxb on it because i'm lazy16:33
noonedeadpunkbut it's same exactly - just need to add `openvswitch` somewhere in openstack_user_config16:33
mgariepyiptables works just fine on lxb and physical interface. while i use flows for everyting else.16:34
noonedeadpunkwe have doc for that I'm pretty sure16:34
noonedeadpunkAnd ovs bridges were created with maas in my case16:34
mgariepymaas :sick:16:34
noonedeadpunkSo it was very-very lazy16:34
mgariepyi don't like maas lol16:34
noonedeadpunkIt has things to love it and to hate it16:35
mgariepythe network part was not good enough for me.16:35
noonedeadpunkThey've release ansible collection to manage it through API in an automated way couple of month ago16:35
mgariepywaiting on incus to release :D. 16:36
noonedeadpunkyeah.....16:36
mgariepyprobably better with ansible than mass | jq ..16:36
noonedeadpunkhttps://github.com/maas/ansible-collection/tree/main/plugins/modules16:37
noonedeadpunkI'd say ironic comparing to maas is too complicated for personel responsible for hardware testing and provisioning16:38
noonedeadpunkbut yeah, it's too narrowed in some cases indeed16:39
mgariepyyeah i did tested bifrost a bit... and finally dropped it 16:39
noonedeadpunklike having DNS but without good clustering and HA16:39
noonedeadpunkno debian :D16:40
noonedeadpunkso, what do you do then?16:40
farbodcan we deploy a test enviroment without cinder or any other block storage?16:41
noonedeadpunkyou can16:41
farbodi mean i tried to run an instance without block storage but i got no volume backend16:41
noonedeadpunkjust don't define storage* in openstack_user_config16:41
noonedeadpunkum. you should be able to avoid using block storage16:41
mgariepyrunning on research grant i dont have new machines oftens. so i only have a pxe with some jinja for the preseeding via curtain16:42
noonedeadpunkbut it depends on the VM creation command16:42
mgariepycurtin16:42
farbodis it possible from dashboard?16:42
farbodyou mean using Ephemeral?16:42
noonedeadpunkUm, I think it is? You need to have flavors with some disk (more then 0) and then when creating server select "create volume: no"16:43
noonedeadpunkyes16:43
farbodaha ok Thanks16:43
noonedeadpunkmgariepy: I see16:43
mgariepyi'ts not perfect but works kinda ok.16:45
farbodshould i deploy storage-infra_hosts when i don't need the storage node?16:49
farbodi mean cinder api service16:49
noonedeadpunkno, neither it nor storage_hosts16:50
admin1mgariepy , this is what got broke -> https://gist.githubusercontent.com/a1git/2efcfd956f342333070f04b7bc048f6f/raw/0bc97dbc4888fae7bd97505557f73a3eb2186480/gistfile1.txt ..    ( i was changing water in the aquarium,  did not see the msg earlier ) 16:50
noonedeadpunkwas it sea aquarium, like with slated water, corrals, etc ?:)16:50
admin1nah  .. tap water kind with guppies 16:51
noonedeadpunkthat is too easy lol16:51
mgariepyhaha.16:52
admin1well, my aquarium is in a corner and i have to run a 30 meter pipe through living room from kitchen to do it :) 16:52
noonedeadpunkWell, we were running a sea aquarium in the office back in the days16:52
noonedeadpunkAnd from time to time creatures were just disspearing there, including fish16:53
noonedeadpunkI assume fish was feeling bad and has been eaten by corals16:53
noonedeadpunkBut nobody knows for sure16:54
noonedeadpunkThey were just never seen again16:54
mgariepyweird 16:54
mgariepywere they flying fish ? ;p16:55
noonedeadpunkwell, when we tried to re-arrange rocks there, we found like thousands of worms hinding in there, that we didn't see during the day16:56
noonedeadpunkand we never put any worms there, just in case - they likely evolved somehow16:56
mgariepyrocky9 disk full16:56
mgariepy:/ why ?16:57
mgariepyhttps://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/89377316:57
noonedeadpunkbecause of ovs 2.1716:57
noonedeadpunkbut no idea why it is still 2.1716:57
noonedeadpunkWe've released the lock we had 16:57
noonedeadpunkAnd I wasn't able to see 2.17 installed in local aio16:57
noonedeadpunkbut it somehow is in CI16:57
noonedeadpunkso try to re-check16:58
noonedeadpunkah, and we also have patch on master that allows to limit journald size16:58
noonedeadpunkit wasn't backported though16:58
noonedeadpunkbtw, would be awesome to land adjutant things on master as well https://review.opendev.org/q/project:openstack/openstack-ansible-os_adjutant+status:open16:58
mgariepyshoudl we backport the journald bits>?17:02
mgariepyha. not fully merged yert.17:03
mgariepyyet**17:03
noonedeadpunkworth to understand how in the world we get ovs 2.17 instead of 3.117:06
noonedeadpunkanyway, checking out for the week :)17:07
mgariepyok have fun17:07
farbodwhy i get this error: Danger: An error occurred. Please try again later. when i want to create network from admin section in horizon?18:00
jamesdentonnoonedeadpunk i will take a look at those docs, thank you18:21

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!