Wednesday, 2020-11-18

*** Gimmix has quit IRC00:14
*** klamath_atx has joined #openstack-ansible00:43
*** cshen has joined #openstack-ansible01:00
*** cshen has quit IRC01:04
*** gshippey has quit IRC01:05
*** jawad_axd has quit IRC01:19
*** spatel has joined #openstack-ansible02:48
*** lemko has quit IRC02:53
*** lemko has joined #openstack-ansible02:53
*** cshen has joined #openstack-ansible03:00
*** cshen has quit IRC03:05
*** klamath_atx has quit IRC04:38
*** cshen has joined #openstack-ansible05:01
*** cshen has quit IRC05:05
*** klamath_atx has joined #openstack-ansible05:08
*** klamath_atx has quit IRC05:18
*** klamath_atx has joined #openstack-ansible05:19
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-ansible05:33
*** gyee has quit IRC06:08
*** pto has joined #openstack-ansible06:20
*** pto_ has joined #openstack-ansible06:22
*** cyberpear has quit IRC06:23
*** pto has quit IRC06:24
*** spatel has quit IRC06:30
*** cshen has joined #openstack-ansible06:38
*** cshen has quit IRC06:42
*** miloa has joined #openstack-ansible06:42
*** spatel has joined #openstack-ansible06:49
*** rpittau|afk is now known as rpittau06:49
*** SiavashSardari has joined #openstack-ansible06:53
*** spatel has quit IRC06:53
*** jbadiapa has quit IRC06:53
*** jamesgibo has joined #openstack-ansible07:05
*** jamesgibo has quit IRC07:21
*** pcaruana has joined #openstack-ansible07:49
*** pcaruana has quit IRC07:51
*** pcaruana has joined #openstack-ansible07:51
*** klamath_atx has quit IRC07:58
*** klamath_atx has joined #openstack-ansible07:59
*** jamesgibo has joined #openstack-ansible08:08
*** jamesgibo has quit IRC08:08
*** andrewbonney has joined #openstack-ansible08:16
*** cshen has joined #openstack-ansible08:27
*** tosky has joined #openstack-ansible08:45
*** MickyMan77 has joined #openstack-ansible09:07
*** shyamb has joined #openstack-ansible09:13
openstackgerritAndrew Bonney proposed openstack/openstack-ansible-os_zun master: DNM: Update zun role to match current requirements  https://review.opendev.org/76314109:25
pto_What are you guys doing to backup ceph block devices in your openstack clusters?09:28
SiavashSardarithere are couple of ways to tackle this issue, depends on how much you can spend on resources09:30
jrosserwe put a separate NFS/ZFS setup behind cinder-backup09:30
jrosseralso it depends what you want to mitigate09:30
*** shyamb has quit IRC09:30
jrosseruser error -> i need my old version09:30
jrosserceph fail -> i need to put everything back09:31
*** kukacz has quit IRC09:31
jrosseryou end up with different solutions depending what you decide the problem to solve is09:31
jrosseras you can cinder-backup onto ceph itself09:31
jrosserin a completely different ceph environment that's still using RBD but not openstack, we export RBD snapshots off of that with cron + scripts09:33
pto_Right now i am exploring my options. I am looking at the native openstack backup, backy2 and rclone to make incremental backups of block devices09:33
jrosserthats really addressing disaster recovery rather than backup-as-a-service for users09:33
*** pto_ is now known as pto09:34
ptoMy client has two use cases, backup of ceph block devices and maybe a pool to s3 and specific openstack instances to s3 if possible09:34
kleiniCeph should be pretty resilient against any data loss, right? If you Ceph cluster is large enough, I can not imagine any Ceph failure.09:35
SiavashSardarifrom ceph point of view you can snap your images and use export import or you can use mirroring for backup only or you can setup full mirroring and use both ceph clusters as acrive-backup09:35
jrosserkleini: we destroyed a ceph cluster entirely because the NTP to the mons broke09:36
ptokleini: The cluster is on 7.2 pb right now in one DC. We plan to split it into two which will protect against e.g. fire09:36
jrossertotal loss of everything :(09:36
SiavashSardariI too had an horror story with ceph that caused us data loss09:36
jrossermany years ago we also had a total loss at ceph major upgrade, due to a bug09:36
ptoI am working for Aalborg University and they are archiving research data to ceph, and some data sets are very important and must be stored offsite due to seggregation og duties and compliance. Therefor I need to come up with a practical solution to copy rbd block deveis to s309:37
*** kukacz has joined #openstack-ansible09:38
jrosserpto: let me see if my colleague can join here who did our snapshot/export of RBD09:38
ptojrosser: I am testing backy2 right now and it looks very nice, but its a manual task to setup each pool/rbd. I would be nice if the integrated openstack backup could be used too09:39
SiavashSardariwe tried snap export import but that needs another ceph cluster to import09:39
*** Carcer has joined #openstack-ansible09:39
ptoIs the openstack backup to swift compliant with s3?09:40
jrosseri would guess probably not as swift is a different API to s309:42
jrosserwhat was slightly disappointing for me was cinder-backup does not seem to be able to do scheduled backups09:43
ptojrosser: really?09:43
SiavashSardaripto I know you said you need to backup block devices, but if you can handle backing up your data in your platform layer. that can solve your problem, I mean vm users upload their important data to s309:43
ptoI think backy2.com looks very promising, as it can very easy make a backup of rbd to disk or s3. It support schedules and sla reports09:43
SiavashSardariif that is possible09:43
ptoSiavashSardari: Not really. The data is abstracted from them to a mount point in the archive solution. It could work for the OpenStack part09:44
ptoI am just a big fan of doing things automatic, E.g. when an instance is spawned, a backup could be automatical setup09:45
jrosserso i think we handcrafted something a bit like backy2 which went RBD->NFS09:46
SiavashSardarijrosser we are looking for a solution like that, could you please tell us more about your solution?09:47
ptojrosser: Thanks. I think i will need to do something like this. I am just a little worried that i will get the right volumes09:47
SiavashSardaripto I like automating stuff too, but sometimes that comes at a very high price.09:49
*** yolanda__ has joined #openstack-ansible09:50
ptoSiavashSardari: I have tested backy2, and it look promising. Its easy to setup a backup job: backy2 backup rbd://production-volumes/volume1 my-backup109:50
SiavashSardariwe wanted to setup full mirroring in ceph but we do not have enough resources for that. so we are improvising to find a middle ground09:50
SiavashSardariBWT thanks for introducing backy2, we didn't find that in our research.09:51
sepSiavashSardari, i use backy2 to backup rbd image onto a different ceph cluster's cephfs09:52
CarcerI wrote a backup script for rbds in an opennebula installation but it looks a lot like that backy2 does everything I did more elegantly09:52
SiavashSardariI will look into that. but QQ, backy2 will backup all ceph images or you can pick some of images?09:53
sepit backup the ones you tell it to.09:58
admin0quick question .. i have ceph+osa in one cluster .. in that cluster, if i want to add a 2nd backend for cinder, which is another ceph, how do i do it ?09:58
admin0has anyone added 2 diff ceph backends to cinders ?09:59
SiavashSardarithanks guys, backy2 really looks promising09:59
ptoSiavashSardari: your welcome :-)10:00
*** shyamb has joined #openstack-ansible10:03
*** yolanda__ is now known as yolanda11:00
damiandabrowskii also used benji before, it's also great tool based on backy2. Maybe worth a try: https://benji-backup.me/11:00
*** SiavashSardari has quit IRC11:03
*** pcaruana has quit IRC11:08
*** spatel has joined #openstack-ansible11:31
*** spatel has quit IRC11:36
*** shyamb has quit IRC12:05
openstackgerritAndrew Bonney proposed openstack/openstack-ansible-os_zun master: DNM: Update zun role to match current requirements  https://review.opendev.org/76314112:05
*** yasemind34 has joined #openstack-ansible13:06
openstackgerritAndrew Bonney proposed openstack/openstack-ansible master: Add Zun CI requirement to Zuul required projects  https://review.opendev.org/76317713:22
openstackgerritAndrew Bonney proposed openstack/openstack-ansible-os_zun master: DNM: Update zun role to match current requirements  https://review.opendev.org/76314113:24
*** cyberpear has joined #openstack-ansible13:36
*** simondodsley has joined #openstack-ansible13:58
*** spatel has joined #openstack-ansible14:00
*** cshen has quit IRC14:07
*** macz_ has joined #openstack-ansible14:08
*** macz_ has quit IRC14:12
*** mbuil has quit IRC14:12
openstackgerritJames Denton proposed openstack/openstack-ansible-os_neutron master: Test OVS/OVN deployments on CentOS 8  https://review.opendev.org/76266114:24
openstackgerritJames Denton proposed openstack/openstack-ansible-os_neutron master: Test OVS/OVN deployments on CentOS 8  https://review.opendev.org/76266114:26
spateljamesdenton: i have put my OVS+DPDK stuff on my blog for fun: https://satishdotpatel.github.io//openstack-ansible-add-compute-node-using-openvswitch-dpdk/14:35
spatelIn my lab i have added each compute node using different neutron network deployment14:36
jamesdentonnice! now for benchmarks14:36
spatelYes i need to install Trex and give it a try on each compute node and put some number here14:37
spatelDoes Trex is client server style tool ? I never used before but i will look at up on internet.14:38
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-role-systemd_service master: Add possibility to configure systemd sockets  https://review.opendev.org/76319414:40
jamesdentonIIRC it's all generated by one node and traffic is routed thru the DUT14:40
jamesdenton(device under test)14:40
jamesdentonBTW i did verify the /usr/local/var/run/... path no longer used14:42
spatelcool :)14:42
spateljamesdenton: lest that NFV patch get it out so we have working OVS deployment for CentOS14:43
jamesdentoni keep finding things wrong :D14:44
spateljamesdenton: soon i want to make my hand dirty on OVN deployment (for CentOS 8)14:44
ThiagoCMCspatel, I tried it before, got burned!  :-P14:44
spatelThiagoCMC: i will just play in Lab not for production :)14:45
ThiagoCMCCool... lol14:45
ThiagoCMCHere is what happened with me: https://bugs.launchpad.net/neutron/+bug/168988014:45
openstackLaunchpad bug 1689880 in neutron "[OVN] The "neutron_sync_mode = repair" option breaks the whole cloud!" [Undecided,New]14:45
ThiagoCMCGonna test it again someday14:45
spateli don't know what is the big advantage of OVN over current deployment?  (SDN is good if you control your physical switch also using OVN)14:46
jamesdentonnoonedeadpunk is there a plan for suse support going forward?14:49
jamesdentonjust curious14:49
noonedeadpunkjamesdenton: no, we've dropped support of it14:49
noonedeadpunkbut didn't do cleanup14:49
noonedeadpunkwell, did just partially14:50
jamesdentongot it14:50
jamesdentoni'll try and purge it when i come across it14:50
noonedeadpunk+114:50
ThiagoCMCCanonical messed it up? E: The repository 'http://security.ubuntu.com/ubuntu focal-security Release' does not have a Release file.14:56
ThiagoCMCO_O14:56
*** miloa has quit IRC14:56
openstackgerritMerged openstack/openstack-ansible-openstack_openrc master: Adding support of system scoped openrc and clouds.yaml  https://review.opendev.org/76209015:01
*** Miouge has quit IRC15:03
*** miouge3625368681 has quit IRC15:03
openstackgerritMerged openstack/openstack-ansible-galera_server master: Fix to mariadb backup script  https://review.opendev.org/76299015:06
noonedeadpunkjrosser: I'm already thinking if we should just mask these sockets and use traditional conf file as we used to lol15:23
jrosserto make a small change only?15:23
noonedeadpunkyeah..15:24
noonedeadpunkas otherwise we kind of need to override provied by libvirtd package sockets and services...15:24
noonedeadpunkand it feels it's aspplicable only for focal at the moment...15:25
noonedeadpunkbut yeah, probably we should move futher here....15:26
spatelrandom question, if i enable Two Factor authentication on my GitHub does that required me to try MFA code each time i do pull/push/commit ?15:54
spatels/try/type/15:54
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-role-systemd_service master: Added bunch of systemd_*_targets variables  https://review.opendev.org/76321116:02
*** yasemind34 has quit IRC16:10
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-role-systemd_service master: Add possibility to configure systemd sockets  https://review.opendev.org/76319416:12
*** macz_ has joined #openstack-ansible16:13
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/ansible-role-systemd_service master: Add possibility to configure systemd sockets  https://review.opendev.org/76319416:24
openstackgerritMerged openstack/openstack-ansible-os_manila master: Add Manila key generation and distribution  https://review.opendev.org/70501916:26
*** cshen has joined #openstack-ansible16:49
*** cshen has quit IRC16:53
*** klamath_atx has quit IRC16:54
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_nova master: Use systemd sockets for libvirt  https://review.opendev.org/76321616:58
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_nova master: Use systemd sockets for libvirt  https://review.opendev.org/76321617:00
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed openstack/openstack-ansible-os_neutron master: Return calico to voting  https://review.opendev.org/70265717:11
openstackgerritJames Denton proposed openstack/openstack-ansible-os_neutron master: Test OVS/OVN deployments on CentOS 8  https://review.opendev.org/76266117:19
spatelDo you guys sync all RabbitMQ queue in HA or just handful for performance?17:35
spatelI am thinking to remove notification.* from HA group17:35
openstackgerritJames Denton proposed openstack/openstack-ansible-os_neutron master: Test OVS/OVN deployments on CentOS 8  https://review.opendev.org/76266117:53
*** cshen has joined #openstack-ansible17:55
*** mmercer has joined #openstack-ansible17:56
*** cshen has quit IRC17:59
*** miloa has joined #openstack-ansible18:05
*** miloa has quit IRC18:06
nurdieI just tried to create an image out of an ephemeral VM and now the disk seems to be gone? I can't even find it on Ceph. Anyone have any advice?18:14
*** cshen has joined #openstack-ansible18:21
*** kleini has quit IRC18:30
*** kleini has joined #openstack-ansible18:31
*** cshen has quit IRC18:32
-openstackstatus- NOTICE: The Gerrit service at review.opendev.org is being restarted quickly as a pre-upgrade sanity check, estimated downtime is less than 5 minutes.18:35
*** kleini has quit IRC18:35
*** kleini has joined #openstack-ansible18:36
*** kleini has quit IRC18:37
*** cshen has joined #openstack-ansible18:39
*** kleini has joined #openstack-ansible18:39
*** rpittau is now known as rpittau|afk18:39
*** cshen has quit IRC18:45
*** cshen has joined #openstack-ansible19:02
*** kleini has quit IRC19:02
*** openstackgerrit has quit IRC19:02
*** andrewbonney has quit IRC19:04
*** kleini has joined #openstack-ansible19:16
*** kleini has quit IRC19:19
*** cshen has quit IRC19:34
*** kleini has joined #openstack-ansible19:42
noonedeadpunkjrosser: seems we have isues with parallel git clone19:56
noonedeadpunkhttps://zuul.opendev.org/t/openstack/build/6fc2de8bf52242c4acc8aafa1759a021/log/job-output.txt#339319:57
noonedeadpunknot sure why we run it in gates at all, but goot that we've catched19:58
*** cshen has joined #openstack-ansible19:59
noonedeadpunk"git reset --force --hard 7b7f20c636c12a02c0452c977e7952716fc9edcb\\n  stderr: 'error: unknown option `force'"20:00
noonedeadpunkI also not sure what is "--force"20:00
*** gyee has joined #openstack-ansible20:10
ThiagoCMCQuick question: I deployed a new OSA Usurri today with 2 haproxy_hosts but, the VIP is still at the first Controller. I was expecting that the IP to be only at one of the two HAProxy hosts, am I wrong?20:32
ThiagoCMCThe haproxy daemon was installed in all controllers anyway, is this expected?20:35
jrosserthat is not right20:50
jrosserwell, also neutron l3 node requires haproxy, so that would be the controllers20:51
jrosserThiagoCMC: is the haproxy service running on the controller? we may need to inhibit that20:53
*** openstackgerrit has joined #openstack-ansible20:53
openstackgerritMerged openstack/openstack-ansible master: Unfreeze roles  https://review.opendev.org/76218520:53
ThiagoCMCjrosser, my L3 is at the compute nodes, not at the controllers20:55
jrosserhmm would be interesting to find out how haproxy got installed20:56
ThiagoCMCCool20:57
jrosseryou can cross reference the apt log with the ansible log20:57
ThiagoCMCRight, I'll do another fresh install, with clean facts, just to make sure. And grab the logs right after haproxy playbook.20:58
jrossernoonedeadpunk: i think that theres an error before the 'force' thing,  git reset --hard 7b7f20c636c12a02c0452c977e7952716fc9edcb\n  stderr: 'fatal: Could not parse object '7b7f20c636c12a02c0452c977e7952716fc9edcb'21:36
spatelas soon as i added veth for lb-lbaas my controller nodes flooded with this packets - http://paste.openstack.org/show/800178/21:52
spatelany idea what is this IPv6 and where its coming from21:52
jrosseryou have some interface somewhere with ipv6 enabled21:59
jrosserwhat you see there is just "how ipv6 works", it's the link local addresses (fe80::...) looking for something that is a router22:00
spateli do have ipv6 enabled but not using them22:00
spatelas soon as i reboot all infra nodes flood stopped22:01
spatellook like as soon as i added veth pair to connect br-lbaas with br-vlan created loop or something. (very odd) i do have same setup in other datacenter22:02
spateljrosser: do you think i should complete disable IPv6 ?22:03
jrosserthe rate there looks surprising22:04
spatelit was bad flooding even i can't ssh to box and traffic was spike up on network over 100mbps22:05
jrosseryou can tell whats generating them becasue its eui-64 addresses22:06
jrosserthat will give you the mac addresses to figure out where it's coming from22:06
spatellet me enable veth again and see if its spiral loop22:06
spatelanyway i would like to disable IPv6 if i am not using it.22:07
spateli don't want it create any issue in future22:07
spatelAs soon as i enable veth on two infra node this flood start, look like its getting in loop22:18
spatelthose MAC address are part of new newly created v-br-lbaas pair22:21
spatelThis is the script i am running to add veth - http://paste.openstack.org/show/800180/22:22
ThiagoCMCjrosser, I just tried again a fresh deployment, the HAProxy and Keepalived are being installed in all controllers, despite the fact that the haproxy_hosts is configured and points to different machines. Any tips to where are the logs that I need to take a look?22:23
spatelsame script running fine on centos7 with ipv6 enabled22:23
spatellook like centOS8 issue22:23
jrosserThiagoCMC: you could run manually the haproxy install playbook with --list-hosts to see what it thinks it is targetting22:23
jrosserif the controllers are listed then its an inventoy / openstack_user_config problem22:24
ThiagoCMCYep, the controllers are listed!22:24
jrosseri have a deployment seperate haproxy nodes so can compare notes tomorrow if you can't find why22:24
*** rh-jlabarre has joined #openstack-ansible22:25
ThiagoCMCAnd this is a fresh install (from scratch, no previous Inventory/facts under /etc/openstack_deploy)22:25
jrosserthat means that the controllers are in the haproxy ansible group https://github.com/openstack/openstack-ansible/blob/master/playbooks/haproxy-install.yml#L2322:25
jrosserso inventory_manage blah blah and try to find why that is22:26
*** CeeMac has quit IRC22:26
*** johnsom has quit IRC22:27
*** tinwood_ has joined #openstack-ansible22:28
*** rh-jelabarre has quit IRC22:28
*** tinwood has quit IRC22:28
ThiagoCMCHmm... Ok...22:28
*** johnsom has joined #openstack-ansible22:29
*** CeeMac has joined #openstack-ansible22:30
spateljrosser: may be this is what going on - https://bugs.launchpad.net/neutron/+bug/157522522:32
openstackspatel: Error: Could not gather data from Launchpad for bug #1575225 (https://launchpad.net/bugs/1575225). The error has been logged22:32
ThiagoCMCjrosser, found the problem, it was my /etc/openstack_deploy subdir with more files declaring it.  LOL - Sorry about the buzz!22:37
admin0:)22:37
ThiagoCMC:-P22:38
ThiagoCMCInteresting... I don't know how to remove hosts from a deployed environment... Just gonna start from scratch, again.  lol22:38
jrosserscripts/inventory_manage22:39
admin0use inventory manage -r to delete from inventory, cleanup the yaml and delete the containers22:39
ThiagoCMCAmazing! Thanks!22:39
*** cshen has quit IRC22:49
spateljrosser: i think i found my issue, let me try that and see if that fix flooding.23:03
spatelproblem solved, my bad i created loop with veth pair and didn't realized that.. :)23:07
spatelDo you default keep IPv6 enabled on all servers?23:08
*** spatel has quit IRC23:18

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!