| jrosser | morning | 09:24 |
|---|---|---|
| noonedeadpunk | \o/ | 09:31 |
| damiandabrowski | hi! | 10:06 |
| damiandabrowski | yesterday I noticed that it's not possible to use "item is defined" conditional for a loop: https://paste.openstack.org/show/bQkDFF9AFI3a4jktMH4M/ | 10:09 |
| damiandabrowski | did you know about it? do you have any alternative in mind? | 10:09 |
| damiandabrowski | i tried this trick with setting default value but it also doesn't work | 10:10 |
| damiandabrowski | however, everything works fine when using "with_items" | 10:10 |
| jrosser | damiandabrowski: well with_items and loop are not directly equivalent https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_loops.html#with-items | 10:15 |
| damiandabrowski | yeah, i tried using flatten but it didn't work | 10:16 |
| jrosser | interesting | 10:16 |
| jrosser | which ansible version are you using there? | 10:17 |
| damiandabrowski | s/didn't work/didn't solve the issue/ | 10:18 |
| damiandabrowski | 2.13.4 | 10:18 |
| jrosser | hmmm https://paste.opendev.org/show/bIO3iaQM6wMRiZ6J2uSB/ | 10:19 |
| damiandabrowski | O.o that's weird | 10:21 |
| jrosser | thats just regular ansible in a fresh venv, nothing to do with OSA install at all | 10:22 |
| damiandabrowski | something was wrong in my /root/.ansible/plugins (it's my AIO environment) | 10:22 |
| damiandabrowski | i removed this folder and now everything works as expected | 10:22 |
| jrosser | wow :) surreal result that gave | 10:22 |
| damiandabrowski | but the interesting think is that I didn't do anything special on that VM(except deploying openstack and executing ./run_tests.sh on several repositories) | 10:25 |
| damiandabrowski | anyway, thanks for your help! yesterday i spent few hours trying to figure out what is going on :D | 10:25 |
| jrosser | oh doesnt run_tests.sh on things like roles use tox to setup venvs and ansible? | 10:26 |
| jrosser | i can totally see that putting things in ~root/.ansible | 10:26 |
| damiandabrowski | yeah, exactly | 10:27 |
| damiandabrowski | I'm trying to find what part of plugins directory broke my playbook | 10:28 |
| damiandabrowski | so I noticed that removing /root/.ansible/plugins fixed issue only for ansible-playbook command. openstack-ansible still didn't work | 10:46 |
| damiandabrowski | to fix openstack-ansible I had to comment out: `export ANSIBLE_STRATEGY="${ANSIBLE_STRATEGY:-openstack.osa.linear}"` in /usr/local/bin/openstack-ansible.rc | 10:46 |
| damiandabrowski | that's getting really weird | 10:46 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-haproxy_server master: Prepare haproxy role for separated haproxy config https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/871188 | 13:50 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: Prepare service roles for separated haproxy config https://review.opendev.org/c/openstack/openstack-ansible/+/871189 | 13:52 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: Prepare service roles for separated haproxy config https://review.opendev.org/c/openstack/openstack-ansible/+/871189 | 13:54 |
| jrosser | NeilHanlon: do you know if there will be cloud-init v22.2+ in Rocky9 ? | 16:01 |
| NeilHanlon | unlikely, seems centos stream 9 is still on 22.1 | 16:03 |
| NeilHanlon | are there any specific patches? I can try and get them into c9s | 16:04 |
| NeilHanlon | I have this one open for c8s which is also valid for c9s; https://bugzilla.redhat.com/show_bug.cgi?id=2163657 | 16:04 |
| spatel | Question on rescue instance, first time trying it and its asking me to select image. should i be selecting same image which i used for originally? | 16:08 |
| spatel | How does rescues stuff work? | 16:08 |
| noonedeadpunk | spatel: it depends :) | 16:11 |
| noonedeadpunk | depends if you're booted from epehemeral or from volume | 16:12 |
| spatel | ? | 16:12 |
| spatel | I booted from ephemeral from ceph backend | 16:12 |
| noonedeadpunk | as for BFV you might need to have a specific tags set | 16:12 |
| noonedeadpunk | yeah, then you can use same image | 16:12 |
| jrosser | NeilHanlon: we run across this sort of thing https://bugzilla.redhat.com/show_bug.cgi?id=2036060 | 16:12 |
| jrosser | -ish | 16:12 |
| spatel | noonedeadpunk my ceph storage went down because of power and after that my VMs started complaining - blk_update_request: I/O error, dev sda, sector 233000 op 0x1: (WRITE) flags 0x800 phys_seg 8 prio class 0 | 16:16 |
| spatel | Trying to run fsck using rescue disk.. | 16:16 |
| noonedeadpunk | basically what it will do - it will add image as first drive to the VM and will boot it | 16:16 |
| spatel | I did boot from rescue but getting same error I/O error.. | 16:16 |
| noonedeadpunk | well, you can try using different image then.... | 16:18 |
| noonedeadpunk | But is ceph cluster reporting as healthy? | 16:18 |
| spatel | Ceph is showing all good. | 16:18 |
| spatel | I can spin up new VMs etc.. all good but only old vms impacted with filesystem errors | 16:19 |
| noonedeadpunk | But recovery is eventually what you need | 16:22 |
| spatel | Am i missing something in ceph config.. to prevent data corruption with power outage? | 16:23 |
| noonedeadpunk | spatel: you can do try hacky way, like rpd map on hypervisor which will create device | 16:23 |
| spatel | You guys running ceph so you must have more knowledge compare to me :) | 16:24 |
| spatel | what are the basic thing i should be care about to make sure data integrity | 16:24 |
| noonedeadpunk | um... I think you can't do anything as long as any caching is around | 16:26 |
| noonedeadpunk | like writeback for libvirt | 16:26 |
| spatel | disable writeback in libvirt correct? | 16:27 |
| noonedeadpunk | well, it would degrade performance a lot | 16:27 |
| spatel | damn!! | 16:28 |
| spatel | Does rescue image required to be password base? | 16:35 |
| spatel | It won't let you in if password not set on image correct. | 16:35 |
| jrosser | try system rescue cd or something like that | 16:36 |
| spatel | Let me understand rescue disk is just like CD rom and boot from CD and mount disk and run fsck.. am i correct? | 16:36 |
| spatel | rescue cd ? is this a special image? | 16:37 |
| jrosser | just don’t try to rescue using the same image are you are rescuing, in my experience that ends badly | 16:37 |
| spatel | Ohhh! so use other image correct? | 16:37 |
| jrosser | https://www.system-rescue.org/ | 16:37 |
| spatel | Let me upload new image and try | 16:37 |
| spatel | can i import in openstaxck? | 16:38 |
| spatel | i meant glance | 16:38 |
| jrosser | you can | 16:38 |
| spatel | doing it.. | 16:38 |
| jrosser | regardless, if you rescue with another regular image it should get the same metadata as the original, so I believe your ssh key will be resent if you had one on the original vm at boot | 16:39 |
| jrosser | system-rescue image is useful for when you know the ssh key is lost and there’s no password either | 16:40 |
| spatel | +1 | 16:40 |
| spatel | i should be using --disk-format iso correct to import image? | 16:40 |
| jrosser | spatel: i can't remember :) it was a long time ago | 16:46 |
| spatel | jrosser for testing i boot with same ubuntu image in rescue mode and ssh in using same sshkey. How do i find disk which i should run fsck? - https://paste.opendev.org/show/bnzK3ft9GPebNpVmOh6b/ | 16:46 |
| spatel | How do i know that i am really in rescue mode :) | 16:46 |
| jrosser | well that is one of the fun parts with using the same image | 16:47 |
| spatel | jrosser hmm! | 16:47 |
| spatel | let me try systemrescue ISO and see how it goes | 16:48 |
| jrosser | or debian or some other OS | 16:48 |
| jrosser | it's massively confusing about which system you are in, if you are or are not in the right system if you chroot to the "other disk" | 16:49 |
| jrosser | and the FSID is the same too just to make it even worse | 16:49 |
| jrosser | my experience previously with the FSID matching was that is a factor in if you get the right partition mounted at / once rescued | 16:50 |
| spatel | totally agreed with you. | 16:52 |
| spatel | i would prefer using centos for ubuntu and ubuntu for centos :) | 16:53 |
| spatel | easy to distinguish | 16:53 |
| spatel | jrosser FYI - https://help.switch.ch/engines/documentation/rescue-vm/ | 16:57 |
| spatel | How to import ISO to glance | 16:57 |
| spatel | $ openstack image create --file systemrescuecd-6.0.1.iso --disk-format iso --container-format bare --min-ram 800 --property hw_cdrom_bus=scsi systemrescuecd | 16:57 |
| jrosser | not sure how old that doc is but from what it says it's from before rescue supported boot-from-volume | 16:59 |
| jrosser | but anyway - if its working for you.... all good :) | 16:59 |
| spatel | hope.. I will keep it posted here.. | 17:00 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: Prepare service roles for separated haproxy config https://review.opendev.org/c/openstack/openstack-ansible/+/871189 | 17:18 |
| spatel | jrosser so rescue disk works - https://ibb.co/y84BLGY | 18:10 |
| spatel | what should i understand of it? | 18:10 |
| spatel | filesystem not recoverable? | 18:11 |
| noonedeadpunk | are you really sure ceph is healthy? | 18:14 |
| jrosser | ^ this | 18:14 |
| noonedeadpunk | as that does look to me as some PG is inactive | 18:14 |
| noonedeadpunk | `ceph health detail`? | 18:15 |
| spatel | noonedeadpunk - https://paste.opendev.org/show/bnG8rvXJydADTZknd2QD/ | 18:21 |
| spatel | https://paste.opendev.org/show/bRZFgPiNem03oknFJDbi/ | 18:21 |
| spatel | i am not seeing anything wrong in ceph | 18:22 |
| spatel | mgr crash is acceptable because everything got crash | 18:22 |
| spatel | pgs: 258 active+clean | 18:22 |
| spatel | Are there any command to validate ceph data or scrubbing? | 18:24 |
| spatel | Am i missing anything here? | 18:59 |
| NeilHanlon | jrosser: i'm going to validate some stuff but overall, i will be requesting Red Hat rebase to at least cloud-init 22.2 in RHEL 9 | 19:17 |
| NeilHanlon | if you want, i can make a copr repo with it for now.. | 19:18 |
| NeilHanlon | i might do this anyways.. | 19:18 |
| jrosser | we could certainly test it to see if it deals with the trouble we've had | 19:18 |
| jrosser | we're looking at this sort of thing right now so it's a good time to dig a bit | 19:19 |
| jrosser | spatel: you don't have some trouble like MTU error when your hosts rebooted? | 19:36 |
| spatel | Noo | 19:40 |
| spatel | I did test MTU using ping -M do 192.168.1.11 -s 9000 | 19:41 |
| spatel | jrosser does it looks ok to you? - https://paste.opendev.org/show/bWlGk4iA1uSCJ3KCwwSj/ | 19:42 |
| spatel | 9000 not pinging up 8972 works | 19:42 |
| spatel | I have setup MTU 9000 on all my interfaces | 19:42 |
| NeilHanlon | jrosser: https://copr.fedorainfracloud.org/coprs/neil/cloud-init/ - can `dnf copr enable neil/cloud-init` | 19:46 |
| NeilHanlon | source: https://git.shrug.pw/rpms/cloud-init | 19:47 |
| jrosser | awesome thankyou - will give that a spin tomorrow | 19:47 |
| spatel | I can ping 8972 that means MTU setting is good | 19:52 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!