| *** takamatsu has joined #openstack | 00:16 | |
| *** takamatsu has quit IRC | 00:31 | |
| *** cyberworm54 has quit IRC | 00:45 | |
| *** cyberworm54 has joined #openstack | 00:47 | |
| *** brokencycle has quit IRC | 00:54 | |
| *** renich has quit IRC | 00:55 | |
| *** skyraven has joined #openstack | 01:07 | |
| *** skyraven has quit IRC | 01:12 | |
| *** cyberworm54 has quit IRC | 01:12 | |
| *** cyberworm54 has joined #openstack | 01:14 | |
| *** slaweq has joined #openstack | 01:29 | |
| *** TxGirlGeek has joined #openstack | 01:38 | |
| *** Lucas_Gray has quit IRC | 01:38 | |
| *** TxGirlGeek has quit IRC | 01:44 | |
| *** daknob has joined #openstack | 02:00 | |
| *** cyberworm54 has quit IRC | 02:03 | |
| *** cyberworm54 has joined #openstack | 02:05 | |
| *** eggs has joined #openstack | 02:55 | |
| *** skyraven has joined #openstack | 03:08 | |
| jonmasters | DHE: there’s a kernel bug in the CentOS7.7 kernel that causes netfilter nat rules to not run within the qrouter netns namespace | 03:11 |
|---|---|---|
| jonmasters | And that was a huge pain in the rear to find | 03:12 |
| DHE | got a link? | 03:12 |
| jonmasters | Nope | 03:12 |
| jonmasters | I found it this pm | 03:12 |
| *** skyraven has quit IRC | 03:12 | |
| jonmasters | But there are other older issues in RHEL that I did see which were similar | 03:13 |
| jonmasters | Switched to kernel-ml temporarily | 03:13 |
| jonmasters | Works like a charm | 03:13 |
| *** rvd has joined #openstack | 03:13 | |
| jonmasters | I was crawling up and down the entire stack on this one. It was stuff I’d usually enjoy as fun but I also really wanted to get this working. I was very close to giving up on OpenStack yet again as too complex to rely on | 03:14 |
| jonmasters | The problem with things like how l3 agents are implemented is that there’s essentially no way to debug short of manually poking at the netns. I started with manual iptables rules I added to confirm that the nat chains weren’t getting hit but then it quickly went much deeper | 03:16 |
| jonmasters | And I hasten to add that yesterday I had no idea how any of the l3 or openvswitch stuff worked in OpenStack. I’d consider myself pretty good at understanding it today | 03:17 |
| jonmasters | Nothing like intense annoyance at a bug to fuel the learning :) | 03:18 |
| jonmasters | But hey, two all nighters and I have kolla-ansible running on a mixed arch cluster of x86 and arm nodes :) | 03:20 |
| *** wings has joined #openstack | 03:45 | |
| *** davee__ has joined #openstack | 04:03 | |
| *** drznpy has quit IRC | 04:03 | |
| *** drznpy has joined #openstack | 04:04 | |
| *** cyclaw has quit IRC | 04:04 | |
| *** davee_ has quit IRC | 04:04 | |
| *** davee___ has quit IRC | 04:04 | |
| *** gentoorax has joined #openstack | 04:05 | |
| *** davee_ has joined #openstack | 04:05 | |
| *** eggs has quit IRC | 04:13 | |
| *** cyberworm54 has quit IRC | 04:19 | |
| *** cyberworm54 has joined #openstack | 04:20 | |
| *** skyraven has joined #openstack | 05:09 | |
| *** skyraven has quit IRC | 05:13 | |
| *** skyraven has joined #openstack | 05:16 | |
| *** wings has quit IRC | 05:29 | |
| *** links has joined #openstack | 05:47 | |
| *** wings has joined #openstack | 06:00 | |
| *** surpatil has joined #openstack | 06:05 | |
| *** rcernin has quit IRC | 06:12 | |
| *** SurajPatil has joined #openstack | 06:16 | |
| *** surpatil has quit IRC | 06:18 | |
| *** surpatil has joined #openstack | 06:19 | |
| *** slaweq has quit IRC | 06:20 | |
| *** SurajPatil has quit IRC | 06:21 | |
| *** cgfbee has quit IRC | 06:27 | |
| *** soniya29 has joined #openstack | 06:40 | |
| *** ymasson has quit IRC | 06:43 | |
| *** cah_link has joined #openstack | 07:23 | |
| *** jtomasek has joined #openstack | 07:37 | |
| *** avivgt has joined #openstack | 07:53 | |
| *** damien_r has joined #openstack | 07:55 | |
| *** tesseract has joined #openstack | 08:11 | |
| *** bengates has joined #openstack | 08:14 | |
| *** bengates has quit IRC | 08:16 | |
| *** bengates_ has joined #openstack | 08:17 | |
| *** shortparry has joined #openstack | 08:17 | |
| *** bengates_ has quit IRC | 08:18 | |
| *** bengates has joined #openstack | 08:19 | |
| *** rpittau|afk is now known as rpittau | 08:39 | |
| *** jklare has joined #openstack | 08:47 | |
| *** jangutter has joined #openstack | 08:54 | |
| *** bengates_ has joined #openstack | 09:00 | |
| *** gentoorax has quit IRC | 09:00 | |
| *** gentoora- has joined #openstack | 09:00 | |
| *** gentoora- is now known as gentoorax | 09:01 | |
| *** electrofelix has joined #openstack | 09:03 | |
| *** bengates has quit IRC | 09:04 | |
| *** rha has quit IRC | 09:06 | |
| *** avivgta has joined #openstack | 09:11 | |
| *** Relih has joined #openstack | 09:23 | |
| *** slaweq has joined #openstack | 09:30 | |
| *** rha has joined #openstack | 09:36 | |
| *** jawad_axd has joined #openstack | 09:43 | |
| *** alexmcleod has joined #openstack | 09:45 | |
| *** SurajPatil has joined #openstack | 09:49 | |
| *** surpatil has quit IRC | 09:52 | |
| *** pcaruana has joined #openstack | 09:53 | |
| *** surpatil has joined #openstack | 10:01 | |
| *** SurajPatil has quit IRC | 10:04 | |
| *** pcaruana has quit IRC | 10:11 | |
| *** pcaruana has joined #openstack | 10:12 | |
| *** SurajPatil has joined #openstack | 10:26 | |
| *** mulbc has joined #openstack | 10:26 | |
| *** surpatil has quit IRC | 10:28 | |
| *** slaweq has quit IRC | 10:31 | |
| *** aedc has joined #openstack | 10:32 | |
| *** surpatil has joined #openstack | 10:53 | |
| *** SurajPatil has quit IRC | 10:56 | |
| *** surpatil is now known as surpatil|afk | 11:11 | |
| *** rvd has quit IRC | 11:15 | |
| *** admin0 has quit IRC | 11:33 | |
| *** surpatil|afk has quit IRC | 11:36 | |
| *** Lucas_Gray has joined #openstack | 11:38 | |
| *** slaweq has joined #openstack | 11:41 | |
| *** mulbc has quit IRC | 11:53 | |
| *** mulbc has joined #openstack | 11:55 | |
| *** S4ren has joined #openstack | 11:56 | |
| *** slaweq has quit IRC | 11:57 | |
| *** surpatil|afk has joined #openstack | 11:58 | |
| *** verdurin has quit IRC | 12:15 | |
| *** BakaKuna has joined #openstack | 12:16 | |
| *** dviroel has joined #openstack | 12:17 | |
| *** verdurin has joined #openstack | 12:17 | |
| *** surpatil|afk is now known as surpatil | 12:31 | |
| *** jc_ has quit IRC | 12:33 | |
| *** leanderthal has joined #openstack | 12:35 | |
| *** lborda has joined #openstack | 12:59 | |
| *** aconole has joined #openstack | 13:00 | |
| *** SurajPatil has joined #openstack | 13:01 | |
| *** surpatil has quit IRC | 13:04 | |
| *** morazi has joined #openstack | 13:04 | |
| *** mcornea has joined #openstack | 13:23 | |
| *** surpatil has joined #openstack | 13:29 | |
| *** SurajPatil has quit IRC | 13:32 | |
| *** soniya29 has quit IRC | 13:41 | |
| *** Lucas_Gray has quit IRC | 13:53 | |
| *** Lucas_Gray has joined #openstack | 13:55 | |
| *** markllama has joined #openstack | 13:56 | |
| *** markllama has left #openstack | 13:56 | |
| *** petevg has joined #openstack | 14:02 | |
| *** Goneri has quit IRC | 14:03 | |
| *** admin0 has joined #openstack | 14:03 | |
| *** Lucas_Gray has quit IRC | 14:10 | |
| *** jtomasek has quit IRC | 14:13 | |
| *** jtomasek has joined #openstack | 14:15 | |
| *** schwicht has joined #openstack | 14:19 | |
| *** arnoldoree has joined #openstack | 14:22 | |
| *** surpatil has quit IRC | 14:34 | |
| *** links has quit IRC | 14:38 | |
| *** cah_link has quit IRC | 14:39 | |
| *** mcornea has quit IRC | 14:48 | |
| *** aedc has quit IRC | 14:51 | |
| *** aedc has joined #openstack | 14:51 | |
| *** jawad_axd has quit IRC | 14:58 | |
| *** jawad_axd has joined #openstack | 14:59 | |
| *** spiette has quit IRC | 15:00 | |
| *** mcornea has joined #openstack | 15:02 | |
| *** spiette has joined #openstack | 15:03 | |
| *** jawad_axd has quit IRC | 15:04 | |
| *** aedc_ has joined #openstack | 15:08 | |
| *** Goneri has joined #openstack | 15:10 | |
| *** imega has joined #openstack | 15:12 | |
| *** aedc has quit IRC | 15:12 | |
| *** lborda has quit IRC | 15:49 | |
| *** iokiwi0 has joined #openstack | 15:50 | |
| *** iokiwi has quit IRC | 15:51 | |
| *** iokiwi0 is now known as iokiwi | 15:51 | |
| *** electrofelix has quit IRC | 15:56 | |
| *** dayou has quit IRC | 16:07 | |
| *** jawad_axd has joined #openstack | 16:08 | |
| *** jawad_axd has quit IRC | 16:09 | |
| *** BakaKuna has quit IRC | 16:10 | |
| *** arxcruz is now known as arxcruz|off | 16:11 | |
| *** k_mouza has joined #openstack | 16:12 | |
| *** gyee has joined #openstack | 16:15 | |
| *** damien_r has quit IRC | 16:15 | |
| *** TxGirlGeek has joined #openstack | 16:20 | |
| *** jmlowe has quit IRC | 16:27 | |
| *** ymasson has joined #openstack | 16:28 | |
| *** bengates_ has quit IRC | 16:30 | |
| *** eggs has joined #openstack | 16:35 | |
| *** soniya29 has joined #openstack | 16:37 | |
| *** leanderthal has quit IRC | 16:38 | |
| *** k_mouza has quit IRC | 16:39 | |
| *** alexmcleod has quit IRC | 16:42 | |
| *** dtrainor has joined #openstack | 16:46 | |
| *** shortparry_ has joined #openstack | 16:49 | |
| *** shortparry has quit IRC | 16:51 | |
| *** avivgta has quit IRC | 16:53 | |
| *** jathan has joined #openstack | 16:55 | |
| *** rpittau is now known as rpittau|afk | 16:58 | |
| *** morazi has quit IRC | 16:59 | |
| *** morazi has joined #openstack | 17:02 | |
| *** damien_r has joined #openstack | 17:04 | |
| *** avivgt has quit IRC | 17:08 | |
| *** mrtadis has quit IRC | 17:09 | |
| *** benfelin has joined #openstack | 17:12 | |
| *** spiral has joined #openstack | 17:17 | |
| *** dayou has joined #openstack | 17:20 | |
| *** damien_r has quit IRC | 17:20 | |
| *** avivgt has joined #openstack | 17:29 | |
| *** sshnaidm is now known as sshnaidm|afk | 17:31 | |
| *** tomgray has quit IRC | 17:35 | |
| *** soniya29 has quit IRC | 17:46 | |
| *** E1ephant has joined #openstack | 17:51 | |
| *** jmlowe has joined #openstack | 17:59 | |
| *** avivgta has joined #openstack | 18:07 | |
| *** pcaruana has quit IRC | 18:07 | |
| *** bobmel has joined #openstack | 18:14 | |
| *** bobmel has quit IRC | 18:27 | |
| *** tesseract has quit IRC | 18:38 | |
| *** imega has quit IRC | 18:47 | |
| *** S4ren has quit IRC | 18:49 | |
| *** BakaKuna has joined #openstack | 18:50 | |
| *** morazi has quit IRC | 18:51 | |
| *** jmlowe has quit IRC | 19:01 | |
| *** jmlowe has joined #openstack | 19:03 | |
| *** morazi has joined #openstack | 19:04 | |
| *** jstv2 has joined #openstack | 19:11 | |
| *** gentoorax is now known as gentoorax_away | 19:21 | |
| *** gentoorax_away is now known as gentoorax | 19:21 | |
| *** gentoorax is now known as cyclaw | 19:21 | |
| *** iokiwi has quit IRC | 19:24 | |
| *** lifeless has quit IRC | 19:24 | |
| *** lifeless has joined #openstack | 19:25 | |
| *** dayou has quit IRC | 19:27 | |
| *** dayou has joined #openstack | 19:28 | |
| *** iokiwi has joined #openstack | 19:29 | |
| *** ircuser-1 has joined #openstack | 19:39 | |
| *** morazi has quit IRC | 19:41 | |
| *** slaweq has joined #openstack | 19:44 | |
| *** shibboleth has joined #openstack | 19:49 | |
| *** ianychoi has quit IRC | 20:04 | |
| *** cah_link has joined #openstack | 20:09 | |
| *** igordc has joined #openstack | 20:13 | |
| *** igordc has quit IRC | 20:15 | |
| *** jmlowe has quit IRC | 20:16 | |
| *** random_yanek has quit IRC | 20:17 | |
| *** ianychoi has joined #openstack | 20:19 | |
| *** Lucas_Gray has joined #openstack | 20:21 | |
| *** jeh has joined #openstack | 20:23 | |
| *** random_yanek has joined #openstack | 20:23 | |
| *** jmlowe has joined #openstack | 20:35 | |
| *** imega has joined #openstack | 20:35 | |
| *** scousineau has joined #openstack | 20:38 | |
| *** scousineau has quit IRC | 20:42 | |
| *** BakaKuna has quit IRC | 20:45 | |
| *** brokencycle has joined #openstack | 20:47 | |
| *** avivgt has quit IRC | 20:52 | |
| *** zbr has quit IRC | 21:05 | |
| *** zbr has joined #openstack | 21:07 | |
| *** zbr has quit IRC | 21:14 | |
| *** cah_link has quit IRC | 21:18 | |
| *** jonaspaulo has joined #openstack | 21:21 | |
| *** zbr has joined #openstack | 21:22 | |
| *** zbr has quit IRC | 21:26 | |
| *** TxGirlGeek has quit IRC | 21:27 | |
| *** TxGirlGeek has joined #openstack | 21:27 | |
| *** TxGirlGeek has quit IRC | 21:27 | |
| *** zbr has joined #openstack | 21:30 | |
| *** TxGirlGeek has joined #openstack | 21:34 | |
| *** zbr_ has joined #openstack | 21:37 | |
| *** zbr has quit IRC | 21:39 | |
| *** jtro has joined #openstack | 21:49 | |
| jtro | I have an instance whose root disk needs a fsck. How do I make that happen if I can't unmount the root disk? | 21:49 |
| umbSublime | jtro: you can boot in rescue, or force an fsck on next boot | 21:51 |
| jtro | umbsublime: This is an ubuntu node and it's not finishing boot so I don't think I can force fsck on next boot. | 21:52 |
| jtro | How do I boot in Rescue? | 21:52 |
| *** jstv2 has quit IRC | 21:52 | |
| umbSublime | openstack server rescue [--image <image uuid>] <server uuid> | 21:53 |
| jtro | Awesome, thanks a ton. Wasn't aware this was a feature. | 21:54 |
| umbSublime | ^^ | 21:54 |
| umbSublime | saved me many times | 21:54 |
| jtro | Instance <name> cannot be rescued: Cannot rescue a volume-backed instance (HTTP 400) (Request-ID: req-7a8f58fb-46b3-47e6-a803-d016a110e122) | 21:56 |
| jtro | rip :( | 21:56 |
| *** slaweq has quit IRC | 21:56 | |
| jtro | umbsublime: any other options? | 21:58 |
| umbSublime | maybe you can boot a new instance (not volume backed) and mount the volume there | 21:58 |
| jtro | I can't unmount the root volume though. | 21:58 |
| umbSublime | If it's just to backup data before scrapping the borked VM maybe you can take a snapshot of the volume and mount/fsck/backup | 21:59 |
| jtro | I was actually hoping to get the vps back live. :( | 21:59 |
| umbSublime | what do you see in VNC or `openstack console log show <instance uuid>` | 22:00 |
| umbSublime | jtro: are you the operator of the openstack deployment ? | 22:01 |
| jtro | Guest does not have a console available. | 22:01 |
| jtro | It's currently off. | 22:01 |
| jtro | I am as close as this stack is ever going to get to an operator. :D | 22:01 |
| umbSublime | what's currently off ? | 22:01 |
| jtro | The instance. | 22:02 |
| *** zbr_ has quit IRC | 22:02 | |
| umbSublime | start it and check the output of the command I pasted | 22:02 |
| *** zbr has joined #openstack | 22:02 | |
| *** TxGirlGeek has quit IRC | 22:04 | |
| jtro | http://paste.openstack.org/show/788096/ | 22:04 |
| umbSublime | jtro: asside from ssh on the compute node the only thing I can think of is stopping the VM and mounting the volume on another instance | 22:09 |
| jtro | It doesn't complete boot | 22:10 |
| jtro | So no SSH. | 22:10 |
| jtro | And it's the root disk, so no mounting. :( | 22:10 |
| umbSublime | yes that's why I'm suggesting you create a NEW vm and mount the volume there | 22:10 |
| idlemind | can you snap the volume | 22:11 |
| *** slaweq has joined #openstack | 22:11 | |
| umbSublime | yah I've suggested that, but if he wants to get the original VM back up that won't help :/ altough it's a good option to backup | 22:11 |
| jtro | So fsck the snapshot and build a new instance from it? | 22:13 |
| umbSublime | jtro there are many options on the table. That is one of them yes | 22:13 |
| *** arnoldoree has quit IRC | 22:13 | |
| *** mcornea has quit IRC | 22:13 | |
| jtro | This seems like the most straight foward, I'll give it a shot. | 22:13 |
| jtro | Thanks man. | 22:13 |
| umbSublime | another would be: 1. Stop the broken VM. 2. Create a new VM 3. Mount the broken VMs volume on VM from step 2 | 22:14 |
| umbSublime | 4. try to fix 5. if it works, unmount from step 2 VM and reboot broken VM | 22:14 |
| jtro | But I can't double mount the volume. | 22:14 |
| jtro | And I can't unmount boot volume. | 22:14 |
| umbSublime | Ahh right, I would usualy manually delete the attachment entry in DB when I did that last time (I don't suggest that approach) | 22:15 |
| *** slaweq has quit IRC | 22:15 | |
| jtro | Yeah, I was reading this: https://www.manago.cloud/en/blog/how-repair-volume-backed-windows-instance | 22:15 |
| jtro | Seems to suggest something similar. | 22:15 |
| jtro | Why wouldn't you recommend it? | 22:16 |
| umbSublime | well it's error prone for 1 | 22:16 |
| umbSublime | If there are ways around changing stuff in DB i usualy prefer that | 22:16 |
| umbSublime | in the end it's all up to you :p | 22:16 |
| idlemind | it's not a capability that's been explicitly accounted for is the risk | 22:17 |
| idlemind | so you can't predict any knock-on costs / issues it might cause | 22:17 |
| jtro | The snapshot is in an error state. :( | 22:18 |
| umbSublime | what type of volume is it ? | 22:19 |
| jtro | It's ext4 on a ceph cluster, I think. | 22:19 |
| umbSublime | you could probably run ceph commands directly to get a copy of it | 22:20 |
| jtro | Yeah, I'll have to look into that I guess. | 22:21 |
| *** skyraven has quit IRC | 22:22 | |
| *** TxGirlGeek has joined #openstack | 22:23 | |
| *** zbr has quit IRC | 22:25 | |
| *** zbr has joined #openstack | 22:25 | |
| *** Relih has quit IRC | 22:25 | |
| *** TxGirlGeek has quit IRC | 22:28 | |
| umbSublime | best of luck ! | 22:28 |
| *** zbr has quit IRC | 22:30 | |
| *** TxGirlGeek has joined #openstack | 22:33 | |
| *** rcernin has joined #openstack | 22:40 | |
| *** TxGirlGeek has quit IRC | 22:42 | |
| *** Goneri has quit IRC | 22:46 | |
| *** bobmel has joined #openstack | 22:47 | |
| *** dviroel has quit IRC | 22:51 | |
| *** bobmel has quit IRC | 22:52 | |
| *** jtro has quit IRC | 23:00 | |
| *** TxGirlGeek has joined #openstack | 23:00 | |
| *** zbr has joined #openstack | 23:04 | |
| *** bobmel has joined #openstack | 23:08 | |
| *** skyraven has joined #openstack | 23:08 | |
| *** zbr has quit IRC | 23:09 | |
| *** zbr_ has joined #openstack | 23:09 | |
| *** Goneri has joined #openstack | 23:10 | |
| *** TxGirlGeek has quit IRC | 23:11 | |
| *** zbr_ has quit IRC | 23:14 | |
| *** schwicht has quit IRC | 23:15 | |
| *** skyraven has quit IRC | 23:18 | |
| *** bobmel has quit IRC | 23:20 | |
| *** jonaspaulo has quit IRC | 23:20 | |
| *** zbr has joined #openstack | 23:21 | |
| *** aconole has quit IRC | 23:23 | |
| *** TxGirlGeek has joined #openstack | 23:28 | |
| *** jathan has quit IRC | 23:36 | |
| *** Lucas_Gray has quit IRC | 23:53 | |
| *** schwicht has joined #openstack | 23:54 | |
| *** shortparry__ has joined #openstack | 23:55 | |
| *** KeithMnemonic has quit IRC | 23:56 | |
| *** KeithMnemonic has joined #openstack | 23:57 | |
| *** TxGirlGeek has quit IRC | 23:57 | |
| *** KeithMnemonic has quit IRC | 23:58 | |
| *** KeithMnemonic has joined #openstack | 23:58 | |
| *** shortparry_ has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!