this occurs on both hosts at the very same second, which leads me to believe it's not a hardware problem:
("segfault" is the name of the server. don't ask.)
Dec 29 14:00:10 segfault kernel: [403596.851539] NFS: Cache request denied due to non-unique superblock keys
Dec 29 14:00:10 segfault kernel: [403596.930361] ata1: hard resetting link
Dec 29 14:00:10 segfault kernel: [403597.237778] ata1: SATA link down (SStatus 0 SControl 300)
Dec 29 14:00:10 segfault kernel: [403597.239721] ata1: EH complete
Dec 29 14:00:10 segfault kernel: [403597.241759] ata2: hard resetting link
Dec 29 14:00:10 segfault kernel: [403597.548726] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Dec 29 14:00:10 segfault kernel: [403597.559061] ata2.00: configured for UDMA/133
Dec 29 14:00:10 segfault kernel: [403597.559066] ata2: EH complete
Dec 29 14:00:10 segfault kernel: [403597.559279] ata3: hard resetting link
Dec 29 14:00:11 segfault kernel: [403597.866689] ata3: SATA link down (SStatus 0 SControl 300)
Dec 29 14:00:11 segfault kernel: [403597.868680] ata3: EH complete
Dec 29 14:00:11 segfault kernel: [403597.868928] ata4: hard resetting link
Dec 29 14:00:11 segfault kernel: [403598.176588] ata4: SATA link down (SStatus 0 SControl 300)
Dec 29 14:00:11 segfault kernel: [403598.178470] ata4: EH complete
Dec 29 14:00:11 segfault kernel: [403598.178707] ata5: hard resetting link
Dec 29 14:00:11 segfault kernel: [403598.485535] ata5: SATA link down (SStatus 0 SControl 300)
Dec 29 14:00:11 segfault kernel: [403598.487472] ata5: EH complete
Dec 29 14:00:11 segfault kernel: [403598.487656] ata6: hard resetting link
Dec 29 14:00:12 segfault kernel: [403598.795437] ata6: SATA link down (SStatus 0 SControl 300)
Dec 29 14:00:12 segfault kernel: [403598.797325] ata6: EH complete
Dec 29 14:00:12 segfault kernel: [403598.797583] ata7: hard resetting link
Dec 29 14:00:12 segfault kernel: [403599.105616] ata7: SATA link down (SStatus 0 SControl 300)
Dec 29 14:00:12 segfault kernel: [403599.107307] ata7: EH complete
Dec 29 14:00:12 segfault kernel: [403599.109318] ata8: hard resetting link
Dec 29 14:00:12 segfault kernel: [403599.416217] ata8: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Dec 29 14:00:12 segfault kernel: [403599.416650] ata8.00: configured for UDMA/66
Dec 29 14:00:12 segfault kernel: [403599.416769] ata8: EH complete
Dec 29 14:00:12 segfault kernel: [403599.416922] ata9: hard resetting link
Dec 29 14:00:13 segfault kernel: [403599.722240] ata9: SATA link down (SStatus 0 SControl 300)
Dec 29 14:00:13 segfault kernel: [403599.722256] ata9: EH complete
Dec 29 14:00:13 segfault kernel: [403599.722545] ata10: hard resetting link
Dec 29 14:00:13 segfault kernel: [403600.027130] ata10: SATA link down (SStatus 0 SControl 300)
Dec 29 14:00:13 segfault kernel: [403600.027234] ata10: EH complete
Dec 29 14:00:13 segfault kernel: [403600.027483] ata11.00: hard resetting link
Dec 29 14:00:13 segfault kernel: [403600.331952] ata11.01: hard resetting link
Dec 29 14:00:14 segfault kernel: [403600.787975] ata11.00: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Dec 29 14:00:14 segfault kernel: [403600.787989] ata11.01: SATA link down (SStatus 0 SControl 300)
Dec 29 14:00:14 segfault kernel: [403600.829480] ata11.00: configured for UDMA/133
Dec 29 14:00:14 segfault kernel: [403600.829487] ata11: EH complete
Dec 29 14:00:14 segfault kernel: [403600.829638] ata12.00: hard resetting link
Dec 29 14:00:14 segfault kernel: [403601.133734] ata12.01: hard resetting link
Dec 29 14:00:14 segfault kernel: [403601.589741] ata12.00: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Dec 29 14:00:14 segfault kernel: [403601.589757] ata12.01: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Dec 29 14:00:14 segfault kernel: [403601.600049] ata12.00: configured for UDMA/133
Dec 29 14:00:14 segfault kernel: [403601.604444] ata12.01: configured for UDMA/133
Dec 29 14:00:14 segfault kernel: [403601.604450] ata12: EH complete
Dec 29 14:00:14 segfault kernel: [403601.604745] ata13: hard resetting link
Dec 29 14:00:15 segfault kernel: [403601.921125] ata13: SATA link down (SStatus 0 SControl 300)
Dec 29 14:00:15 segfault kernel: [403601.921144] ata13: EH complete
Dec 29 14:00:15 segfault kernel: [403601.921300] ata14: hard resetting link
Dec 29 14:00:15 segfault kernel: [403602.377523] ata14: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Dec 29 14:00:15 segfault kernel: [403602.386842] ata14.00: configured for UDMA/100
Dec 29 14:00:15 segfault kernel: [403602.392491] ata14: EH complete
Dec 29 14:00:15 segfault kernel: [403602.392695] ata15: soft resetting link
Dec 29 14:00:15 segfault kernel: [403602.543523] ata15: EH complete
Dec 29 14:00:15 segfault kernel: [403602.543727] ata16: soft resetting link
Dec 29 14:00:16 segfault kernel: [403602.706229] ata16: EH complete
Dec 29 14:00:16 segfault vdsm Storage.LVM WARNING lvm vgs failed: 5 [] [' Volume group "78b1f41d-29cf-4e1a-a84d-fb9175f4388e" not found']
Dec 29 14:00:16 segfault kernel: [403603.273230] ata1: hard resetting link
Dec 29 14:00:16 segfault kernel: [403603.580216] ata1: SATA link down (SStatus 0 SControl 300)
Dec 29 14:00:16 segfault kernel: [403603.582803] ata1: EH complete
Dec 29 14:00:16 segfault kernel: [403603.585201] ata2: hard resetting link
Dec 29 14:00:17 segfault kernel: [403603.892210] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Dec 29 14:00:17 segfault kernel: [403603.901635] ata2.00: configured for UDMA/133
Dec 29 14:00:17 segfault kernel: [403603.901641] ata2: EH complete
Dec 29 14:00:17 segfault kernel: [403603.901873] ata3: hard resetting link
Dec 29 14:00:17 segfault kernel: [403604.209120] ata3: SATA link down (SStatus 0 SControl 300)
Dec 29 14:00:17 segfault kernel: [403604.210966] ata3: EH complete
Dec 29 14:00:17 segfault kernel: [403604.211156] ata4: hard resetting link
Dec 29 14:00:17 segfault kernel: [403604.517966] ata4: SATA link down (SStatus 0 SControl 300)
Dec 29 14:00:17 segfault kernel: [403604.519902] ata4: EH complete
Dec 29 14:00:17 segfault kernel: [403604.520157] ata5: hard resetting link
Dec 29 14:00:18 segfault kernel: [403604.826862] ata5: SATA link down (SStatus 0 SControl 300)
Dec 29 14:00:18 segfault kernel: [403604.828830] ata5: EH complete
Dec 29 14:00:18 segfault kernel: [403604.829067] ata6: hard resetting link
Dec 29 14:00:18 segfault kernel: [403605.135731] ata6: SATA link down (SStatus 0 SControl 300)
Dec 29 14:00:18 segfault kernel: [403605.137698] ata6: EH complete
Dec 29 14:00:18 segfault kernel: [403605.137887] ata7: hard resetting link
Dec 29 14:00:18 segfault kernel: [403605.445714] ata7: SATA link down (SStatus 0 SControl 300)
Dec 29 14:00:18 segfault kernel: [403605.447619] ata7: EH complete
Dec 29 14:00:18 segfault kernel: [403605.449682] ata8: hard resetting link
Dec 29 14:00:19 segfault kernel: [403605.756684] ata8: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Dec 29 14:00:19 segfault kernel: [403605.757166] ata8.00: configured for UDMA/66
Dec 29 14:00:19 segfault kernel: [403605.757266] ata8: EH complete
Dec 29 14:00:19 segfault kernel: [403605.757485] ata9: hard resetting link
Dec 29 14:00:19 segfault kernel: [403606.064643] ata9: SATA link down (SStatus 0 SControl 300)
Dec 29 14:00:19 segfault kernel: [403606.064660] ata9: EH complete
Dec 29 14:00:19 segfault kernel: [403606.064839] ata10: hard resetting link
Dec 29 14:00:19 segfault kernel: [403606.369548] ata10: SATA link down (SStatus 0 SControl 300)
Dec 29 14:00:19 segfault kernel: [403606.369562] ata10: EH complete
Dec 29 14:00:19 segfault kernel: [403606.369842] ata11.00: hard resetting link
Dec 29 14:00:20 segfault kernel: [403606.675420] ata11.01: hard resetting link
Dec 29 14:00:20 segfault kernel: [403607.131342] ata11.00: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Dec 29 14:00:20 segfault kernel: [403607.131355] ata11.01: SATA link down (SStatus 0 SControl 300)
Dec 29 14:00:20 segfault kernel: [403607.248752] ata11.00: configured for UDMA/133
Dec 29 14:00:20 segfault kernel: [403607.248759] ata11: EH complete
Dec 29 14:00:20 segfault kernel: [403607.249045] ata12.00: hard resetting link
Dec 29 14:00:20 segfault kernel: [403607.554194] ata12.01: hard resetting link
Dec 29 14:00:21 segfault kernel: [403608.011215] ata12.00: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Dec 29 14:00:21 segfault kernel: [403608.011244] ata12.01: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Dec 29 14:00:21 segfault kernel: [403608.021439] ata12.00: configured for UDMA/133
Dec 29 14:00:21 segfault kernel: [403608.028239] ata12.01: configured for UDMA/133
Dec 29 14:00:21 segfault kernel: [403608.028243] ata12: EH complete
Dec 29 14:00:21 segfault kernel: [403608.028442] ata13: hard resetting link
Dec 29 14:00:21 segfault kernel: [403608.345178] ata13: SATA link down (SStatus 0 SControl 300)
Dec 29 14:00:21 segfault kernel: [403608.345195] ata13: EH complete
Dec 29 14:00:21 segfault kernel: [403608.345443] ata14: hard resetting link
Dec 29 14:00:22 segfault kernel: [403608.800805] ata14: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
Dec 29 14:00:22 segfault kernel: [403608.810109] ata14.00: configured for UDMA/100
Dec 29 14:00:22 segfault kernel: [403608.815632] ata14: EH complete
Dec 29 14:00:22 segfault kernel: [403608.815863] ata15: soft resetting link
Dec 29 14:00:22 segfault kernel: [403608.966859] ata15: EH complete
Dec 29 14:00:22 segfault kernel: [403608.967054] ata16: soft resetting link
Dec 29 14:00:22 segfault kernel: [403609.130010] ata16: EH complete
Dec 29 14:00:22 segfault vdsm Storage.LVM WARNING lvm vgs failed: 5 [] [' Volume group "f0175c0c-75f1-4518-ba77-f7476171f6c6" not found']
Dec 29 14:00:22 segfault vdsm Storage.StorageDomain WARNING Resource namespace f0175c0c-75f1-4518-ba77-f7476171f6c6_imageNS already registered
Dec 29 14:00:22 segfault vdsm Storage.StorageDomain WARNING Resource namespace f0175c0c-75f1-4518-ba77-f7476171f6c6_volumeNS already registered
Dec 29 14:00:23 segfault vdsm Storage.LVM WARNING lvm vgs failed: 5 [] [' Volume group "78b1f41d-29cf-4e1a-a84d-fb9175f4388e" not found']