[PATCH v2] hugetlbfs: Fix integer overflow check in hugetlbfs_file_mmap()

From: Linke Li
Date: Thu Jul 20 2023 - 09:49:50 EST


From: Linke Li <lilinke99@xxxxxxxxx>

```
vma_len = (loff_t)(vma->vm_end - vma->vm_start);
len = vma_len + ((loff_t)vma->vm_pgoff << PAGE_SHIFT);
/* check for overflow */
if (len < vma_len)
return -EINVAL;
```

There is a signed integer overflow in the code, which is undefined
behavior according to the C stacnard. Although kernel disables some
optimizations by using the "-fno-strict-overflow" option, there is
still a risk.

Using macro "check_add_overflow" to do the overflow check can
effectively detect integer overflow and avoid any undefined behavior.

Signed-off-by: Linke Li <lilinke99@xxxxxxxxx>
---
fs/hugetlbfs/inode.c | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 7b17ccfa039d..60f3010b0f71 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -155,9 +155,7 @@ static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma)
return -EINVAL;

vma_len = (loff_t)(vma->vm_end - vma->vm_start);
- len = vma_len + ((loff_t)vma->vm_pgoff << PAGE_SHIFT);
- /* check for overflow */
- if (len < vma_len)
+ if (check_add_overflow(vma_len, (loff_t)vma->vm_pgoff << PAGE_SHIFT, &len))
return -EINVAL;

inode_lock(inode);
--
2.25.1