Re: [PATCH 2/4] io_uring/cmd: Introduce SOCKET_URING_OP_GETSOCKOPT

[Breno Leitao]

On Mon, Jul 24, 2023 at 06:58:04PM -0400, Willem de Bruijn wrote:
> Breno Leitao wrote:
> > Add support for getsockopt command (SOCKET_URING_OP_GETSOCKOPT), where
> > level is SOL_SOCKET. This is leveraging the sockptr_t infrastructure,
> > where a sockptr_t is either userspace or kernel space, and handled as
> > such.
> > 
> > Function io_uring_cmd_getsockopt() is inspired by __sys_getsockopt().
> > 
> > Differently from the getsockopt(2), the optlen field is not a userspace
> > pointers. In getsockopt(2), userspace provides optlen pointer, which is
> > overwritten by the kernel.  In this implementation, userspace passes a
> > u32, and the new value is returned in cqe->res. I.e., optlen is not a
> > pointer.
> > 
> > Important to say that userspace needs to keep the pointer alive until
> > the CQE is completed.
> > 
> > Signed-off-by: Breno Leitao <leitao@xxxxxxxxxx>
> > ---
> >  include/uapi/linux/io_uring.h |  7 ++++++
> >  io_uring/uring_cmd.c          | 43 +++++++++++++++++++++++++++++++++++
> >  2 files changed, 50 insertions(+)
> > 
> > diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h
> > index 9fc7195f25df..8152151080db 100644
> > --- a/include/uapi/linux/io_uring.h
> > +++ b/include/uapi/linux/io_uring.h
> > @@ -43,6 +43,10 @@ struct io_uring_sqe {
> >  	union {
> >  		__u64	addr;	/* pointer to buffer or iovecs */
> >  		__u64	splice_off_in;
> > +		struct {
> > +			__u32	level;
> > +			__u32	optname;
> > +		};
> >  	};
> >  	__u32	len;		/* buffer size or number of iovecs */
> >  	union {
> > @@ -79,6 +83,7 @@ struct io_uring_sqe {
> >  	union {
> >  		__s32	splice_fd_in;
> >  		__u32	file_index;
> > +		__u32	optlen;
> >  		struct {
> >  			__u16	addr_len;
> >  			__u16	__pad3[1];
> > @@ -89,6 +94,7 @@ struct io_uring_sqe {
> >  			__u64	addr3;
> >  			__u64	__pad2[1];
> >  		};
> > +		__u64	optval;
> >  		/*
> >  		 * If the ring is initialized with IORING_SETUP_SQE128, then
> >  		 * this field is used for 80 bytes of arbitrary command data
> > @@ -729,6 +735,7 @@ struct io_uring_recvmsg_out {
> >  enum {
> >  	SOCKET_URING_OP_SIOCINQ		= 0,
> >  	SOCKET_URING_OP_SIOCOUTQ,
> > +	SOCKET_URING_OP_GETSOCKOPT,
> >  };
> >  
> >  #ifdef __cplusplus
> > diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c
> > index 8e7a03c1b20e..16c857cbf3b0 100644
> > --- a/io_uring/uring_cmd.c
> > +++ b/io_uring/uring_cmd.c
> > @@ -166,6 +166,47 @@ int io_uring_cmd_import_fixed(u64 ubuf, unsigned long len, int rw,
> >  }
> >  EXPORT_SYMBOL_GPL(io_uring_cmd_import_fixed);
> >  
> > +static inline int io_uring_cmd_getsockopt(struct socket *sock,
> > +					  struct io_uring_cmd *cmd)
> > +{
> > +	void __user *optval = u64_to_user_ptr(READ_ONCE(cmd->sqe->optval));
> > +	int optname = READ_ONCE(cmd->sqe->optname);
> > +	int optlen = READ_ONCE(cmd->sqe->optlen);
> > +	int level = READ_ONCE(cmd->sqe->level);
> > +	void *koptval;
> > +	int err;
> > +
> > +	err = security_socket_getsockopt(sock, level, optname);
> > +	if (err)
> > +		return err;
> > +
> > +	koptval = kmalloc(optlen, GFP_KERNEL);
> > +	if (!koptval)
> > +		return -ENOMEM;
> 
> This will try to kmalloc any length that userspace passes?

Yes, this value is coming directly from userspace.

> That is unnecessary ..
> > +
> > +	err = copy_from_user(koptval, optval, optlen);
> > +	if (err)
> > +		goto fail;
> > +
> > +	err = -EOPNOTSUPP;
> > +	if (level == SOL_SOCKET) {
> > +		err = sk_getsockopt(sock->sk, level, optname,
> > +				    KERNEL_SOCKPTR(koptval),
> > +				    KERNEL_SOCKPTR(&optlen));
> 
> .. sk_getsockopt defines a union of acceptable fields, which
> are all fairly small.

Right, and they are all I need for SOL_SOCKET level for now.

> I notice that BPF added BPF_CGROUP_GETSOCKOPT_MAX_OPTLEN to
> work around the issue of pre-allocating for the worst case.

I am having a hard time how to understand how
BPF_CGROUP_GETSOCKOPT_MAX_OPTLEN gets the MAX_OPTLEN. Reading this
function, it seems it is conditionally get_user().


	#define BPF_CGROUP_GETSOCKOPT_MAX_OPTLEN(optlen)
	({
		int __ret = 0;
		if (cgroup_bpf_enabled(CGROUP_GETSOCKOPT))
			get_user(__ret, optlen);
		__ret;
	})

That said, how is BPF_CGROUP_GETSOCKOPT_MAX_OPTLEN being used to
workaroundthe pre-allocating for the worst case?

> But that also needs to deal woth other getsockopt levels.

Right, and we also have a similar kmalloc() on the setsockopt path
(SOCKET_URING_OP_SETSOCKOPT).

What about if I pass the userspace sockptr (USER_SOCKPTR) to the
{g,s}etsockopt callback directly, instead of kmalloc() in io_uring(), as
I was doing int the RFC[1]? It avoids any extra kmalloc at all.

Something as:

	static inline int io_uring_cmd_getsockopt(struct socket *sock,
						  struct io_uring_cmd *cmd)
	{
		void __user *optval = u64_to_user_ptr(READ_ONCE(cmd->sqe->optval));
		int optlen = READ_ONCE(cmd->sqe->optlen);
		int optname = READ_ONCE(cmd->sqe->optname);
		int level = READ_ONCE(cmd->sqe->level);
		int err;

		err = security_socket_getsockopt(sock, level, optname);
		if (err)
			return err;

		if (level == SOL_SOCKET) {
			err = sk_getsockopt(sock->sk, level, optname,
					    USER_SOCKPTR(optval),
					    KERNEL_SOCKPTR(&optlen));
			if (err < 0)
				return err;
			return optlen;
		}

		return -EOPNOTSUPP;

Thanks for the review!

[1] Link: https://lore.kernel.org/all/20230719102737.2513246-3-leitao@xxxxxxxxxx/