Re: [PATCH 1/5] initdev:kernel: Asynchronously-discovered devicesynchronization, v5

[Sergey Vlasov]

On Fri, May 01, 2009 at 07:25:51PM -0700, David VomLehn wrote:
> In days of yore, great fields of USB devices were discovered and initialized
> long before they were harvested for use. Race conditions were unknown, and
> all was well. As time passed in the land of Linux, the tilling of the bit
> fields yielded great reductions in boot time, and there was much rejoicing.
> Yet, some dispaired, for things were not as they had once been. The now-
> hasty arrival at /bin/init startup meant some USB devices were not yet fully
> ripened, leaving /dev/console unopened and network devices unconfigured.
> 
> In less rural language, my embedded system has no USB console and no USB
> network device to telnet into.  Since 2.6.28.  Plenty of woe.
> 
> This patch adds infrastructure to allow bus support code to indicate
> when it is in the process of asynchronously discovering devices. This allows
> code that might want to use those devices during kernel boot to wait until
> all attached devices are discovered and initialized.  Synchronization is
> done by device type, so there is no need for block devices to wait for the
> console to be initialized. Block, console, and network devices are currently
> supported on USB and SCII buses.
                       ^^^^ typo

> 
> And all was, well, better.
> 
> (Many thanks to Alan Stern, who said we could solve this without offensive
> timeouts and then set about doing the integration into USB and SCSI)
> 
> History
> v5	Change nomenclature to refer to initdevs instead of bootdevs.
> 	Devices supported by loadable modules can't be init devices, so
> 	create initdev_* functions that do nothing if MODULE is defined.
> v4	Update documentation and move to its own file. Change names of
> 	functions to remove leading "bus_". Move functions into
> 	drivers/base/bootdev.c. Fix NETCONSOLE handling. Add function to
> 	be called from subsystems to wakeup waiters.
> v3	Modify bus_bootdev_type_found to avoid superfluous wakeups.
> v2      Remove global wait_queue for buses in favor of using the per-boot
> 	device type wait_queues. Correct wait condition in bus_wait_for_bootdev
> 	so that the done function will be called even if devices are still
> 	being discovered. Added bus_bootdev_type_found.
> v1	Original release
> 
> Signed-off-by: David VomLehn <dvomlehn@xxxxxxxxx>
> ---
>  Documentation/driver-model/initdev.txt |  115 +++++++++++++++++++
>  drivers/base/Makefile                  |    3 +-
>  drivers/base/base.h                    |    1 +
>  drivers/base/init.c                    |    1 +
>  drivers/base/initdev.c                 |  189 ++++++++++++++++++++++++++++++++
>  include/linux/device.h                 |   51 +++++++++
>  6 files changed, 359 insertions(+), 1 deletions(-)
> 
> diff --git a/Documentation/driver-model/initdev.txt b/Documentation/driver-model/initdev.txt
> new file mode 100644
> index 0000000..efc0679
> --- /dev/null
> +++ b/Documentation/driver-model/initdev.txt
> @@ -0,0 +1,115 @@
> +Init Device Discovery Synchronization
> +=====================================
> +Init devices are those devices that must be used or configured before
> +starting up the /bin/init process.  They may be explicitly specified as
> +kernel command line parameters, such as console=ttyUSB0,115200, or
> +implicitly specified, such as ip=dhcp.
> +
> +Earlier versions of the Linux kernel used a single-threaded approach to
> +boot initialization. This took a number of seconds, which meant that
> +devices were generally set up before being used or configured. Modern
> +kernels use a multithreaded approach, which requires synchronization between
> +code that probes and initializes init devices, and code that uses and
> +configures them.  Support of fine-grained boot concurrency requires
> +distinguishing between types of init devices, so that devices can be used as
> +soon as they are initialized.
> +
> +There are several types of init devices:
> +-	consoles
> +-	network devices
> +-	block devices
> +
> +There is a distinction between the hardware type and the init device type.
> +From the hardware view, most any serial device can be used as a console,
> +but console support is generally configured separately. For example, USB
> +serial devices should be considered console init devices only if the
> +kernel is configured to support this usage. This is done by enabling the
> +CONFIG_USB_SERIAL_CONSOLE option. If this option is disabled, the USB bus
> +driver should not report that it has found any console devices.
> +
> +The sequence for buses with asynchronously-discoverable init devices is:
> +1.	As each possible init devices is discovered, call initdev_found.
> +2.	Optionally, if the device type wasn't known at the time
> +	initdev_found was called but is determined later, call
> +	initdev_type_found when the type is found.
> +3.	As initialization is complete for each device, call
> +	initdev_probe_done.
> +
> +Per-Device Functions
> +--------------------
> +Functions used to indicate the status of asynchronous device discovery on
> +a device-by-device basis are:
> +
> +initdev_found(int initdev_mask)
> +	This function must be called each time a possible init device device
> +	is found.  It is passed a bit mask created by ORing any of the
> +	following flags that apply to the device found:
> +		BOOTDEV_CONSOLE_MASK
> +		BOOTDEV_NETDEV_MASK
> +		BOOTDEV_BLOCK_MASK

Should these also be renamed to INITDEV_* ?

> +	There is no need to call this function for a given device if it is
> +	known that it cannot be used as a init device. If it is not
> +	possible to determine whether a device is usable as a init device,
> +	or the specific type of a init device, the argument BOOTDEV_ANY_MASK
> +	should be passed.  This should be used only when necessary, as it
> +	reduces the level of boot-time concurrency.
> +
> +initdev_type_found(int initdev_mask, enum initdev_type type)
> +	If the type of a init device could not be found, but was determined
> +	later, this function can be used to indicate the device type. It
> +	is passed the mask that was originally passed to initdev_found.
> +	Note that, if a device for which initdev_found is called is
> +	subsequently determine not to be a possible init device, this function
> +	should not be called. Instead, initdev_probe_done should be
> +	called.

This function is never used in your patches - maybe drop it for now
and introduce when a need for it arises?

> +
> +initdev_probe_done(int initdev_mask)
> +	This function indicates that initialization is complete for a device
> +	which may be one of the init device types indicated by initdev_mask.
> +	Note that calling this function does not imply that device
> +	initialization has been successful; if initdev_found is called for a
> +	device, initdev_probe_done must be called even if an error occurred.
> +
> +Some rules about correct usage:
> +o	Each call to initdev_found *must* be matched by a call to
> +	initdev_probe_done.
> +o	The function initdev_type_found may be used only once for a given
> +	device and must be called between the calls to initdev_found and
> +	initdev_probe_done for that device.
> +o	If initdev_type_found is called:
> +	-	The value of initdev_mask must be the same as that passed to
> +		initdev_found.
> +	-	The bit corresponding to the type argument must have been set
> +		in the initdev_mask argument of initdev_found.
> +	-	The value of initdev_mask in the call to initdev_probe_done
> +		must consist only of the bit corresponding to the argument
> +		type passed to initdev_type_found.
> +o	If initdev_type_found is not called, the value of initdev_mask
> +	passed to initdev_found and initdev_probe_done for a given
> +	device must be identical.
> +
> +Subsystem Functions
> +-------------------
> +Boot devices are registered with the appropriate subsystem as they are found
> +Each subsystem determines when the init devices is manages are ready for use.
> +
> +initdev_wait(enum initdev_type type, bool (*done)(void))
> +	Wait until one of two conditions is met:
> +	o	All possible init devices of the given type have been probed
> +	o	The "done" function returns true
> +
> +	The type is one of:
> +		BOOTDEV_CONSOLE
> +		BOOTDEV_NETDEV
> +		BOOTDEV_BLOCK
> +	The "done" function is subsystem-dependent and returns true if all
> +	init devices required for that subsystem to continue booting have
> +	been registered. Registration is indicated through use of the
> +	initdev_registered function. If done always returns false,
> +	initdev_wait will not return until all possible init devices of the
> +	given type have been probed.
> +
> +initdev_registered(enum initdev_type type)
> +	Called by each software subsystem that handles init devices of a given
> +	type. For example, register_console would call this function with
> +	a type of BOOTDEV_CONSOLE.
> diff --git a/drivers/base/Makefile b/drivers/base/Makefile
> index b5b8ba5..a288a46 100644
> --- a/drivers/base/Makefile
> +++ b/drivers/base/Makefile
> @@ -3,7 +3,8 @@
>  obj-y			:= core.o sys.o bus.o dd.o \
>  			   driver.o class.o platform.o \
>  			   cpu.o firmware.o init.o map.o devres.o \
> -			   attribute_container.o transport_class.o
> +			   attribute_container.o transport_class.o \
> +			   initdev.o
>  obj-y			+= power/
>  obj-$(CONFIG_HAS_DMA)	+= dma-mapping.o
>  obj-$(CONFIG_ISA)	+= isa.o
> diff --git a/drivers/base/base.h b/drivers/base/base.h
> index b528145..90dede0 100644
> --- a/drivers/base/base.h
> +++ b/drivers/base/base.h
> @@ -90,6 +90,7 @@ struct device_private {
>  	container_of(obj, struct device_private, knode_bus)
>  
>  /* initialisation functions */
> +extern int initdevs_init(void);
>  extern int devices_init(void);
>  extern int buses_init(void);
>  extern int classes_init(void);
> diff --git a/drivers/base/init.c b/drivers/base/init.c
> index 7bd9b6a..683bc0d 100644
> --- a/drivers/base/init.c
> +++ b/drivers/base/init.c
> @@ -21,6 +21,7 @@ void __init driver_init(void)
>  {
>  	/* These are the core pieces */
>  	devices_init();
> +	initdevs_init();
>  	buses_init();
>  	classes_init();
>  	firmware_init();
> diff --git a/drivers/base/initdev.c b/drivers/base/initdev.c
> new file mode 100644
> index 0000000..eaf53e3
> --- /dev/null
> +++ b/drivers/base/initdev.c
> @@ -0,0 +1,189 @@
> +/*
> + *			initdev.c
> + *
> + * Primitives to synchronize boot device discovery and initialization with
> + * their boot-time use.
> + *
> + * Copyright (C) 2009  Scientific-Atlanta, Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> + *
> + * Author: David VomLehn
> + */
> +
> +#include <linux/types.h>
> +#include <linux/wait.h>
> +#include <linux/device.h>
> +#include <linux/sched.h>
> +#include <linux/async.h>
> +
> +/*
> + * Information about how many boot devices have been found, but for which
> + * probing activity is not yet complete, by type, along with wait queues on
> + * which a wake up will be done as the probing completes.
> + */
> +static struct {
> +	atomic_t		pending;
> +	wait_queue_head_t	wq;
> +} initdev_info[BOOTDEV_NUM];
> +
> +__init int initdevs_init(void)
> +{
> +	int	i;
> +
> +	for (i = 0; i < ARRAY_SIZE(initdev_info); i++) {
> +		atomic_set(&initdev_info[i].pending, 0);
> +		init_waitqueue_head(&initdev_info[i].wq);
> +	}
> +
> +	return 0;

The return value is useless - it is never checked.

> +}
> +
> +/*
> + * Adds a console bit to the mask of boot devices if network consoles are used
> + * @mask:	Starting mask
> + *
> + * Returns mask with the %BOOTDEV_CONSOLE bit set if network consoles are
> + * configured into the system, since, from the boot device standpoint,
> + * a network device might be used as a console.
> + */
> +#ifdef CONFIG_NETCONSOLE
> +static int initdev_add_netconsole(int initdev_mask)
> +{
> +	return (initdev_mask & BOOTDEV_NETDEV_MASK) ?
> +		initdev_mask | BOOTDEV_CONSOLE : initdev_mask;

This should be BOOTDEV_CONSOLE_MASK.

> +}
> +#else
> +static int initdev_add_netconsole(int initdev_mask)
> +{
> +	return initdev_mask;
> +}
> +#endif
> +
> +/**
> + * initdev_found - called when a new device is discovered on a bus.
> + * @initdev_mask:	Mask for possible devices
> + *
> + * The initdev_mask allows a caller to specify a restricted set of boot
> + * devices that might be probed. For example, if the bus is USB, it may be
> + * the case that %CONFIG_USB_SERIAL_CONSOLE is not defined. The call to this
> + * function might then omit %BOOTDEV_CONSOLE from the mask. When we go to
> + * wait for console devices, we won't wait for USB probing to complete. If
> + * a given bus type has no way to determine the type of device being probed,
> + * it can simply pass %BOOTDEV_ALL_MASK, but finer granularity will generally
> + * result in faster boot times.
> + */
> +void initdev_found(int initdev_mask)
> +{
> +	int	i;
> +
> +	initdev_mask = initdev_add_netconsole(initdev_mask);
> +
> +	for (i = 0; i < ARRAY_SIZE(initdev_info); i++) {
> +		if (initdev_mask & (1 << i))
> +			atomic_inc(&initdev_info[i].pending);
> +	}
> +}
> +
> +/**
> + * initdev_type_found - called upon determining the type of a boot device
> + * @old_mask:	Mask used in the call to initdev_found
> + * @type:	Type of the boot device
> + *
> + * If initdev_found() was called but the type of the boot device was not
> + * yet available, this function can be called when the type is available.
> + * It allows waiters for boot devices of types other than the given @type
> + * to proceed.
> + */
> +void initdev_type_found(int old_mask, enum initdev_type type)
> +{
> +	int	i;
> +
> +	old_mask = initdev_add_netconsole(old_mask);
> +
> +	for (i = 0; i < ARRAY_SIZE(initdev_info); i++) {
> +		/* If this is one of the types we are dropping and we reduce
> +		 * the pending count for that type of boot device to zero,
> +		 * we need to wake up any waiters for that device. Otherwise,
> +		 * we still have other devices being waited on and the wakeup
> +		 * would be superfluous. */
> +		if (i != type && (old_mask & (1 << i))) {
> +			if (atomic_dec_return(&initdev_info[i].pending) == 0)

May be replaced by a more commonly used atomic function:

			if (atomic_dec_and_test(&initdev_info[i].pending))

(although atomic_dec_return() seems to be significantly more expensive
than atomic_dec_and_test() only on m68k and i386 with CONFIG_M386).

> +				wake_up_all(&initdev_info[i].wq);
> +		}
> +	}
> +}

What if type == BOOTDEV_NETDEV, and CONFIG_NETCONSOLE is defined?
In this case we must keep both BOOTDEV_NETDEV and BOOTDEV_CONSOLE
(otherwise the subsequent initdev_probe_done(BOOTDEV_NETDEV_MASK) will
mess up the pending counter for BOOTDEV_CONSOLE).

Or, as noted above, just drop this function if is it not actually
needed anywhere.

> +
> +/**
> + * initdev_probe_done - indicate probing is complete for a device on a bus
> + * @initdev_mask:	Mask for possible devices
> + *
> + * The definition of probing complete for a given device is that the driver
> + * for that device must have been able to register its presence with its
> + * associated subsystem. So, devices supporting consoles must have called
> + * register_console(), networking device must have called register_netdevice(),
> + * etc.
> + */
> +void initdev_probe_done(int initdev_mask)
> +{
> +	int	i;
> +
> +	initdev_mask = initdev_add_netconsole(initdev_mask);
> +
> +	for (i = 0; i < ARRAY_SIZE(initdev_info); i++) {
> +		/* Decrement the count of pending probes and, if we reach
> +		 * zero, wake up all waiters. */
> +		if (initdev_mask & (1 << i)) {
> +			if (atomic_dec_return(&initdev_info[i].pending) == 0)

			if (atomic_dec_and_test(&initdev_info[i].pending))

> +				wake_up_all(&initdev_info[i].wq);
> +		}
> +	}
> +}
> +
> +/**
> + * initdev_wait - wait until desired boot devices are registered or probing done
> + * @type:	Type of boot device to wait for
> + * @done:	Pointer to function that determines whether all boot devices
> + *		have been acquired.
> + *
> + * This function exits if all devices of the given @type have been probed or
> + * the passed function indicates that all the expected boot devices have been
> + * acquired. Say, for example the kernel command line specifies the name of
> + * a boot device to use. The @done function can check to see whether that
> + * device has been registered and, if so, return true. This function will
> + * return even though probing has not completed on some devices, which allows
> + * faster boot times.
> + */
> +void initdev_wait(enum initdev_type type, bool (*done)(void))

This function should probably be __init - there is no point to invoke
it outside of the boot-time initialization code.

Unfortunately, the rest of code and data in this file (which will be
useless after the boot-time initialization is complete) cannot be
discarded, which will cause typical complaints that the kernel gets
more and more bloated with each release.

> +{
> +	/* Wait until initdev_probe_done has been called for all of the devices
> +	 * of the type for which we are waiting, or for some minimal set of
> +	 * those devices to be ready. This last condition is defined by the
> +	 * caller through the implementation of the callback function. */
> +	wait_event(initdev_info[type].wq,
> +		done() || atomic_read(&initdev_info[type].pending) == 0);
> +}
> +
> +/**
> + * initdev_registered - indicate boot device registration by its subsystem
> + * @type:	Type of boot device
> + *
> + * This will wake up all threads waiting on a given boot device @type so that
> + * they can see if they are ready to continue.
> + */
> +void initdev_registered(enum initdev_type type)
> +{
> +	wake_up_all(&initdev_info[type].wq);
> +}
> diff --git a/include/linux/device.h b/include/linux/device.h
> index 6a69caa..1e6b729 100644
> --- a/include/linux/device.h
> +++ b/include/linux/device.h
> @@ -94,6 +94,57 @@ int __must_check bus_for_each_drv(struct bus_type *bus,
>  void bus_sort_breadthfirst(struct bus_type *bus,
>  			   int (*compare)(const struct device *a,
>  					  const struct device *b));
> +
> +/*
> + * Definitions for synchronizing discovery of asynchronously-discoverable
> + * devices with their use as boot devices.
> + */
> +/*
> + * Define boot device types. These are not the same as the device classes
> + * supported by various buses, but are tied to support of specific Linux kernel
> + * devices. For example, USB knows about serial devices, but a serial device
> + * is only a BOOTDEV_CONSOLE if CONFIG_USB_SERIAL_CONSOLE is defined.
> + */
> +#define	BOOTDEV_CONSOLE_MASK	(1 << BOOTDEV_CONSOLE)
> +#define	BOOTDEV_NETDEV_MASK	(1 << BOOTDEV_NETDEV)
> +#define BOOTDEV_BLOCK_MASK	(1 << BOOTDEV_BLOCK)
> +#define	BOOTDEV_ANY_MASK	((1 << BOOTDEV_NUM) - 1)
> +
> +enum initdev_type {
> +	BOOTDEV_CONSOLE,		/* Device usable as a console */
> +	BOOTDEV_NETDEV,			/* Autoconfigurable network device */
> +	BOOTDEV_BLOCK,			/* Block device for boot filesystem */
> +	BOOTDEV_NUM			/* Number of boot devices */
> +};
> +
> +#ifndef MODULE
> +extern void initdev_found(int initdev_mask);
> +extern void initdev_type_found(int initdev_mask, enum initdev_type type);
> +extern void initdev_probe_done(int initdev_mask);
> +extern void initdev_registered(enum initdev_type type);
> +extern void initdev_wait(enum initdev_type type, bool (*done)(void));
> +#else
> +static inline void initdev_found(int initdev_mask)
> +{
> +}
> +
> +static inline void initdev_type_found(int initdev_mask, enum initdev_type type)
> +{
> +}
> +
> +static inline void initdev_probe_done(int initdev_mask)
> +{
> +}
> +
> +static inline void initdev_registered(enum initdev_type type)
> +{
> +}
> +
> +static inline void initdev_wait(enum initdev_type type, bool (*done)(void))
> +{
> +}
> +#endif
> +
>  /*
>   * Bus notifiers: Get notified of addition/removal of devices
>   * and binding/unbinding of drivers to devices.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-usb" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Attachment: signature.asc
Description: Digital signature