[GIT PULL] Btrfs updates for 5.2
From: David Sterba
Date: Mon May 06 2019 - 09:31:09 EST
Hi,
this time the majority of changes are cleanups, though there's still a
number of changes of user interest. Please pull, thanks.
User visible changes:
- better read time and write checks to catch errors early and before
writing data to disk (to catch potential memory corruption on data that
get checksummed)
- qgroups + metadata relocation: last speed up patch int the series to
address the slowness, there should be no overhead comparing balance with
and without qgroups
- FIEMAP ioctl does not start a transaction unnecessarily, this can result
in a speed up and less blocking due to IO
- LOGICAL_INO (v1, v2) does not start transaction unnecessarily, this can
speed up the mentioned ioctl and scrub as well
- fsync on files with many (but not too many) hardlinks is faster, finer
decision if the links should be fsynced individually or completely
- send tries harder to find ranges to clone
- trim/discard will skip unallocated chunks that haven't been touched
since the last mount
Fixes:
- send flushes delayed allocation before start, otherwise it could miss
some changes in case of a very recent rw->ro switch of a subvolume
- fix fallocate with qgroups that could lead to space accounting underflow,
reported as a warning
- trim/discard ioctl honours the requested range
- starting send and dedupe on a subvolume at the same time will let only
one of them succeed, this is to prevent changes that send could miss due
to dedupe; both operations are restartable
Core changes:
- more tree-checker validations, errors reported by fuzzing tools
- device item
- inode item
- block group profiles
- tracepoints for extent buffer locking
- async cow preallocates memory to avoid errors happening too deep in the
call chain
- metadata reservations for delalloc reworked to better adapt in
many-writers/low-space scenarios
- improved space flushing logic for intense DIO vs buffered workloads
- lots of cleanups
- removed unused struct members
- redundant argument removal
- properties and xattrs
- extent buffer locking
- selftests
- use common file type conversions
- many-argument functions reduction
----------------------------------------------------------------
The following changes since commit 37624b58542fb9f2d9a70e6ea006ef8a5f66c30b:
Linux 5.1-rc7 (2019-04-28 17:04:13 -0700)
are available in the Git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git for-5.2-tag
for you to fetch changes up to b1c16ac978fd40ae636e629bb69a652df7eebdc2:
btrfs: Use kvmalloc for allocating compressed path context (2019-05-02 13:48:19 +0200)
----------------------------------------------------------------
Anand Jain (22):
btrfs: merge _btrfs_set_prop helpers
btrfs: drop redundant forward declaration in props.c
btrfs: rename fs_info argument to fs_private
btrfs: refactor btrfs_set_prop and add btrfs_set_prop_trans
btrfs: drop unused parameter in mount_subvol
btrfs: prop: open code btrfs_set_prop in inherit_prop
btrfs: rename btrfs_setxattr to btrfs_setxattr_trans
btrfs: rename do_setxattr to btrfs_setxattr
btrfs: export btrfs_setxattr
btrfs: remove redundant readonly root check in btrfs_setxattr_trans
btrfs: split btrfs_setxattr calls regarding transaction
btrfs: cleanup btrfs_setxattr_trans and drop transaction parameter
btrfs: refactor btrfs_set_props to validate externally
btrfs: export btrfs_set_prop
btrfs: start transaction in btrfs_ioctl_setflags()
btrfs: drop useless inode i_flags copy and restore
btrfs: modify local copy of btrfs_inode flags
btrfs: drop old_fsflags in btrfs_ioctl_setflags
btrfs: drop local copy of inode i_mode
btrfs: start transaction in xattr_handler_set_prop
btrfs: delete unused function btrfs_set_prop_trans
btrfs: merge calls of btrfs_setxattr and btrfs_setxattr_trans in btrfs_set_prop
Arnd Bergmann (1):
btrfs: use BUG() instead of BUG_ON(1)
David Sterba (107):
btrfs: scrub: return EAGAIN when fs is closing
btrfs: switch extent_io_tree::track_uptodate to bool
btrfs: add assertion helpers for spinning writers
btrfs: use assertion helpers for spinning writers
btrfs: add assertion helpers for spinning readers
btrfs: use assertion helpers for spinning readers
btrfs: add assertion helpers for extent buffer read lock counters
btrfs: use assertion helpers for extent buffer read lock counters
btrfs: add assertion helpers for extent buffer write lock counters
btrfs: use assertion helpers for extent buffer write lock counters
btrfs: switch extent_buffer::lock_nested to bool
btrfs: tests: handle fs_info allocation failure in extent_io tests
btrfs: tests: don't leak fs_info in extent_io bitmap tests
btrfs: tests: print file:line for error messages
btrfs: tests: add table of most common errors
btrfs: tests: use standard error message after fs_info allocation failure
btrfs: tests: use standard error message after root allocation failure
btrfs: tests: use standard error message after extent buffer allocation failure
btrfs: tests: use standard error message after path allocation failure
btrfs: tests: use standard error message after inode allocation failure
btrfs: tests: use standard error message after block group allocation failure
btrfs: tests: properly initialize fs_info of extent buffer
btrfs: tests: return errors from extent map tests
btrfs: tests: return errors from extent map test case 1
btrfs: tests: return errors from extent map test case 2
btrfs: tests: return errors from extent map test case 3
btrfs: tests: return errors from extent map test case 4
btrfs: tests: return error from all extent map test cases
btrfs: tests: use standard error message after extent map allocation failure
btrfs: tests: use SZ_ constants everywhere
btrfs: tests: fix comments about tested extent map ranges
btrfs: tests: drop messages when some tests finish
btrfs: tests: unify messages when tests start
btrfs: remove stale definition of BUFFER_LRU_MAX
btrfs: move tree block wait and write helpers to tree-log
btrfs: get fs_info from eb in lock_extent_buffer_for_io
btrfs: get fs_info from eb in repair_eb_io_failure
btrfs: get fs_info from eb in write_one_eb
btrfs: get fs_info from eb in leaf_data_end
btrfs: get fs_info from eb in btrfs_exclude_logged_extents
btrfs: get fs_info from eb in check_tree_block_fsid
btrfs: get fs_info from eb in tree_mod_log_eb_copy
btrfs: get fs_info from eb in clean_tree_block
btrfs: get fs_info from eb in btrfs_leaf_free_space
btrfs: get fs_info from eb in read_node_slot
btrfs: get fs_info from eb in btree_read_extent_buffer_pages
btrfs: get fs_info from eb in btrfs_verify_level_key
btrfs: qgroup: remove obsolete fs_info members
btrfs: tree-checker: get fs_info from eb in generic_err
btrfs: tree-checker: get fs_info from eb in file_extent_err
btrfs: tree-checker: get fs_info from eb in check_csum_item
btrfs: tree-checker: get fs_info from eb in dir_item_err
btrfs: tree-checker: get fs_info from eb in check_dir_item
btrfs: tree-checker: get fs_info from eb in block_group_err
btrfs: tree-checker: get fs_info from eb in check_block_group_item
btrfs: tree-checker: get fs_info from eb in check_extent_data_item
btrfs: tree-checker: get fs_info from eb in check_leaf_item
btrfs: tree-checker: get fs_info from eb in check_leaf
btrfs: tree-checker: get fs_info from eb in chunk_err
btrfs: tree-checker: get fs_info from eb in dev_item_err
btrfs: tree-checker: get fs_info from eb in check_dev_item
btrfs: tree-checker: get fs_info from eb in check_inode_item
btrfs: get fs_info from eb in btrfs_check_leaf_full
btrfs: get fs_info from eb in btrfs_check_leaf_relaxed
btrfs: get fs_info from eb in btrfs_check_node
btrfs: get fs_info from eb in should_balance_chunk
btrfs: get fs_info from eb in btrfs_check_chunk_valid
btrfs: get fs_info from eb in read_one_chunk
btrfs: get fs_info from eb in read_one_dev
btrfs: get fs_info from trans in write_one_cache_group
btrfs: get fs_info from trans in btrfs_setup_space_cache
btrfs: get fs_info from trans in btrfs_write_dirty_block_groups
btrfs: get fs_info from trans in update_block_group
btrfs: get fs_info from trans in btrfs_create_tree
btrfs: get fs_info from trans in btrfs_need_log_full_commit
btrfs: get fs_info from trans in btrfs_set_log_full_commit
btrfs: get fs_info from trans in create_free_space_inode
btrfs: get fs_info from trans in btrfs_write_out_cache
btrfs: get fs_info from trans in push_node_left
btrfs: get fs_info from trans in balance_node_right
btrfs: get fs_info from trans in insert_ptr
btrfs: get fs_info from trans in copy_for_split
btrfs: get fs_info from trans in init_first_rw_device
btrfs: get fs_info from trans in btrfs_finish_sprout
btrfs: get fs_info from trans in btrfs_run_dev_stats
btrfs: get fs_info from trans in btrfs_run_dev_replace
btrfs: get fs_info from block group in next_block_group
btrfs: get fs_info from block group in pin_down_extent
btrfs: get fs_info from block group in lookup_free_space_inode
btrfs: get fs_info from block group in load_free_space_cache
btrfs: get fs_info from block group in write_pinned_extent_entries
btrfs: get fs_info from block group in btrfs_find_space_cluster
btrfs: get fs_info from block group in search_free_space_info
btrfs: get fs_info from eb in __push_leaf_right
btrfs: get fs_info from eb in __push_leaf_left
btrfs: get fs_info from device in btrfs_rm_dev_item
btrfs: get fs_info from device in btrfs_scrub_cancel_dev
btrfs: get fs_info from device in btrfs_rm_dev_replace_free_srcdev
btrfs: remove unused parameter fs_info from split_item
btrfs: remove unused parameter fs_info from btrfs_truncate_item
btrfs: remove unused parameter fs_info from btrfs_extend_item
btrfs: remove unused parameter fs_info from tree_move_down
btrfs: remove unused parameter fs_info from from tree_advance
btrfs: remove unused parameter fs_info from CHECK_FE_ALIGNED
btrfs: remove unused parameter fs_info from emit_last_fiemap_cache
btrfs: remove unused parameter fs_info from btrfs_add_delayed_extent_op
btrfs: remove unused parameter fs_info from btrfs_set_disk_extent_flags
Dennis Zhou (1):
btrfs: zstd: remove indirect calls for local functions
Filipe Manana (8):
Btrfs: remove no longer used 'sync' member from transaction handle
Btrfs: remove no longer used member num_dirty_bgs from transaction
Btrfs: remove no longer used function to run delayed refs asynchronously
Btrfs: do not start a transaction at iterate_extent_inodes()
Btrfs: do not start a transaction during fiemap
Btrfs: send, flush dellaloc in order to avoid data loss
Btrfs: fix race between send and deduplication that lead to failures and crashes
Btrfs: improve performance on fsync of files with multiple hardlinks
Goldwyn Rodrigues (2):
btrfs: Initialize inode::i_mapping once in btrfs_symlink
btrfs: Perform locking/unlocking in btrfs_remap_file_range()
Jeff Mahoney (1):
btrfs: replace pending/pinned chunks lists with io tree
Johannes Thumshirn (3):
btrfs: factor our read/write stage off csum_tree_block into its callers
btrfs: warn if extent buffer mapping crosses a page boundary in csum_tree_block
btrfs: reduce kmap_atomic time for checksumming
Josef Bacik (3):
btrfs: fix panic during relocation after ENOSPC before writeback happens
btrfs: track DIO bytes in flight
btrfs: reserve delalloc metadata differently
Nathan Chancellor (1):
btrfs: Turn an 'else if' into an 'else' in btrfs_uuid_tree_add
Nikolay Borisov (36):
btrfs: Remove EXTENT_WRITEBACK
btrfs: Remove EXTENT_IOBITS
btrfs: Exploit the fact that pages passed to extent_readpages are always contiguous
btrfs: Remove unused -EIO assignment in end_bio_extent_readpage
btrfs: Correctly free extent buffer in case btree_read_extent_buffer_pages fails
btrfs: Use less confusing condition for uptodate parameter to btrfs_writepage_endio_finish_ordered
btrfs: Honour FITRIM range constraints during free space trim
btrfs: combine device update operations during transaction commit
btrfs: Handle pending/pinned chunks before blockgroup relocation during device shrink
btrfs: Rename and export clear_btree_io_tree
btrfs: Populate ->orig_block_len during read_one_chunk
btrfs: Introduce new bits for device allocation tree
btrfs: Implement set_extent_bits_nowait
btrfs: Stop using call_rcu for device freeing
btrfs: Transpose btrfs_close_devices/btrfs_mapping_tree_free in close_ctree
btrfs: Remove 'trans' argument from find_free_dev_extent(_start)
btrfs: Factor out in_range macro
btrfs: Optimize unallocated chunks discard
btrfs: Implement find_first_clear_extent_bit
btrfs: Switch btrfs_trim_free_extents to find_first_clear_extent_bit
btrfs: Remove redundant inode argument from btrfs_add_ordered_sum
btrfs: Define submit_bio_hook's type directly
btrfs: Change submit_bio_hook to taking an inode directly
btrfs: Remove 'tree' argument from read_extent_buffer_pages
btrfs: Pass 0 for bio_offset to btrfs_wq_submit_bio
btrfs: Always pass 0 bio_offset for btree_submit_bio_start
btrfs: Remove bio_offset argument from submit_bio_hook
btrfs: Document btrfs_csum_one_bio
btrfs: Preallocate chunks in cow_file_range_async
btrfs: Rename async_cow to async_chunk
btrfs: Remove fs_info from struct async_chunk
btrfs: Make compress_file_range take only struct async_chunk
btrfs: Replace clear_extent_bit with unlock_extent
btrfs: Set io_tree only once in submit_compressed_extents
btrfs: Factor out common extent locking code in submit_compressed_extents
btrfs: Use kvmalloc for allocating compressed path context
Phillip Potter (1):
btrfs: use common file type conversion
Qu Wenruo (39):
btrfs: Don't panic when we can't find a root key
btrfs: Introduce fs_info to extent_io_tree
btrfs: Introduce extent_io_tree::owner to distinguish different io_trees
btrfs: tracepoints: Add trace events for extent_io_tree
btrfs: reloc: Fix NULL pointer dereference due to expanded reloc_root lifespan
btrfs: Make btrfs_(set|clear)_header_flag return void
btrfs: Check the first key and level for cached extent buffer
btrfs: Always output error message when key/level verification fails
btrfs: extent_io: Move the BUG_ON() in flush_write_bio() one level up
btrfs: extent_io: Handle errors better in extent_write_full_page()
btrfs: extent_io: Handle errors better in btree_write_cache_pages()
btrfs: extent_io: Kill dead condition in extent_write_cache_pages()
btrfs: extent_io: Handle errors better in extent_write_locked_range()
btrfs: extent_io: add proper error handling to lock_extent_buffer_for_io()
btrfs: extent_io: Handle errors better in extent_writepages()
btrfs: disk-io: Show the timing of corrupted tree block explicitly
btrfs: Move btrfs_check_chunk_valid() to tree-check.[ch] and export it
btrfs: tree-checker: Make chunk item checker messages more readable
btrfs: tree-checker: Make btrfs_check_chunk_valid() return EUCLEAN instead of EIO
btrfs: tree-checker: Check chunk item at tree block read time
btrfs: tree-checker: Verify dev item
btrfs: tree-checker: Enhance chunk checker to validate chunk profile
btrfs: tree-checker: Verify inode item
btrfs: inode: Verify inode mode to avoid NULL pointer dereference
btrfs: tree-checker: Remove comprehensive root owner check
btrfs: Do mandatory tree block check before submitting bio
btrfs: trace: Introduce trace events for sleepable tree lock
btrfs: trace: Introduce trace events for all btrfs tree locking events
btrfs: delayed-ref: Introduce better documented delayed ref structures
btrfs: extent-tree: Open-code process_func in __btrfs_mod_ref
btrfs: delayed-ref: Use btrfs_ref to refactor btrfs_add_delayed_tree_ref()
btrfs: delayed-ref: Use btrfs_ref to refactor btrfs_add_delayed_data_ref()
btrfs: ref-verify: Use btrfs_ref to refactor btrfs_ref_tree_mod()
btrfs: extent-tree: Use btrfs_ref to refactor add_pinned_bytes()
btrfs: extent-tree: Use btrfs_ref to refactor btrfs_inc_extent_ref()
btrfs: extent-tree: Use btrfs_ref to refactor btrfs_free_extent()
btrfs: qgroup: Don't scan leaf if we're modifying reloc tree
btrfs: tree-checker: Allow error injection for tree-checker
btrfs: ctree: Dump the leaf before BUG_ON in btrfs_set_item_key_safe
Robbie Ko (2):
Btrfs: send, improve clone range
Btrfs: fix data bytes_may_use underflow with fallocate due to failed quota reserve
fs/btrfs/acl.c | 6 +-
fs/btrfs/backref.c | 38 +-
fs/btrfs/btrfs_inode.h | 8 -
fs/btrfs/compression.c | 2 +-
fs/btrfs/ctree.c | 254 +++++++------
fs/btrfs/ctree.h | 78 ++--
fs/btrfs/delayed-inode.c | 5 +-
fs/btrfs/delayed-ref.c | 46 ++-
fs/btrfs/delayed-ref.h | 122 +++++-
fs/btrfs/dev-replace.c | 8 +-
fs/btrfs/dev-replace.h | 3 +-
fs/btrfs/dir-item.c | 5 +-
fs/btrfs/disk-io.c | 225 ++++++------
fs/btrfs/disk-io.h | 7 +-
fs/btrfs/extent-tree.c | 651 ++++++++++++---------------------
fs/btrfs/extent_io.c | 356 ++++++++++++------
fs/btrfs/extent_io.h | 89 +++--
fs/btrfs/extent_map.c | 38 ++
fs/btrfs/file-item.c | 32 +-
fs/btrfs/file.c | 47 ++-
fs/btrfs/free-space-cache.c | 45 +--
fs/btrfs/free-space-cache.h | 18 +-
fs/btrfs/free-space-tree.c | 24 +-
fs/btrfs/free-space-tree.h | 1 -
fs/btrfs/inode-item.c | 8 +-
fs/btrfs/inode.c | 329 +++++++++--------
fs/btrfs/ioctl.c | 181 ++++-----
fs/btrfs/locking.c | 157 ++++++--
fs/btrfs/ordered-data.c | 14 +-
fs/btrfs/ordered-data.h | 3 +-
fs/btrfs/print-tree.c | 2 +-
fs/btrfs/props.c | 242 ++++++------
fs/btrfs/props.h | 7 +-
fs/btrfs/qgroup.c | 5 +-
fs/btrfs/ref-verify.c | 53 +--
fs/btrfs/ref-verify.h | 10 +-
fs/btrfs/relocation.c | 123 ++++---
fs/btrfs/root-tree.c | 13 +-
fs/btrfs/scrub.c | 6 +-
fs/btrfs/send.c | 114 +++++-
fs/btrfs/super.c | 4 +-
fs/btrfs/tests/btrfs-tests.c | 17 +-
fs/btrfs/tests/btrfs-tests.h | 17 +-
fs/btrfs/tests/extent-buffer-tests.c | 8 +-
fs/btrfs/tests/extent-io-tests.c | 35 +-
fs/btrfs/tests/extent-map-tests.c | 213 +++++++----
fs/btrfs/tests/free-space-tests.c | 11 +-
fs/btrfs/tests/free-space-tree-tests.c | 14 +-
fs/btrfs/tests/inode-tests.c | 34 +-
fs/btrfs/tests/qgroup-tests.c | 20 +-
fs/btrfs/transaction.c | 64 +---
fs/btrfs/transaction.h | 4 +-
fs/btrfs/tree-checker.c | 513 ++++++++++++++++++++------
fs/btrfs/tree-checker.h | 11 +-
fs/btrfs/tree-log.c | 289 +++++++++++----
fs/btrfs/tree-log.h | 10 +-
fs/btrfs/uuid-tree.c | 6 +-
fs/btrfs/volumes.c | 469 ++++++++----------------
fs/btrfs/volumes.h | 39 +-
fs/btrfs/xattr.c | 65 ++--
fs/btrfs/xattr.h | 7 +-
fs/btrfs/zstd.c | 11 +-
include/trace/events/btrfs.h | 243 +++++++++++-
include/uapi/linux/btrfs_tree.h | 2 +
64 files changed, 3287 insertions(+), 2194 deletions(-)