Added support for testing the negative cases of btree.c in a unit test.

https://github.com/openzfs/zfs/pull/19009