Added support for testing the negative cases of btree.c in a unit test.
Tag: OpenZFS
-
Original Motivation
SHA-256, SHA-512 and SHA-512/256 are secure hash algorithms specified in FIPS 180-4, NIST’s Secure Hash Standard. ZFS uses the above as block checksums (
checksum=sha256,checksum=sha512) for data integrity and deduplication. There is currently no unit test coverage for this family of hash algorithms.What does this commit introduce?
It adds a
tests/unit/test_sha2.cwith three types of tests that follow the FIPS 180-4 standard as defined in https://nvlpubs.nist.gov/nistpubs/fips/nist.fips.180-4.pdf :- known answer tests (KAT) for SHA-256, SHA-512 and SHA-512/256, using the same NIST test messages and digests as
tests/zfs-tests/cmd/checksum/sha2_test.c - an incremental test: split updates across the 64- and 128-byte message block boundaries must produce the same digest as a single update
- a cross-implementation test via
zfs_impl_get_ops(): every supported implementation must reproduce the known answer and compute the same digest for a multi-block message
Testing perfomed:
$ make unit T=sha2UNITTEST tests/unit/test_sha2Running test suite with seed 0xcf392006...sha2.sha256_known [ OK ] [ 0.00002286 / 0.00001675 CPU ]sha2.sha512_known [ OK ] [ 0.00002044 / 0.00001995 CPU ]sha2.sha512_256_known [ OK ] [ 0.00001542 / 0.00001524 CPU ]sha2.incremental [ OK ] [ 0.00019737 / 0.00019336 CPU ]sha2.impls [ OK ] [ 0.00026169 / 0.00026171 CPU ]5 of 5 (100%) tests successful, 0 (0%) test skipped. - known answer tests (KAT) for SHA-256, SHA-512 and SHA-512/256, using the same NIST test messages and digests as
-
Just added a new unit test suite that covers basic B-Tree operations including empty trees, add_find (), remove () and iteration functions.
> make unit T=btreeCC tests/unit/test_btree-test_btree.oCCLD tests/unit/test_btreeUNITTEST tests/unit/test_btreeRunning test suite with seed 0xd9494352...btree.empty [ OK ] [ 0.00000157 / 0.00000060 CPU ]btree.add_find [ OK ] [ 0.00000719 / 0.00000690 CPU ]btree.remove [ OK ] [ 0.00000606 / 0.00000575 CPU ]btree.walk [ OK ] [ 0.00000379 / 0.00000380 CPU ]4 of 4 (100%) tests successful, 0 (0%) test skipped. -
“OpenZFS 2.4.3 is out today as the newest stable point release to this open-source ZFS file-system implementation as well as point releases for the OpenZFS 2.3 and 2.2 series too.
One month after OpenZFS 2.4.2, OpenZFS 2.4.3 is now available with additional fixes. On the Linux side the kernel support still extends up through the Linux 7.0 stable kernel even with Linux v7.1 expected for release this coming Sunday. Hopefully another OpenZFS point release will be out shortly thereafter with blessed Linux 7.1 kernel support.
OpenZFS 2.4.3 adds an encryption key check for block cloning in ZVOL, some FreeBSD-specific work like being able to build the kernel module with sanitizers, fixing some double free conditions, fixing a possible panic, some Linux compatibility updates, a number of continuous integration (CI) updates, and various other minor fixes throughout.
Details on the OpenZFS 2.4.3 changes in full and downloads via GitHub.
In addition to the OpenZFS 2.4.3 point release, OpenZFS 2.3.8 and OpenZFS 2.2.10 are available with many of the same bug fixes back-ported to those prior series plus other relevant fixes.”
-
Along with zfs-2.2.10 and zfs-2.3.8.
-
Unit tests are deterministic tests that complement the ZTS testing infrastructure. They were first implemented by robn in test_zap.c to cover the ZAP API including microzap and fatzap.
My commit introduces a new namecheck validity testing framework for zfs pools, datasets, snapshots etc. It covers the full namecheck.c functions.
To run the test, execute make unit T=namecheck
> make unit T=namecheckUNITTEST tests/unit/test_namecheckRunning test suite with seed 0x5842ef3e...namecheck.pool [ OK ] [ 0.00000954 / 0.00000873 CPU ]namecheck.dataset [ OK ] [ 0.00001064 / 0.00000984 CPU ]namecheck.snapshot [ OK ] [ 0.00000589 / 0.00000590 CPU ]namecheck.bookmark [ OK ] [ 0.00000654 / 0.00000605 CPU ]namecheck.component [ OK ] [ 0.00000508 / 0.00000508 CPU ]namecheck.permset [ OK ] [ 0.00000608 / 0.00000561 CPU ]namecheck.mountpoint [ OK ] [ 0.00000337 / 0.00000329 CPU ]namecheck.depth [ OK ] [ 0.00000204 / 0.00000159 CPU ]8 of 8 (100%) tests successful, 0 (0%) test skipped.
https://github.com/chrislongros/zfs/commit/7e054b2e7ea80c7c838f7fd44b7d517eea5c9d18
-
The patch set with 64 commits has been created!
-
Happy to see my ZFS commits get upstream to FreeBSD 🙂
-
robn introduced a test suite for the ZFS Attribute Processor (ZAP) with this commit: https://github.com/robn/zfs/commit/1d601eb83b1b849edba047feae5137f0adb93ee2
Since then several API functions of ZAP were implemented as unit tests. With my new commit I introduce a uint64 keys test that provides coverage for binary uint64-array keys that are also used by the dedup table (DDT) and the block reference table (BRT).
The test runs dnode operations including: add, lookup, length, lookup_length, update and remove as they are implemented in module/zfs/zap.c
To build and run the test use the instructions here: https://github.com/openzfs/zfs/blob/master/tests/unit/README.md
The results:
> make unitUNITTEST tests/unit/test_zapRunning test suite with seed 0x8f27c767...zap.mock_microzap_sanity [ OK ] [ 0.00001072 / 0.00001014 CPU ]zap.mock_fatzap_sanity [ OK ] [ 0.00002379 / 0.00002299 CPU ]zap.zap_basictype=micro [ OK ] [ 0.00002169 / 0.00002172 CPU ]type=fat [ OK ] [ 0.00002119 / 0.00002120 CPU ]zap.zap_addtype=micro [ OK ] [ 0.00000964 / 0.00000965 CPU ]type=fat [ OK ] [ 0.00002727 / 0.00002665 CPU ]zap.zap_updatetype=micro [ OK ] [ 0.00001184 / 0.00001185 CPU ]type=fat [ OK ] [ 0.00001811 / 0.00001806 CPU ]zap.zap_removetype=micro [ OK ] [ 0.00001190 / 0.00001192 CPU ]type=fat [ OK ] [ 0.00002998 / 0.00002978 CPU ]zap.zap_counttype=micro [ OK ] [ 0.00001882 / 0.00001843 CPU ]type=fat [ OK ] [ 0.00001707 / 0.00001706 CPU ]zap.zap_containstype=micro [ OK ] [ 0.00001400 / 0.00001359 CPU ]type=fat [ OK ] [ 0.00001650 / 0.00001651 CPU ]zap.zap_lengthtype=micro [ OK ] [ 0.00001039 / 0.00001038 CPU ]type=fat [ OK ] [ 0.00002450 / 0.00002413 CPU ]zap.zap_incrementtype=micro [ OK ] [ 0.00001459 / 0.00001457 CPU ]type=fat [ OK ] [ 0.00002773 / 0.00002737 CPU ]zap.zap_inttype=micro [ OK ] [ 0.00002339 / 0.00002288 CPU ]type=fat [ OK ] [ 0.00003203 / 0.00003164 CPU ]zap.zap_int_keystype=micro [ OK ] [ 0.00001585 / 0.00001579 CPU ]type=fat [ OK ] [ 0.00004489 / 0.00004479 CPU ]zap.microzap_stats [ OK ] [ 0.00001210 / 0.00001212 CPU ]zap.fatzap_stats [ OK ] [ 0.00001901 / 0.00001899 CPU ]zap.uint64_keys [ OK ] [ 0.00001770 / 0.00001764 CPU ]zap.cursortype=micro [ OK ] [ 0.00002167 / 0.00002166 CPU ]type=fat [ OK ] [ 0.00003274 / 0.00003226 CPU ]zap.cursor_serializetype=micro [ OK ] [ 0.00002177 / 0.00002173 CPU ]type=fat [ OK ] [ 0.00002936 / 0.00002924 CPU ]zap.cursor_release_unusedtype=micro [ OK ] [ 0.00001119 / 0.00001115 CPU ]type=fat [ OK ] [ 0.00001867 / 0.00001856 CPU ]zap.cursor_release_advancetype=micro [ OK ] [ 0.00001058 / 0.00001056 CPU ]type=fat [ OK ] [ 0.00001544 / 0.00001544 CPU ]zap.cursor_release_emptytype=micro [ OK ] [ 0.00000918 / 0.00000907 CPU ]type=fat [ OK ] [ 0.00001564 / 0.00001563 CPU ]zap.cursor_release_onetype=micro [ OK ] [ 0.00001696 / 0.00001662 CPU ]type=fat [ OK ] [ 0.00001843 / 0.00001841 CPU ]zap.zap_value_searchtype=micro [ OK ] [ 0.00001555 / 0.00001551 CPU ]type=fat [ OK ] [ 0.00002377 / 0.00002379 CPU ]zap.zap_value_search_masktype=micro [ OK ] [ 0.00001525 / 0.00001524 CPU ]type=fat [ OK ] [ 0.00002293 / 0.00002288 CPU ]41 of 41 (100%) tests successful, 0 (0%) test skipped. -
This workflow automates the detection of OpenZFS CI failure detection via Gotify notifications !!
-
My home server configuration:
The server runs TrueNAS SCALE (release 26.0.0-BETA.1) on an AMD Ryzen 7 PRO 8845HS, with 32 GiB of memory and no swap configured. Alongside ZFS it hosts a substantial collection of services including Portainer-managed containers, immich, Forgejo, FreshRSS, Jellyfin an automation platform, several PostgreSQL databases, an identity provider, and a number of smaller tools. At the moment of measurement, the operating system reported the following:
total used free buff/cache available Mem: 30Gi 27Gi 1.2Gi 3.8Gi 3.3GiUptime stats:
The following figures come from
/proc/spl/kstat/zfs/arcstats, accumulated over an uptime of 53 days:Metric Value ARC size 5.04 GiB ARC target (c) 5.09 GiB Maximum (c_max) 29.68 GiB Minimum (c_min) 0.96 GiB L2ARC none configured The hit and miss rates, derived from the raw counters, break down as follows:
Access class Hit rate Miss rate Overall 96.4% 3.6% Demand data 98.05% 1.95% Demand metadata 97.91% 2.09% Prefetch data 6.0% 94.0% Prefetch metadata 66.9% 33.1% Live view of hit/miss performance:
Because the counters above are cumulative since boot, they represent an average spanning nearly two months and cannot, on their own, describe the system’s present behaviour. To capture that, I sampled the cache once per second under active load using
arcstat:time read ddread ddh% dmread dmh% pread ph% size c avail 22:23:43 949 867 98.8 77 98.7 5 0 5.3G 5.3G 118M 22:23:44 668 384 97.7 284 100 0 0 5.3G 5.3G 425M 22:23:45 1.5K 1.1K 98.5 432 100 39 2.6 5.3G 5.3G 367M 22:23:46 686 421 98.6 248 100 17 0 5.3G 5.3G 344MWith reads peaking at roughly 1500 per second, the demand-data hit rate held steady at approximately 98% matching that of the uptime, thus confirming that the long-term average is not concealing a recent decline in performance. The cache is presently serving requests just as effectively as it has, on average, throughout its uptime.
ddh% : Demand-data hit
dmh% : Demand-metadata hit
Stats were obtained from
/proc/spl/kstat/zfs/arcstats, with live sampling viaarcstat.The equivalent figures are available on FreeBSD under
kstat.zfs.misc.arcstats.
