DedupCommando - DedupCommando release notes

DedupCommando v0.9.2 release notes

2026-09-29T00:00:00+00:00

For anyone running 0.9.1. This release checks every action before the first snapshot is taken, makes a saved plan script as careful as applying with Y</code>, stops a scan within about a second, keeps hashing fast in a large scan, refuses command lines it would half-ignore, and makes mouse clicks and keys act on what the screen shows. The database stays at schema v6, so nothing is migrated.</p>

Applying actions</h2> Before the first snapshot, every action is checked: a read-only file system; chattr +i</code> or +a</code> on the file, its directory or the quarantine directory; no space or an exhausted quota; a file reached through a second mount (a bind mount) or lying outside its dataset's mount point; for a hardlink, the keeper as well. Such an action is refused with its reason and without reading any file; the others run. A snapshot is taken only of the datasets where there is work.</li> If nothing in a batch can run, nothing changes and no snapshot is taken: nothing done — N actions cannot run: <reason>; no snapshot taken, marks kept; first: <path></code>, and the plan goes back to the window where it was confirmed.</li> Reflink on a host without block cloning (OpenZFS older than 2.2.1, or zfs_bclone_enabled=0</code>), or on a pool whose feature@block_cloning</code> is disabled, is refused before the confirmation window, with the number of marks and what to do instead.</li> A hardlink or reflink whose keeper is on another dataset is refused by the plan itself, before any snapshot or read: cannot hardlink or reflink across datasets (N marks) — …</code>. Reflink across the datasets of one pool never worked; the manual no longer promises it.</li> Hardlink and reflink of a file whose name is up to the 255-byte limit no longer fail with File name too long</code>, and a CSV export can be written to a file with such a name.</li> A failed replacement names the real cause: the prepared replacement vanished, no space, the quota, a read-only file system, an I/O error. If another file took the original's place, so that the original cannot go back, the message gives its exact path in the quarantine. A failed database write during a scan names the operating system's error, and a problem with the database file or the lock says what was found under that name.</li> </ul> A saved plan script</h2> The script that F11 → S saves now takes the same steps as applying with Y</code>:</p> Before each action it compares the file with its keeper byte for byte (cmp</code>) and leaves alone a file that has changed or become a symbolic link (dedcom: skip <file>: …</code>).</li> The replacement, a link or a clone, is made beside the file first. Only then does the file go to the quarantine, and it comes back if the replacement cannot be put in its place.</li> A reflink copy gets the owner, mode, times, extended attributes and ACL of the original.</li> An mv -n</code> that moved nothing no longer counts as a deletion, and a file whose quarantine is on another mount (a bind mount, or a link to another file system) is refused before the move.</li> One failed action no longer stops the rest. The script ends with dedcom: N of M actions not carried out — each is named above</code> and exit code 1.</li> The Commands tab shows one function call per action; the functions are defined at the top of the script.</li> </ul> Scanning</h2> A stop during hashing (Esc on the scan screen; Ctrl+C, SIGTERM</code> or SIGHUP</code> for --scan</code>) takes effect within about a second, even in the middle of a large file. Before, the scan first finished reading every file of the current batch of up to 64: tens of minutes for a 500 GB disk image. The files being read at that moment get no hash and are read again from the start on resume.</li> Hashing no longer slows down as a large scan goes on: 100,000 small files take 11 s instead of 6 minutes on this project's test machine, and a scan of 2 million files on a home server hashes about 300 files per second instead of 14.</li> --scan</code> of a root that does not exist or cannot be read is refused with exit code 1 before the lock and the database are touched, instead of ending as an empty "successful" scan that trimmed the history. A resumed scan prints the settings it keeps and refuses a flag it would ignore; --no-resume</code> starts over.</li> A scan that finds no file under a root where earlier scans of the same roots found files, usually the empty mount point of a dataset that did not mount, no longer sends those scans to the trash and says History kept: …</code>.</li> A directory that cannot be opened (removed during the scan, a path longer than 4,095 bytes, no search permission, a disk error) is one walk error instead of the end of the scan: Omissions: … walk errors</code>, ⚠ gaps</code> in the interface, and a walk error: <path>: <reason></code> line in dedcom.log</code>.</li> --include-ext tar.gz</code> finds photos.tar.gz</code>, '.JPG'</code> means jpg</code> on the command line and in presets.json</code>, and a value with /</code> is refused.</li> </ul> The command line and config.json</code></h2> An argument that is not valid UTF-8 no longer stops dedcom with a panic: the state directory and the export file accept such names, and a scan root gets a clear refusal.</li> Two modes at once, a repeated --export-csv</code>, or a flag the run does not read are refused with exit code 2 and an explanation, instead of half of the command being dropped. So are --verify</code>, --merkle-dirs</code>, --strict-verify</code> and --no-resume</code> beside --read-only</code>. A window started with --read-only</code> answers an action with (started with --read-only)</code>.</li> Refusals show invisible direction-control characters escaped.</li> A config.json</code> that cannot be read (not JSON, not an object, not a regular file) is no longer rewritten and decides nothing. Each writing run says so in one line; automatic VACUUM and history trimming are skipped until the file is fixed, and the lock policy falls back to ask</code>. A field of the wrong kind turns off only its own decision. config.json</code> is now written by replacing the file, with mode 0600.</li> </ul> Clicks and keys act on what is drawn</h2> A click on a function-key number in the commander's bottom bar runs that key at any terminal width. Before, at widths such as 80, 100, 160 and 200 columns, eight of the twelve numbers ran the key to their left: "9 Menu" set a Delete mark, "11 Exec" quit.</li> A click or a double-click in a list of groups or of a group's files, in the wizard and in the commander, scrolled or not, takes the row drawn under the pointer. Before, the wizard took the row below it, and a double-click saved the wrong keeper; in the commander such a click dropped the cursor. PgUp and PgDn move by the visible part of the list.</li> In the wizard's group view, 1 opens Folders and 2 opens Files, as labelled.</li> While the help, the start-up notice, the role prompt, a function-key window or a yes/no question is open, a click does nothing: close the window first. Before, a click under the help could set a Delete mark. On the Triage Board, a click no longer moves the focus while a receiver waits for its digit.</li> The "Please wait" box is no longer drawn over a question or a window that is waiting for a key. Before, it could hide "Move to trash?" or "Purge from trash?", where Enter means yes.</li> </ul> A late answer does not take over an open window</h2> When F2, F3 or F11 answers after you opened the F9 menu, another window, the help or the Triage Board, its window no longer opens over them, and no scan starts from under them. The status line says what arrived and which key to press again ("Close this window, press F2 again: …").</li> A scan result that finishes opening after you opened a menu, a window or the Triage Board, or moved to another screen, opens without switching the screen; the status line names the scan and says how to show it. The help does not count: the groups open under it. Before, the group view took over, and an Enter meant for the menu saved a keeper. A plan finished after you left the group view is dropped; r</code> builds it again.</li> </ul> Names that are not valid UTF-8</h2> For a file whose name shows with a ?</code>, F3 no longer shows another file's checksum and duplicates, "duplicates of the cursor" no longer shows another file's group, and a directory no longer shows another directory's total size. Both screens say instead that the scan has no such name, and why.</li> The manual (§11.4) no longer says such a name is exported. It says where these files are counted (Omissions:</code> in the --scan</code> output, ⚠ gaps</code> in the TUI) and how to ask about one file (F3 in the commander). The sample --scan</code> output in §11.1 matches what the program prints.</li> </ul> The cursor in a large scan</h2> In "duplicates of the cursor" a cursor step costs a fraction of a millisecond and no longer grows with the scan; in 0.9.1 it was about 70 ms on a scan of a million files, on this project's test machine. Building the F11 plan no longer walks the scan for every mark and group.</p> The manual</h2> A new §8.9 on backups and the stores of other programs (Time Machine, restic, borg, kopia, Arq, Duplicacy, Proxmox Backup Server, iPhone backups, virtual machine disks): do not scan a store, stop the program before applying, apply with Y</code> rather than a saved script, verify with the program's own tools before cleaning anything. Wherever the manual cleans the quarantine, the command is dedcom --purge-quarantine --yes</code>.</li> §11 gives a ready cron line that runs at low priority and checks that the pool is mounted. Chapter 13 quotes every reason an action can be refused before the snapshots, and every line a saved script prints; a test checks the quotes against the program.</li> </ul> What it does not establish</h2> Unicode formatting characters in names, such as bidirectional overrides and zero-width characters, are not escaped on screen; a terminal with bidi support may reorder a row that shows one. Command-line refusals now show them escaped.</li> A plan saved as .sh</code> holds the real bytes of the names, as it must to run, so cat</code> of that file still passes an escape sequence in a name to the terminal. The Commands tab shows them escaped.</li> The confirmation window does not yet show which actions cannot run; their refusal comes after you confirm. Planned for 0.9.3.</li> A stop is not seen during the --verify</code> byte comparison, nor while Y</code> re-reads both files before an action: those finish first. Planned for 0.9.3.</li> nohup</code> does not keep --scan</code> running after an SSH session drops: the hang-up stops the scan. Run a long scan over SSH in tmux</code> or screen</code>. Planned for 0.9.3.</li> A stopped --scan</code> exits with code 0, as a finished one does. Planned for 0.9.3.</li> The memory warning before grouping advises Esc, but a stop is noticed only after grouping has built its structures. Planned for 0.9.3.</li> A dataset below a scan root that did not mount is not noticed: the scan finds files in the rest of the root and trims the history as usual.</li> A path within about 60 bytes of the 4,095-byte limit can still fail a hardlink or reflink after the snapshot. A directory the walk entered but could not read the metadata of still stops the scan.</li> The first visit to a very large directory still costs one pass over its rows, seconds at the top of a pool of millions of files.</li> A files panel shows only the marks set while it was open. Marks saved earlier show in "group files" and on the F11 confirmation.</li> While a scan opens, the "Please wait" box still covers the middle of the screen, and keys act on what is under it: in the commander that can be the file under the cursor.</li> If the list of scans is read again while you are on it, the cursor can go back to the first row, and the next Enter or Del acts on that row; "Move to trash?" and "Purge from trash?" do not name the scan. Planned for 0.9.3.</li> If a scan finishes but its result fails to open, the scanning screen stays, and only q</code> leaves it. Planned for 0.9.3.</li> </ul> On upgrade</h2> The 0.9.1 database opens as it is.</li> A command line that combines modes, repeats --export-csv</code>, or passes a flag the run does not read now fails with exit code 2. Check cron lines and aliases: alias dedcom='dedcom --strict-verify'</code> now breaks dedcom --stats</code> and dedcom --read-only</code>.</li> An unfinished scan started by 0.9.1 with --include-ext '.jpg'</code> does not resume from the same command line (it kept *.jpg</code>, the new run asks for jpg</code>): start it over with --no-resume</code>.</li> A saved plan script needs cmp</code> (diffutils) and findmnt</code> (util-linux) besides coreutils, all three required packages on Debian 12 and 13. It reads every file it works on and its keeper in full: a pair of 100 GiB images on a pool of hard disks at 150–200 MB/s takes 17–23 minutes.</li> Snapshots are taken only of the datasets where at least one action can run. The pools' feature@block_cloning</code> is read once at start: restart dedcom after changing it.</li> dedcom --read-only</code> no longer starts when dedcom.lock</code> is a directory.</li> After --stats</code>, --export-csv</code> or a --read-only</code> window, dedcom.db-wal</code> (0 bytes) and dedcom.db-shm</code> (32 KiB) can stay in the state directory. They are harmless and the next writing run removes them; do not delete them by hand.</li> From 0.9.0-beta.4 or earlier, what the v0.9.1 notes say on upgrade applies as well.</li> </ul>



DedupCommando v0.9.1 release notes
2026-09-23T00:00:00+00:00

For anyone running 0.9.0-beta.4 or an earlier pre-release. This release closes ways a file name or an
argument could reach the operator's terminal or files, and makes the commander fast inside a large
scan. The database stays at schema v6, so nothing is migrated. The number drops the -beta</code> suffix;
the leading 0 still marks the project as a beta.</p>
A file name can no longer drive the terminal</h2>
A file name holding an escape sequence could retitle the operator's terminal or clear the screen
when the commander listed it. Every screen now shows paths escaped, the status line is escaped again
where it is drawn, and each frame is checked before it reaches the terminal. In a CSV export, control
characters in a name are written as \n</code>, \u{1b}</code> and so on: printing the file runs nothing, and each
record stays one line. The reason a scan stopped, the --stats</code> environment line and error messages
are escaped as well.</p>
No writing through links or into a directory that is not dedcom's</h2>

--export-csv</code> onto a symlink replaced the symlink itself; as root, a typo could turn /dev/stdout</code>
into a regular file until reboot. It now refuses, and the export never writes through a link.</li>
dedcom.lock</code> and consent.json</code> were written through a symlink, a hardlink or a device node under
their name. Only a regular file with exactly one name is accepted now, and the refusal names what
was found.</li>
--state-dir</code> took any directory and changed its permissions to 0700. An existing directory that
holds neither dedcom's database nor its lock, but holds something, is now refused with an
explanation, and so is an empty directory directly under /</code>. A config.json</code> put into a new
directory by hand before the first start is refused too. --stats</code> and --export-csv</code> no longer
create the state directory.</li>
</ul>
Two panics fixed</h2>
F3 in a window lower than five rows panicked and left the terminal in raw mode. dedcom --stats | head</code> exited with 101 and a panic; it now exits quietly.</p>
The commander inside a large scan</h2>
Changing directory inside a scanned tree took seconds of CPU per change: on a scan of 20,000 files,
about 3.8 s in beta.4, about 20 ms now, and F3, opening a group and marking no longer wait behind it.
At the top of a large scan the panel no longer reads the whole subtree on every refresh: the first
visit to a directory is one pass over its rows, and going back up or coming back is immediate while
the database does not change.</p>
Marks show and do what the database holds</h2>

"group files" and "duplicates of the cursor" show a mark as soon as it is saved; before, a new mark
appeared only after the cursor left the group and came back.</li>
F9 → "Clear all marks" asks first and then clears every saved mark of the scan, keepers and marks
set in an earlier session included. Before, it cleared one panel on screen while the database kept
every mark, and F11 still built its plan from them.</li>
A mark on a path that is not valid UTF-8 is refused, and the status says why: a scan never records
such a path, and the mark could land on another file whose path looks the same.</li>
</ul>
The manual</h2>
The backup and restore scripts in §12 work for databases from 0.9.0-beta.1 and beta.2, step 8 of the
quickstart can be followed with the keys it names, and the sample screens and the --purge-quarantine</code>
output match the program.</p>
What it does not establish</h2>

Unicode formatting characters in names, such as bidirectional overrides and zero-width characters,
are not escaped; a terminal with bidi support may reorder a row that shows one.</li>
A command-line argument that is not valid UTF-8 still stops dedcom with a panic at start-up.</li>
A plan saved as .sh</code> holds the real bytes of the names, as it must to run, so cat</code> of that file
still passes an escape sequence in a name to the terminal. The Commands tab shows them escaped.</li>
The first visit to a very large directory still costs one pass over its rows, seconds at the top of
a pool of millions of files.</li>
In "duplicates of the cursor" each step of the cursor over a file reads that file's group again:
about 70 ms on a scan of a million files, on this project's test machine.</li>
A files panel shows only the marks set while it was open. Marks saved earlier show in "group files"
and on the F11 confirmation.</li>
For a name that is not valid UTF-8, F3 and "duplicates of the cursor" can answer for another file
whose name reads the same. Planned for 0.9.2.</li>
If the F2 check of saved scans answers while the F9 menu is open, its window replaces the menu, and
an Enter meant for the menu opens that scan's results, or resumes its unfinished scan. Planned for
0.9.2.</li>
The manual (§11.4) says a name that is not valid UTF-8 is exported with U+FFFD</code>; a scan does not
record such a name, so the export does not hold it. Corrected in 0.9.2.</li>
Hardlink and reflink refuse names longer than about 213 bytes, and a path longer than 4,096 bytes
stops the scan. Both are planned for 0.9.2.</li>
</ul>
On upgrade</h2>
--storage-type</code> accepts only hdd</code>, ssd</code> and nvme</code>; any other value is now an error. A
--state-dir</code> that dedcom does not recognise as its own is refused: point it to a new or empty
directory. beta.4 databases open as they are. From beta.1 to beta.3 the first start upgrades
dedcom.db</code> to schema v6, as beta.4 did: back it up first with the procedure in the manual (§12,
"Backing up and restoring dedcom.db</code>").</p>
With this release, 0.9.0-beta.1 to 0.9.0-beta.4 are no longer supported.</p>


DedupCommando v0.9.0-beta.4 release notes
2026-09-11T00:00:00+00:00

For anyone running 0.9.0-beta.3. Most of this release is about the checkpoint database,
dedcom.db</code>: how its move journal stores pathnames, and what the shape guard accepts before it
writes anything. One fix changes what you see on disk and in the duplicate list.</p>
Order and names on disk no longer depend on the machine</h2>
Two habits made one tree come out differently on two machines. Directory order was taken from
readdir</code>, and a file name was rebuilt through a lossy string that maps distinct bytes onto one.
Siblings are now ordered by their raw bytes, and a name keeps the bytes it had. A failed read no
longer counts as an answer to whether two files are duplicates, and a merge that leaves the source
behind is not reported as clean.</p>
The checkpoint moves to schema v6</h2>
The first start of this build upgrades dedcom.db</code> to schema v6 in one transaction. The
move_event</code> journal used to pass pathnames through a lossy UTF-8 conversion, so two names that
differ only in bytes outside UTF-8 collapsed into one record. v6 stores both pathnames as raw bytes
and marks each row with path_fidelity</code>: 1 for rows this build wrote, 0 for rows carried over from
the v5 journal.</p>
The table is rebuilt only after its stored definition and everything attached to it (indexes,
triggers, views, foreign keys) prove to be what this product wrote. Anything else stops the upgrade
with a message, and the data and the schema version stay as they were.</p>
The shape guard refuses more before it writes</h2>
Before the database is switched to WAL, the guard now refuses:</p>

a product name held in another letter case. SQLite resolves names without regard to ASCII case,
the guard compared them byte for byte, and a foreign SCAN_ROOT</code> could be adopted;</li>
two objects under one product name, such as a table and a trigger;</li>
a hidden or generated column under a product column name, and a column of a later schema in a
checkpoint that declares an older one;</li>
a virtual or shadow table (FTS, R*Tree and the like) under a product table name. It could carry
the right column names and be taken for the product's own.</li>
</ul>
A refused file keeps its bytes. Objects under names the product does not use, virtual tables
included, are left alone.</p>
Also in this release</h2>
The manual matches this release: the F9 menu, the wording of the size figure on the confirmation
and summary screens, and the session-retention default. A test now holds the manual to what the
code does. Two paths gained tests: the order of a walk over several roots that cannot be keyed, and
a refused stat</code> while listing files of the same size.</p>
What it does not establish</h2>

Journal rows carried over from v5 keep the text v5 had. Bytes lost then are not recovered, and
those rows are not claimed to be exact.</li>
The journal is still best-effort: a move whose record could not be written is not reported.</li>
The declared types and defaults of required columns are not compared, and a WITHOUT ROWID</code> or
STRICT</code> table under a product name passes as an ordinary table.</li>
</ul>
On upgrade</h2>
Back up dedcom.db</code> first with the procedure in the manual (§12, "Backing up and restoring
dedcom.db</code>"). A plain cp</code> misses the -wal</code> file. beta.2 and beta.3 refuse to open a v6 database
before writing to it, so going back means restoring that backup.</p>

DedupCommando - DedupCommando release notes

DedupCommando v0.9.2 release notes

DedupCommando v0.9.1 release notes

Two panics fixed</h2>
F3 in a window lower than five rows panicked and left the terminal in raw mode. `dedcom --stats | head</code> exited with 101 and a panic; it now exits quietly.</p>`

The manual</h2>
The backup and restore scripts in §12 work for databases from 0.9.0-beta.1 and beta.2, step 8 of the quickstart can be followed with the keys it names, and the sample screens and the `--purge-quarantine</code> output match the program.</p>`

DedupCommando v0.9.0-beta.4 release notes

On upgrade</h2>
Back up `dedcom.db</code> first with the procedure in the manual (§12, "Backing up and restoring dedcom.db</code>"). A plain cp</code> misses the -wal</code> file. beta.2 and beta.3 refuse to open a v6 database before writing to it, so going back means restoring that backup.</p>`

DedupCommando - DedupCommando release notes

DedupCommando v0.9.2 release notes

DedupCommando v0.9.1 release notes

Two panics fixed</h2> F3 in a window lower than five rows panicked and left the terminal in raw mode. dedcom --stats | head</code> exited with 101 and a panic; it now exits quietly.</p>

The manual</h2> The backup and restore scripts in §12 work for databases from 0.9.0-beta.1 and beta.2, step 8 of the quickstart can be followed with the keys it names, and the sample screens and the --purge-quarantine</code> output match the program.</p>

DedupCommando v0.9.0-beta.4 release notes

On upgrade</h2> Back up dedcom.db</code> first with the procedure in the manual (§12, "Backing up and restoring dedcom.db</code>"). A plain cp</code> misses the -wal</code> file. beta.2 and beta.3 refuse to open a v6 database before writing to it, so going back means restoring that backup.</p>

Two panics fixed</h2>
F3 in a window lower than five rows panicked and left the terminal in raw mode. `dedcom --stats | head</code> exited with 101 and a panic; it now exits quietly.</p>`

The manual</h2>
The backup and restore scripts in §12 work for databases from 0.9.0-beta.1 and beta.2, step 8 of the quickstart can be followed with the keys it names, and the sample screens and the `--purge-quarantine</code> output match the program.</p>`

On upgrade</h2>
Back up `dedcom.db</code> first with the procedure in the manual (§12, "Backing up and restoring dedcom.db</code>"). A plain cp</code> misses the -wal</code> file. beta.2 and beta.3 refuse to open a v6 database before writing to it, so going back means restoring that backup.</p>`