AbstractCreates a new token with an optional data payload.
Optionaldata: unknownArbitrary data to associate with this token.
AbstractgetReturns a short, unformatted string identifying this token.
A concise string label.
AbstractisDetermines whether this token matches another token.
The token to compare against.
true if the tokens are considered equivalent.
AbstracttoReturns a formatted string representation of this token, indented by the specified number of spaces.
Number of leading spaces.
A formatted, human-readable string.
Abstract base class for tokens used in longest-common-subsequence comparison.
Remarks
Subclass
ComparisonUnitTokento define how individual tokens are matched. Two built-in implementations are provided: ComparisonUnitTokenString (exact string equality with bullet-character normalisation) and ComparisonUnitTokenRegex (regular-expression matching).Each token may carry an optional data payload that is preserved through comparison but does not affect matching.
Example