kdrant-core/dev.kdrant.model/Tokenizer

Tokenizer

@Serializable

enum Tokenizer : Enum<Tokenizer>

How a text index splits a value into the tokens a matchText filter is matched against.

Entries

@SerialName(value = "prefix")

Every prefix of every word, so a partial word matches. Largest index.

@SerialName(value = "whitespace")

Split on whitespace only, keeping punctuation attached to the token.

@SerialName(value = "word")

Split on whitespace and punctuation. The usual choice for prose.

@SerialName(value = "multilingual")

Word-splitting that also handles scripts without spaces, such as Chinese and Japanese.

Properties

val entries: EnumEntries<Tokenizer>

Returns a representation of an immutable list of all enum entries, in the order they're declared.

expect val name: String

expect val ordinal: Int

Functions

fun valueOf(value: String): Tokenizer

Returns the enum constant of this type with the specified name. The string must match exactly an identifier used to declare an enum constant in this type. (Extraneous whitespace characters are not permitted.)

fun values(): Array<Tokenizer>

Returns an array containing the constants of this enum type, in the order they're declared.