NoSignups (FckSignups)
NoSignups search under a microscope: ASCII tokens, substrings and ties
Read tokenize and matchScore with concrete edge cases, then distinguish runtime JSON validation from TypeScript types and README promises.
What you will learn
- ASCII tokenization can turn meaningful input into an empty query.
- includes performs substring matching, and repeated tokens count repeatedly.
- The JSON-LD schema file is not runtime catalogue validation.
Before you start
- Basic JavaScript and JSON knowledge
- A harmless sample task and catalogue fixture
Explain the chapter boundary and verify it using the proposed fixture or review record.
Key takeaways
- ASCII tokenization can turn meaningful input into an empty query.
- includes performs substring matching, and repeated tokens count repeatedly.
- The JSON-LD schema file is not runtime catalogue validation.
Read the tokenizer literally
tokenize lowercases its input, splits on characters outside a-z, 0-9 and plus, then removes empty pieces. Consequently video-editor and video_editor become the same two tokens, and c++ retains its plus signs. Accented letters are separators rather than normalized letters, and a query containing only Chinese characters produces no tokens. The regular expression defines the behavior; the interface language does not change it.
This matters because zero tokens are treated as no search. A visitor entering a meaningful Chinese query can receive the whole selected category rather than an explicit unsupported-query message. A multilingual fork should first define its intended behavior for Unicode words, accents and identifiers. Changing the regular expression without fixtures can accidentally damage programming-language queries such as C++ or shorten meaningful names.
A substring match is not a word-boundary match
matchScore builds one lowercase string from the name, description and tags, replaces the same non-ASCII separators with spaces, and counts tokens for which haystack.includes returns true. A query token art therefore matches the substring in cart. Repeated query tokens are not deduplicated: art art can score two because the same substring satisfies both occurrences. These examples follow the function body, not a live search benchmark.
The hook retains only records whose score equals the query-token count. Searching video editor means both tokens must occur somewhere in the combined text, but not necessarily next to each other or in the same field. After that filter, score ties are inevitable, so stars determine the remaining sort order before editorial grouping. A future ranking change should be described as a product decision, not a correction that is automatically better for every reader.
Types do not inspect a downloaded object
The loadTools return expression asserts that parsed JSON is ToolsData. TypeScript removes that assertion at runtime; the inspected function only catches HTTP or JSON parsing failures in this path. It does not prove unique tool IDs, valid category references, supported URL schemes or a tags array for every entry. The file named schema.js contains website JSON-LD, not a catalogue validator.
A useful source-reading exercise is to build a small fixture matrix before modifying anything: a missing tags field, a duplicate id, an unknown category and non-Latin search. Define whether each case should be rejected, repaired or visibly reported. The sample below reproduces the tokenizer in isolation; it is not the complete upstream hook, and passing it does not validate the full application or its public catalogue.
Implementation steps
- 1
Read tokenize and matchScore at the pinned commit.
- 2
Run the isolated token fixture and record expected results.
- 3
Add substring, repeated-token and malformed-record cases before changing search.
Copy-ready example
const tokenize = text => text.toLowerCase()
.split(/[^a-z0-9+]+/).filter(Boolean);
for (const query of ["video-editor", "c++", "café", "视频编辑", "art art"]) {
console.log(JSON.stringify(query), tokenize(query));
}
// Isolated logic fixture, not the full React hook.Frequently asked questions
Does search use exact words?
No. The inspected matcher uses includes on combined text, so a shorter token can match inside a longer word.
Does schema.js validate tools.json?
No. That file exports WebSite structured data for HTML injection. The inspected loader uses a TypeScript assertion rather than per-record runtime validation.
Sources
- README.mdSource checked 2026-09-07
- package.jsonSource checked 2026-09-07
- vite.config.mtsSource checked 2026-09-07
- src/hooks/useTools.tsSource checked 2026-09-07
- src/components/Home/Tools/Tools.tsxSource checked 2026-09-07
- src/components/Home/Tools/ToolCard/ToolCard.tsxSource checked 2026-09-07
- src/types/index.tsSource checked 2026-09-07
- src/constants/fallbackData.tsSource checked 2026-09-07
- src/data/schema.jsSource checked 2026-09-07
- cloudflare-worker/worker.tsSource checked 2026-09-07
- cloudflare-worker/urlHandlers/handleSubmitTool.tsSource checked 2026-09-07
- cloudflare-worker/utils.tsSource checked 2026-09-07