TF-IDF Skill Search in Matimo

Overview

Matimo uses TF-IDF (Term Frequency–Inverse Document Frequency) as the default search engine for discovering and ranking skills by keyword relevance. It matches words, not meaning. This document explains the algorithm, its implementation, practical usage, and when to consider alternatives.

Key takeaway: TF-IDF provides lightweight, zero-dependency keyword-relevance search suitable for 10–200 skills. For larger deployments or specialized ranking, you can plug in OpenAI, Cohere, or custom embedding providers.

---

TF-IDF Algorithm Explained

What is TF-IDF?

TF-IDF is a statistical measure that evaluates how important a word is to a document in a collection of documents. It consists of two parts:

1. Term Frequency (TF), How often a term appears in a document

TF(term, doc) = (1 + log(count of term in doc)) if count > 0 else 0

• Sublinear scaling prevents high-frequency words from dominating
• Logarithm dampens the effect of repeated terms

2. Inverse Document Frequency (IDF), How rare the term is across all documents

IDF(term) = log((total documents + 1) / (documents containing term + 1)) + 1

• Rare terms get higher weights (more discriminative)
• Smoothing constants prevent division by zero
• Common words (like “the”, “and”) get low IDF scores

3. TF-IDF Score, Product of TF and IDF

TF-IDF(term, doc) = TF(term, doc) × IDF(term)

Why TF-IDF for Skills?

Pros:

• Zero dependencies: Pure JavaScript, works out of the box
• Fast: O(n) on document size, O(m) on query size (m = vocabulary size)
• Deterministic: Same ranking every time (no API calls)
• Transparent: Easy to debug; you can see term weights
• Good enough: Accurately ranks 10–200 documents

Cons:

• Bag-of-words: Ignores word order and semantics (“dog bites man” vs “man bites dog” score identically)
• No synonyms: “tool creation” and “build tool” considered different terms
• Limited context: Can’t understand if similar words have different meanings
• Doesn’t scale: For 1000+ skills or multi-language, use neural embeddings

---

Implementation in Matimo

Core Components

1. TfIdfEmbeddingProvider Class

Located in packages/core/src/core/tfidf-embedding.ts:

export class TfIdfEmbeddingProvider implements EmbeddingProvider {
  private vocabulary: Map<string, number> = new Map();
  private idf: Float64Array = new Float64Array(0);
  private corpusSize = 0;
  private _dimensions = 0;

  fit(documents: string[]): void { ... }
  async embed(text: string): Promise<number[]> { ... }
  async embedBatch(texts: string[]): Promise<number[][]> { ... }
  embedSync(text: string): number[] { ... }
}

Methods:

• fit(documents: string[]): Build vocabulary and IDF weights from all skill files

  • Tokenizes each document (lowercase, splits on non-alphanumeric)
  • Counts document frequency for each term
  • Pre-computes IDF weights for fast lookup
  • Must be called once before embedding

• embed(text: string): Promise<number[]>: Convert text to TF-IDF vector

  • Tokenizes query
  • Computes TF weights
  • Multiplies by pre-computed IDF
  • Returns L2-normalized vector, each element divided by vector magnitude, so final length = 1 (preserves direction, removes length bias)

• embedBatch(texts: []): Promise<number[][]>: VectorBatch embedding

  • Maps embed() over array of texts
  • Useful for embedding all skills at startup

• embedSync(text: string): number[]: Synchronous version (internal use)

  • No async overhead in hot path

2. Cosine Similarity

export function cosineSimilarity(a: number[], b: number[]): number {
  // Dot product / (norm_a × norm_b)
  // Returns [-1, 1]: 1 = identical, 0 = orthogonal, -1 = opposite
  // Vectors pre-normalized, so result is in [0, 1] for positive weights
}

Why cosine similarity?

• Works in high-dimensional space (vocabulary size = dimension count)
• Metric-agnostic to vector magnitude (unit vectors)
• Mathematically stable and fast

Note on L2-Normalization: All vectors are L2-normalized before similarity computation, meaning each vector is divided by its Euclidean magnitude (√(sum of squares)) to produce unit vectors. This makes vectors comparable regardless of length:

L2-normalized = vector / √(x₁² + x₂² + ... + xₙ²)
Result: final length = 1, direction preserved

With L2-normalized vectors, cosine similarity simplifies to just the dot product, making computation faster.

3. Stopwords Filter

const STOPWORDS = new Set([
  'a', 'an', 'the', 'and', 'or', 'but', ...
]);

Purpose: Remove common English words that add noise without meaning

• 50+ stopwords built-in
• Reduces vocabulary size by ~40–60%
• Improves ranking by removing irrelevant matches

4. Result Mapping Optimization: O(1) Lookup

// OLD APPROACH (O(n²) complexity):
results = scored
  .filter((r) => r.score > 0.1)
  .sort((a, b) => b.score - a.score)
  .map((r) => {
    // Linear search inside map = nested loop!
    const skill = results.find((s) => s.name === r.skill.name);
    return skill!; // Unsafe non-null assertion
  });

// NEW APPROACH (O(n) complexity):
const skillByName = new Map(
  results.map((skill) => [skill.name, skill] as const)
);
results = scored
  .filter((r) => r.score > 0.1)
  .sort((a, b) => b.score - a.score)
  .map((r) => skillByName.get(r.skill.name))
  .filter((skill): skill is NonNullable<typeof skill> => skill !== undefined);

Why this matters:

Performance Improvement:

• Small catalogs (10–50 skills): Negligible difference (~0.1ms)
• Medium catalogs (50–200 skills): 2–5x faster (~1–3ms savings)
• Large catalogs (200–500 skills): 5–10x faster (~5–8ms savings)
• Very large catalogs (500+ skills): 10–50x faster (from 50ms → 5ms)

Complexity Analysis:

Approach	Time Complexity	Why
.find() inside .map()	O(n²)	For each scored result, scan entire results array
Map precompute	O(n)	Build map once (O(n)), then O(1) lookups × n scored results

Trade-offs:

Factor	O(n²) .find()	O(n) Map
Memory	Minimal	+~1KB per 100 skills (Map overhead small)
Startup	Slightly faster	+~0.5ms to precompute Map
Query time	Linear in result size	Constant per result
Type safety	Unsafe ! assertion	Safe type guard

Real-world impact:

• Semantic search results typically 5–20 items, so O(n²) worst-case is actually O(k² × m) where k = scored count, m = full result list
• With 200 skills and returning top 10: .find() does 10 × 200 = 2,000 comparisons; Map does 10 lookups
• Winner at scale: Map approach wins decisively, especially with pagination or large skill catalogs

When this matters: Use Map optimization when:

• Skill catalog > 100 items
• Frequent searches (user-facing agent)
• Query latency sensitive (sub-10ms target)
• Results paginated (can re-rank without full .find() scan)

When it doesn’t matter:

• Tiny catalogs (5–10 skills): negligible difference
• Offline batch processing: not user-facing, speed less critical

---

Integration with Matimo Skills System

Flow: From YAML to Search Results

1. MatimoInstance.init({ autoDiscover: true } or { skillPaths: [...] })
   ↓
2. SkillLoader loads all SKILL.md files from disk
   ↓
3. SkillContentParser extracts sections (name, version, description, content)
   ↓
4. SkillRegistry stores SKILL.md metadata + full text
   ↓
5. On the first search: TfIdfEmbeddingProvider.fit([all skill texts])
   ↓ (Vocabulary + IDF computed once, again only after skills change)
   ↓
6. matimo.semanticSearchSkills(query) called by agent
   ↓
7. Query → TF-IDF vector → Cosine similarity vs all skills
   ↓
8. Ranked results returned to agent

SDK APIs

semanticSearchSkills(query, options?): Promise<SemanticSearchResult[]>

options is { limit?: number; minScore?: number } (defaults: limit: 10, minScore: 0.1).

const results = await matimo.semanticSearchSkills('how to create a tool', { limit: 3 });

// Returns (scores from the 0.2.0 built-in skills):
[
  { skill: { name: 'meta-tools-lifecycle', description: '...', metadata: {...}, source: 'builtin' }, score: 0.36 },
  { skill: { name: 'tool-creation', ... }, score: 0.36 },
  { skill: { name: 'policy-validation', ... }, score: 0.26 },
]

score is the cosine similarity (0–1). With TF-IDF over short skill texts, a good match usually scores 0.2–0.4, so rank by score rather than expecting values near 1.

getSkillSections(name): Array<{ path, level, tokenEstimate }> | null

const sections = matimo.getSkillSections('tool-creation');
// [
//   { path: 'Tool Creation for Matimo SDK', level: 1, tokenEstimate: 189 },
//   { path: 'Tool Creation for Matimo SDK.Matimo Architecture Overview', level: 2, tokenEstimate: 4 },
//   { path: 'Tool Creation for Matimo SDK.Matimo Architecture Overview.Tool Execution Flow', level: 3, tokenEstimate: 55 },
//   ...
// ]

null means no skill has that name.

getSkillContent(name, options?): string | null

Synchronous. options is SkillContentOptions: sections (headings, case-insensitive partial match), maxTokens, includePreamble (default true) and maxDepth.

// Load the full skill
const full = matimo.getSkillContent('tool-creation');

// Load selected sections only (token-efficient)
const partial = matimo.getSkillContent('tool-creation', {
  sections: ['Tool Execution Flow'],
  maxTokens: 500,
});

---

Practical Examples

Example 1: Agent Discovering Skills by Natural Language

Agent Query: “I need to understand how to approve tools in the system”

const matimo = await MatimoInstance.init({ autoDiscover: true });

const results = await matimo.semanticSearchSkills('approve tools policy', { limit: 3 });
// → meta-tools-lifecycle (0.32), policy-validation (0.22)

// Load only the section the agent needs from the top match
const approval = matimo.getSkillContent('policy-validation', {
  sections: ['Human approval'],
});
// → "### Human approval\n\nA call asks a human first when its tool has `requires_approval: true`, ..."

Example 2: Agents Use the matimo_search_skills Meta-Tool

An agent does not call the SDK; it calls the meta-tools, which wrap these methods. Give a LangChain agent the skill meta-tools and it can search, inspect and load skills itself:

import { MatimoInstance, convertToolsToLangChain } from '@matimo/core';

const matimo = await MatimoInstance.init({ autoDiscover: true });
const skillTools = [
  'matimo_search_skills',
  'matimo_get_skill_sections',
  'matimo_get_skill_content',
].map((name) => matimo.getTool(name)!);

const tools = await convertToolsToLangChain(skillTools, matimo);
// Bind `tools` to your model. A typical run:
// 1. matimo_search_skills(query: "tool creation")
// 2. matimo_get_skill_sections(name: "tool-creation")
// 3. matimo_get_skill_content(name: "tool-creation", sections: [...])

See LangChain integration for a complete agent loop.

Example 3: Several Queries at Once

const queries = ['how do I validate a YAML tool?', 'how to write a skill.md file?'];

const allResults = await Promise.all(
  queries.map((q) => matimo.semanticSearchSkills(q, { limit: 1 }))
);

allResults.forEach(([best], idx) => {
  console.log(`${queries[idx]} → ${best?.skill.name} (${best?.score.toFixed(2)})`);
});
// how do I validate a YAML tool? → tool-creation (0.33)
// how to write a skill.md file? → skill-creator (0.36)

Example 4: Programming: Direct TF-IDF Vectors

For advanced use cases, manipulate embeddings directly:

import { TfIdfEmbeddingProvider, cosineSimilarity } from '@matimo/core';

const provider = new TfIdfEmbeddingProvider();

// Fit on corpus
const skillContent = ['Tool creation workflow...', 'Policy tiers...', '...'];
provider.fit(skillContent);

// Embed query
const queryVector = provider.embedSync('how to approve a tool');

// Embed all skills (can be cached)
const skillVectors = skillContent.map(s => provider.embedSync(s));

// Manual ranking
const scores = skillVectors.map(v => cosineSimilarity(queryVector, v));
const ranked = scores
  .map((score, idx) => ({ skillName: skillContent[idx], score }))
  .sort((a, b) => b.score - a.score);

console.log(ranked.slice(0, 3));

---

Performance Characteristics

Startup Cost

Rough estimates for planning, not measured benchmarks:

Skill Count | Fit Time  | Memory     | Query Time*
10          | 5ms       | 50KB       | 0.5ms
50          | 25ms      | 200KB      | 1.5ms
100         | 45ms      | 400KB      | 2.5ms
200         | 90ms      | 800KB      | 4.5ms
500         | 220ms     | 2MB        | 8ms
1000+       | 450ms+    | 4MB+       | 15ms+

Notes:

• Fit runs on the first search, and again only after skills are added or reloaded
• Query time linear in vocabulary size, not skill count
• Query time is synchronous (no I/O or external API)
• Memory grows with vocabulary (not #skills directly)
• *Query times assume Map-based O(1) lookup optimization (see Result Mapping Optimization)
• Without optimization, query times would be 3–10x slower at scale (e.g., 200 skills: 12–45ms instead of 4.5ms)

Scalability Limits

• Ideal range: 10–200 skills (sub-5ms queries with optimization)
• Acceptable: 200–500 skills (queries ~5–8ms, Map optimization essential)
• Degraded: 500+ skills (queries ~8–15ms, consider caching or alternative)
• Not suitable: 1000+ skills (queries >15ms even with optimization, use neural embeddings instead)

---

Extensibility: Plugging in Other Embedding Providers

The EmbeddingProvider Interface

export interface EmbeddingProvider {
  embed(text: string): Promise<number[]>;
  embedBatch(texts: string[]): Promise<number[][]>;
  /** Embedding dimensionality */
  dimensions: number;
}

There is no fit() in the interface: only the built-in TF-IDF provider needs one, and Matimo calls it itself.

Example: Using OpenAI Embeddings

import OpenAI from 'openai';
import { MatimoInstance, type EmbeddingProvider } from '@matimo/core';

export class OpenAIEmbeddingProvider implements EmbeddingProvider {
  private client = new OpenAI(); // reads OPENAI_API_KEY
  readonly dimensions = 1536; // text-embedding-3-small

  async embed(text: string): Promise<number[]> {
    const [vector] = await this.embedBatch([text]);
    return vector;
  }

  async embedBatch(texts: string[]): Promise<number[][]> {
    const response = await this.client.embeddings.create({
      model: 'text-embedding-3-small',
      input: texts,
    });
    return response.data.map((d) => d.embedding);
  }
}

const matimo = await MatimoInstance.init({ autoDiscover: true });
matimo.setSkillEmbeddingProvider(new OpenAIEmbeddingProvider());

// semanticSearchSkills() and the matimo_search_skills meta-tool now use OpenAI embeddings
const results = await matimo.semanticSearchSkills('policy approval');

Advantages of swapping providers:

• Better semantic understanding (synonyms, paraphrasing)
• Multi-language support
• Scales to 1000+ skills
• API cost per query (~$0.00001–0.0001 per embedding)
• Network latency (~100ms round-trip)
• Requires API keys

---

Common Pitfalls & Troubleshooting

Issue: Low Scores (0.3–0.5 range)

Cause: Query and skill content have few shared terms (stop words filtered out)

Solution:

// Before:
const results = await matimo.semanticSearchSkills('make');
// → Low scores, "make" is 4 letters, might match, but noisy

// After:
const results = await matimo.semanticSearchSkills('create a new tool');
// → Higher scores, more specific terms: "create", "tool"

Issue: Unexpected Ranking

Cause: Term frequency dominance (e.g., if “HTTP” appears 20x in one skill, it ranks high on “HTTP” queries even if not the best match overall)

Debugging:

// Use TfIdfEmbeddingProvider directly to inspect vectors
const provider = new TfIdfEmbeddingProvider();
provider.fit(skillContent);

const query = 'HTTP request';
const vec = provider.embedSync(query);

// High values at specific indices = strong signal for those terms
console.log('Query vector (non-zero indices):', vec
  .map((val, idx) => ({ idx, val }))
  .filter(x => x.val > 0.1)
  .sort((a, b) => b.val - a.val)
  .slice(0, 5)
);

Issue: Stopwords Filtering Too Aggressive

Symptom: Can’t find skills when querying with common terms (“How to…”)

Current stopwords: ~50 English words (see STOPWORDS in tfidf-embedding.ts)

Solution: For very specialized queries, disable stopword filtering:

// Modify tokenization in custom provider:
private tokenize(text: string): string[] {
  // Don't filter stopwords for domain-specific terms
  return text.toLowerCase().split(/[^a-z0-9]+/);
}

---

Best Practices

1. Query Writing

Good queries:

• “how to create a tool” (specific, multiple terms)
• “OAuth2 authentication flow” (domain terms)
• “approve and reload” (action + context)

Poor queries:

• “tool” (too generic, matches everything)
• “a” (single stopword, filtered out)
• “123” (numbers ignored, no semantic value)

2. Skill Content Quality

Well-structured SKILL.md:

# skill-name

> Brief one-line summary describing what agents learn

## Overview
2–3 sentences on the purpose and scope.

## Section 1: Key Concept
Detailed explanation, examples, code blocks.

## Section 2: Workflow
Step-by-step procedures.

## Best Practices
Do's and don'ts specific to this skill.

Poor SKILL.md:

• Single paragraph (no section headers to search on)
• Repeated terms (“tool tool tool”) → high TF but noisy
• Minimal documentation → low IDF discrimination

3. Caching Embeddings

For production systems with frequently re-indexed skills:

// Cache embeddings to file after fit()
const provider = new TfIdfEmbeddingProvider();
provider.fit(skillContent);

// Later: load from cache (avoid re-fit)
const cached = loadEmbeddingsFromCache();
if (cached) {
  setGlobalSkillEmbeddings(cached); // pseudo-code
}

---

References

• TF-IDF Fundamentals: https://en.wikipedia.org/wiki/Tf%E2%80%93idf
• Cosine Similarity: https://en.wikipedia.org/wiki/Cosine_similarity
• Matimo Skills System: Skills System Docs
• Embeddings for Production: OpenAI (text-embedding-3), Cohere, Hugging Face
• Stopwords Lists: NLTK, SpaCy (extensible in Matimo)

---

Conclusion

TF-IDF in Matimo enables lightweight, deterministic keyword-relevance search for discovering skills. For 10–200 skills, it’s production-ready. For larger deployments or specialized use cases (synonyms, multilingual, semantic nuance), plug in a neural embedding provider and enjoy the same API surface.