AI & ML interests

Explainable AI, Rule-based models, Rule learning with LLMs, Hallucination detection, Fact checking LLMs

Recent Activity

adaamko  published a Space 20 days ago
KRLabsOrg/rulechef-live
adaamko  updated a Space 20 days ago
KRLabsOrg/rulechef-live
View all activity

adaamko 
posted an update 3 months ago
view post
Post
161
🥬 LettuceDetect v2 — span-level hallucination detection for code, tool output, and structured documents.

Hallucination detectors are trained on document QA, but agents ground their answers in source code, tool output and markdown. On code-agent answers, existing detectors reach 0.17 span-F1 and even 550B zero-shot judges at most 0.22.

We built a unified span-level benchmark — 74,285 newly constructed examples (145K+ with RAGTruth and 14-language PsiloQA folded in), every span typed and character-labeled — and trained two detectors on it:

🤖 KRLabsOrg/lettucedect-v2-qwen-2b — generative, typed spans + explanations in one pass, 32K context, **0.689 span-F1** (0.60 on code-agent)
⚡ KRLabsOrg/lettucedect-v2-mmbert-base — 307M multilingual encoder for high-throughput setups
🏷️ KRLabsOrg/lettucedect-v2-taxonomy-head — types the spans of any binary detector

It also reaches the best reported English PsiloQA IoU (0.724) and 81.8 RAGTruth example-F1, so specializing on code didn't cost general RAG performance.

📚 Dataset: KRLabsOrg/lettucedetect-code-hallucination
📄 Paper: https://arxiv.org/abs/2607.00895

The models are now integrated natively into vLLM Semantic Router — joint blog post on how it works: https://vllm-sr.ai/blog/lettucedetect-v2-generative-hallucination-detection
  • 3 replies
·
adaamko 
posted an update about 1 year ago
view post
Post
378
We introduced 🥬 TinyLettuce: lightweight hallucination detection with 17–68M encoders

Instead of relying on huge LLM judges that are slow and costly, we built tiny Ettin-based detectors that you can train in hours on a single GPU and run efficiently on CPU.

Here’s what’s inside:
1️⃣ Synthetic Data Generator
A toolkit to create hallucinations with controllable error types—no manual annotation bottlenecks.
2️⃣ TinyLettuce Models (17–68M)
Compact classifiers built on Ettin encoders, designed for efficiency (8K context, modern transformer backbone).
3️⃣ Data & Training Utilities
Scripts and APIs to generate domain-specific labeled pairs at scale, plus ~3.6k examples we used for training.
4️⃣ Open and MIT-licensed
Code, data, and models are freely available for research and production.
5️⃣ Performance Highlights
- TinyLettuce-17M reaches 90.87% F1 (synthetic), outperforming GPT-OSS-120B (83.38%) and Qwen3-235B (79.84%)
- Runs in real-time on CPU: low latency, minimal memory, and pennies per million checks
- Shows competitive results on RAGTruth benchmarks

🔗 Useful links:
Blog: https://huggingface.co/blog/adaamko/tinylettuce
GitHub: https://github.com/KRLabsOrg/LettuceDetect
PyPI: https://pypi.org/project/lettucedetect/
HF Collection: KRLabsOrg/tinylettuce-68b42a66b8b6aaa4bf287bf4
Notebook/Demo:
https://github.com/KRLabsOrg/LettuceDetect/blob/main/demo/tinylettuce.ipynb

⭐ Stars and feedback are always appreciated!