Talos
A team chat platform whose assistant answers from your own documents, with citations.
Talos is workspaces, channels, DMs, and threaded messaging with an in-channel assistant that retrieves, reranks, and answers from a team's uploaded files. It was our graduation project. I owned the AI, retrieval, and evaluation track, and the one thing I'm proudest of is that the retrieval isn't asserted to work. I measured it.
My track: retrieval-augmented assistant, ingestion and retrieval, and the evaluation that proved the fix.
The problem
A team's real knowledge lives in its own files
A general chatbot is useless for a specific team, because the answers it needs are in that team's own documents, not in the model's training data. We wanted the assistant to sit inside the chat people already use, answer only from the workspace's uploaded files, point to the exact source of each answer, and say so when the files don't cover the question. Scoped per workspace, so one team's documents never leak into another team's answers.
What we built
A chat platform, with a grounded assistant inside it
The platform is the usual team surface: workspaces, channels, direct messages, group chats, threads, and mentions that turn into notifications. Sign-in runs over passwords, TOTP, Google and GitHub OAuth, and WebAuthn passkeys, with JWE-encrypted sessions. What makes it Talos is the assistant woven through it.
How it works
Upload, process, retrieve, rerank, answer
A file goes in once and gets processed out of band. A question runs through two retrieval stages before the model ever sees it. Here is the whole path.
Upload
A file lands in MinIO, checked by magic-byte MIME sniffing, capped, and SHA-256 deduped. The API returns 202 and hands off to a background worker.
Process
A taskiq worker parses the document, chunks it by title, embeds each chunk with bge-small, and writes the vectors into the workspace's Milvus collection.
Retrieve
A question runs dense search and BM25 in parallel, fused with reciprocal rank fusion. It fetches about 50 candidates so nothing good gets missed early.
Rerank
A cross-encoder reads the question and each candidate together and keeps the top 10. That second pass is what pulls the right passage up from the pack.
Answer
The model answers from the reranked passages only, streamed token by token with inline citations. Ask something the corpus doesn't cover and it says so instead of guessing.
Processing is asynchronous on purpose. The upload endpoint returns immediately and a taskiq worker does the slow work, so a half-ingested file can never be queried as if it were ready. Storage and search stay separate: MinIO holds the raw bytes and issues short-lived presigned download URLs, Milvus holds the vectors, one collection scoped to each workspace. Soft-deleting a file removes its chunks from Milvus too, so retrieval never surfaces something a user thought they deleted.
The hard problem
The assistant gave weak answers, so I measured why
Early on, the in-channel assistant kept giving thin, hand-wavy answers on a workspace whose corpus was a single 90-page guide. It wasn't the model and it wasn't the prompt. I went and looked at what retrieval was actually pulling, and the problem was upstream of all of it. That one document had been ingested with a recursive splitter that cut it into 1,778 tiny fragments, median 67 characters each. At that size, whole-page boilerplate outranked the real content, so the assistant was reading headers and footers instead of answers.
A hunch isn't a fix, so I ran a real experiment. I wrote 83 questions with page-level gold labels, LLM-authored and then LLM-reviewed, and paraphrase-constrained so a question couldn't win just by sharing words with its source. The harness runs the exact production chunking, retrieval, and prompt. The only thing swapped out is the vector store, replaced with an in-memory one that ranks by the same cosine geometry, so the eval measures what actually ships. I swept 48 retrieval configurations, then took 5 end-to-end arms all the way to a graded answer, scored by a gpt-4o judge against the reference answer, paired per question, with Wilcoxon signed-rank tests, Holm correction, and effect sizes.
| Arm | Config | Correctness | Δ | Wilcoxon p | Effect r |
|---|---|---|---|---|---|
| A0 baseline | recursive chunks + MiniLM, rerank 20→5 | 0.657 | ref | n/a | n/a |
| A1 | by_title chunks + MiniLM, 50→10 | 0.843 | +0.186 | 8.2e-06 | 0.79 |
| A2 (winner) | by_title + bge-small, 50→10, rewrite on | 0.855 | +0.198 | 5.5e-06 | 0.81 |
| A3 | A2 without query rewrite | 0.849 | +0.192 | 3.3e-05 | 0.67 |
| A4 | A2 without reranker | 0.837 | +0.180 | 3.8e-05 | 0.64 |
Judged correctness is the gpt-4o judge's score against the reference answer, paired per question across 83 questions.
The reading is clean. Chunk hygiene was the whole ballgame: fixing the fragmentation alone lifted correctness by 18.6 points, before touching the embedder or anything else. Swapping MiniLM for bge-small added a small, consistent gain on top. The reranker earned its latency, since dropping it cost real accuracy end to end. Query rewrite was marginal on these standalone questions, but I left it on because a separate benchmark showed it worth about +0.41 recall@5 on conversational follow-ups, which is where it matters.
Where it runs
The same assistant, in Slack
A team doesn't always live in a new app, so the assistant meets them where they already are. An MCP server exposes Jira, GitHub, filesystem, chat, and RAG tools, and a Slack bot answers over the same workspace corpus, citing the exact PDF page it drew from.
Outcome
A+, demoed live, then decommissioned
The project got an A+. I deployed the stack for the defense, a Dockerized backend, worker, and scheduler on Railway with managed Milvus on Zilliz Cloud, and we ran the whole thing live, then took the hosted app down afterward. The screenshots here are from that running build.
If I kept going, I know the next moves. The honest gap in the evaluation is the lack of human relevance labels, so real qrels come first. After that, more than one corpus and more than one domain, to see which of these gains hold and which were specific to this document. Then the ordinary work of scaling: the retrieval path is sound, but the indexing and memory would need real load before I'd trust them under a busy team.
Contributors
The team
Talos was a team project. I owned the AI, retrieval, evaluation, and deployment track; here is everyone who built it.