How Marketing Agencies Can Build Context, Memory & MD-File Agent Systems | Dave Sifry

Start building your own agent with memory and MD files → https://value.8figureagency.co/hermesstart

MetaSwarm (open source) — Dave's agentic harness → https://github.com/dsifry/metaswarm

18 default agents, each with definitions and rubrics, plus defined workflows for software development. Three ways to use it: build agentic systems with it, point it at your own SOPs to analyze them, or study how it does agentic decomposition. (Fun fact from the episode: search "metaswarm" and it ranks right behind Meta itself.)

Dave Sifry has started nine companies. Now he's building a company that builds companies.

In this one, he hands agency owners the framework for the thing almost everyone is getting wrong with AI: memory and context. If your agency runs 50, 100, even 600 clients and information keeps falling through the cracks — stakeholders forgotten, context lost, the same questions re-answered — this is the episode.

No fluff. Dave breaks down what actually deserves to be remembered, the exact MD files that run an agent (soul.md, agents.md, heartbeat), and how to wire a team of agents that report to each other and improve themselves every single day.

What You'll Learn

Not all data is information. A JPEG is a mountain of data and almost no information. One sentence from the CEO saying "go do this" carries more than a thousand video keyframes. Store the signal, not the noise.

Memory is a compression problem. Save the process — who you talked to, how the deal got done, where it went off the rails — not every artifact you produced.

Timeliness is value. Information from 30 days ago usually beats information from 7 years ago. Decide what goes into deep storage and what stays live.

Treat every AI agent like a new employee who always needs onboarding. What's the minimum they need to do the job well? That's your context window.

Three layers of control: policy, guidelines, gates. Policy = "be nice to the client, don't cuss." Guidelines = your stop-word list. Gates = a deterministic wall, like a credit card with exactly $2,000 on it that rejects $2,001.

Run an analysis phase first. Find the 5–6 things a role does 80–90% of the day before you build anything. (An account manager is a farmer, not a hunter.)

The MD-file stack that runs an agent:

soul.md — who am I, why am I different from every other agent, what I do and what I don't do.

agents.md — the bootstrap: read your soul, read your heartbeat, and here are your SOPs as links (loaded only when the situation calls for it, so you never pollute the context window).

heartbeat — the recurring loop. The five tasks every hour, the three tasks every four hours. Keep it light or it eats you alive.

Great agent systems are an org-design problem, not a super-intelligence problem. You don't need one all-knowing brain. You need agents that each know their job and who to talk to. It works like an anthill.

Build a COO agent whose only job is writing and revising SOPs. Feed it a daily retrospective and you've built a self-learning organization