Meta-ExternalAgent
Meta-ExternalAgent is the crawler Meta uses to collect publicly accessible web content for training its Llama models and improving Meta's AI products. It identifies itself with the user-agent token `Meta-ExternalAgent` and, per Meta's own developer documentation, honors robots.txt disallow rules for that token. It's a separate crawler from `facebookexternalhit`, which fetches a page only to build the link-preview card shown when someone shares a URL on Facebook, Instagram, or WhatsApp — blocking one has no effect on the other.
What Meta-ExternalAgent does
Meta-ExternalAgent fetches publicly reachable pages to build training data for Meta's foundation models (the Llama family) and to support Meta's AI products more broadly — the kind of content eligibility that can influence how Meta AI answers a question about a brand or product inside WhatsApp, Instagram, Facebook, or Messenger. It identifies itself with the token Meta-ExternalAgent, and Meta's crawler documentation states that it honors robots.txt disallow directives for that user-agent.
Meta-ExternalAgent vs. facebookexternalhit
Site owners frequently conflate these because both come from Meta and both fetch web pages, but they serve unrelated purposes:
| Meta-ExternalAgent | facebookexternalhit | |
|---|---|---|
| Purpose | AI training / Meta AI products | Link-preview generation (title, description, thumbnail) |
| Triggered by | General crawling | A user sharing a URL on Facebook, Instagram, or WhatsApp |
| Blocking it affects | AI training eligibility only | Link previews only |
Blocking Meta-ExternalAgent in robots.txt has no effect on whether a shared link renders a preview card on Facebook or Instagram — that's facebookexternalhit's job, and it needs to keep fetching for previews to keep working. Conflating the two is the most common mistake when writing a bot-blocking rule for Meta's crawlers specifically.
Controlling it in robots.txt
User-agent: Meta-ExternalAgent
Disallow: /
Because Meta documents this token as robots.txt-compliant, a disallow rule is a reasonably reliable opt-out — unlike some other AI training crawlers, there isn't the same body of reporting showing it ignoring the directive. A site that wants to preserve link previews while opting out of AI training crawling should scope the rule to Meta-ExternalAgent specifically rather than blocking every Meta-associated user-agent.
Frequently asked questions
Does Meta-ExternalAgent honor robots.txt?
Yes, per Meta's own developer documentation, which states that a disallow rule targeting the `Meta-ExternalAgent` user-agent token will stop it from crawling a site.
Will blocking Meta-ExternalAgent break my Facebook link previews?
No. Link previews on Facebook, Instagram, and WhatsApp are generated by a different crawler, `facebookexternalhit`, which fetches a page only when a user shares its URL. Blocking Meta-ExternalAgent stops AI training crawling without affecting how shared links render.
What does Meta use Meta-ExternalAgent's data for?
Training Meta's Llama foundation models and improving Meta's AI products, including Meta AI as it appears across Meta's family of apps. It is not used to generate link-preview cards — that's a separate crawler's job.
Is Meta-ExternalAgent the only Meta crawler I need to worry about for AI training?
It's Meta's primary documented crawler for AI training purposes as of this writing, but Meta, like other AI labs, has adjusted its bot identities over time. Check current traffic logs and Meta's developer documentation for any additional tokens before finalizing a robots.txt policy.