vodoo-bot

VODOO is a studio where people make advertising. Part of that is pointing it at a product page — usually their own — and letting it read the title, the price, the pictures and the description so they do not have to retype them.

When that happens the fetch comes from our servers, with our address and our name on it, so we call it a crawler and hold it to a crawler’s manners. It identifies itself on every request as:

vodoo-bot/1.0 (+https://vodoo.ai/bot; on-demand fetch for the account owner)

Only when a signed-in person asks it to, by giving us one address. It does not discover sites, does not crawl outward from a page it has read, and keeps no queue to come back to. If you have seen it, somebody typed your address into VODOO.

The one thing it follows on its own is a redirect, up to four hops — enough for the http → https → www → cdn chain a real shop has, and no more. Every hop is treated as a brand new address and checked again from scratch.

It reads your robots.txt before it reads anything else, and it obeys it. The rule it applies is the standard one: the longest matching path wins, and Allow beats Disallow on a tie, so Disallow: / alongside Allow: /products/ means what you meant it to mean. * and a trailing $ work.

When your file says no, the fetch does not happen. The person who asked is told their page is not available to us, and the refusal is recorded against their request rather than quietly retried.

• Hold a connection open. Ten seconds and it gives up.
• Read a large page. It stops at 2 MB of HTML, and at 12 MB for a single picture.
• Sign in, submit a form, run your JavaScript, or accept a cookie.
• Touch anything that is not a public address. Private and internal ranges are refused before the request is made, on the first address and on every redirect after it.
• Reproduce your wording. Structure may travel into a customer's own brief; your headlines, your script and your footage do not, and the product checks the output against the source and refuses work that has copied text through.

Put this in your robots.txt and we will stop, everywhere on your site:

User-agent: vodoo-bot Disallow: /

Match on the token vodoo-bot rather than the whole string — the version number in our user agent will move and your rule should survive it. To allow some of the site and not the rest, write the paths out; we apply them the way the standard says.

If vodoo-bot has done something you did not expect, or you want it kept away from a site without editing robots.txt, write to [contact address] with the address and we will add it on our side. You do not need an account, and you do not need to explain why.

The acceptable use policy covers what our customers are allowed to do with anything it reads.