User-agent: * Disallow: /admin/* Disallow: /ads/* Disallow: /banner/* Disallow: /api/* Disallow: /check/* Disallow: /feed/*