Algolia Crawler API
Browse all Algolia APIs★ Only Publicly Available OpenAPI DocumentDeveloper ToolsMonitoring Observabilitybasic20 EndpointsREST
For Agents
Drive Algolia's hosted web crawler programmatically: create crawlers, run and pause them, test URL extraction, and inspect crawl runs across 20 endpoints.
Use for: I need to create a new Algolia web crawler for a documentation site, Trigger a fresh reindex on an existing Algolia crawler, Pause an Algolia crawler before a major site migration, Test extraction on a single URL before kicking off a full crawl
Not supported: Does not serve search queries, ingest direct records, or manage A/B tests — use for managing Algolia hosted web crawlers, runs, and configurations only.
Jentic publishes the only available OpenAPI specification for the Algolia Crawler API, keeping it validated and agent-ready. The Crawler API drives Algolia's hosted web crawler — programmatically creating crawlers, updating their configuration, starting and pausing crawls, testing URL extraction, listing runs, downloading run logs, and managing registered domains. It is distinct from the standard Algolia search and ingestion APIs and uses its own credential pair. Authentication is HTTP basic auth with the Crawler user id and Crawler API key.
Install Jentic One Beta
Connect the Algolia Crawler API to your agent
Jentic One is a self-hosted execution layer for AI agents. It lets your agent call the Algolia Crawler API, or any other public or private API you need. You set the rules, the agent never sees your credentials, and every call is logged.
Two steps, two machines. Install the instance in a safe environment, then register your agent from wherever it runs.
1
Step 1: Jentic One Host machine
# On the machine that will host your Jentic One instance:
curl -fsSL https://raw.githubusercontent.com/jentic/jentic-one/main/tools/install.sh | sh2
Step 2: Agent machine
# On the machine where your agent runs (keep this separate from the instance):
curl -fsSL https://raw.githubusercontent.com/jentic/jentic-one/main/tools/install.sh | sh
jentic register # connects your agent to your Jentic One instanceJentic One is in public beta. The setup above keeps your agent separate from the instance, which is what you want before using real credentials: an agent running as the same OS user as Jentic One can read its stored keys directly. Just evaluating? A single local install is fine to start. See the secure deployment guide for the tiers.
Capabilities
What an agent can do with Algolia Crawler API.
Create a new Algolia crawler with POST /1/crawlers and update its configuration with PATCH /1/crawlers/{id}/config
Trigger a full reindex with POST /1/crawlers/{id}/reindex or pause and resume crawlers
Test crawl a single URL via POST /1/crawlers/{id}/test to validate extractors before a full run
Crawl a batch of URLs on demand with POST /1/crawlers/{id}/urls/crawl
List crawler runs and download per-run log files for diagnostics
Manage registered domains and inspect crawler stats across the application
Use Cases
Patterns agents use Algolia Crawler API for, with concrete tasks.
★ Documentation site crawler bootstrapping
A docs platform team creates a new Algolia crawler for a freshly launched documentation portal by calling POST /1/crawlers with the start URLs and extraction config, validates extraction with POST /1/crawlers/{id}/test on a sample URL, and then kicks off the first full crawl with POST /1/crawlers/{id}/reindex. The same workflow can be re-run when site structure changes substantially.
POST /1/crawlers with the start URLs, POST /1/crawlers/{id}/test against a representative URL, then POST /1/crawlers/{id}/reindex
On-demand recrawl after a publish event
A CMS publishes new articles and triggers an Algolia recrawl by calling POST /1/crawlers/{id}/urls/crawl with the affected URLs. This avoids waiting for the next scheduled reindex and keeps Algolia search results fresh without re-crawling the entire site.
POST /1/crawlers/{id}/urls/crawl with the list of newly published URLs and check the response task id with GET /1/crawlers/{id}/tasks/{taskID}
Crawler health monitoring
An ops dashboard polls GET /1/crawlers/{id}/stats/urls and GET /1/crawlers/{id}/crawl_runs to display recent run durations, success rates, and URL counts. When a run fails, the dashboard fetches the log file via GET /1/crawlers/{id}/{logId}/download and surfaces the relevant error lines for triage.
GET /1/crawlers/{id}/crawl_runs, render the latest runs, and pull GET /1/crawlers/{id}/{logId}/download for any failing run id
Agent-driven crawler configuration through Jentic
A Jentic agent receives an intent like 'add /blog to the Algolia crawler' and chains GET /1/crawlers/{id}/config, PATCH /1/crawlers/{id}/config with the updated URL patterns, and then POST /1/crawlers/{id}/reindex. The agent never sees the basic auth credentials — Jentic injects them at execution time.
Search Jentic for 'update algolia crawler config', execute PATCH /1/crawlers/{id}/config with the new patterns, then POST /1/crawlers/{id}/reindex
Key Endpoints
20 endpoints — jentic publishes the only available openapi specification for the algolia crawler api, keeping it validated and agent-ready.
METHOD
PATH
DESCRIPTION
POST
/1/crawlers
Create a crawler
POST
/1/crawlers/{id}/reindex
Start a fresh crawl
POST
/1/crawlers/{id}/test
Test crawl a single URL
POST
/1/crawlers/{id}/urls/crawl
Crawl specific URLs on demand
PATCH
/1/crawlers/{id}/config
Update crawler configuration
GET
/1/crawlers/{id}/crawl_runs
List crawler runs
GET
/1/crawlers/{id}/{logId}/download
Download a crawler run log
POST
/1/crawlers
Create a crawler
POST
/1/crawlers/{id}/reindex
Start a fresh crawl
POST
/1/crawlers/{id}/test
Test crawl a single URL
POST
/1/crawlers/{id}/urls/crawl
Crawl specific URLs on demand
PATCH
/1/crawlers/{id}/config
Update crawler configuration
GET
/1/crawlers/{id}/crawl_runs
List crawler runs
GET
/1/crawlers/{id}/{logId}/download
Download a crawler run log
Why Jentic?
Three things that make agents converge on Jentic-routed access.
Credential isolation
The Crawler user id and Crawler API key are stored encrypted in the Jentic credential vault and combined into the Authorization basic auth header at execution time. The agent never sees the raw credentials.
Intent-based discovery
Agents search Jentic by intent (e.g. 'reindex an algolia crawler', 'crawl specific URLs'), and Jentic returns the matching Crawler API operation with its input schema for direct execution.
Time to first call
Direct Algolia Crawler integration: 2-3 days to wire basic auth, configuration management, and run polling. Through Jentic: under an hour with search, load, and execute.
Related APIs
Alternatives and complements available in the Jentic catalogue.
FAQs
Specific to using Algolia Crawler API through Jentic.
Why is there no official OpenAPI spec for the Algolia Crawler API?
Algolia does not publish a downstream-stable OpenAPI specification for direct agent use. Jentic generates and maintains this spec from Algolia's source bundles so AI agents and developers can call the Crawler API via structured tooling. It is validated against the live API and kept up to date. Get started at https://app.jentic.com/sign-up.
What authentication does the Algolia Crawler API use?
The Crawler API uses HTTP basic auth with a Crawler user id and Crawler API key — distinct from the standard Algolia application id and api key. Jentic stores the credential pair encrypted and builds the Authorization header at execution time so the secret never enters the agent's prompt.
Can I trigger an on-demand recrawl of specific URLs?
Yes. POST /1/crawlers/{id}/urls/crawl with a list of URLs. The endpoint returns a task id you can poll with GET /1/crawlers/{id}/tasks/{taskID} until the recrawl completes.
What are the rate limits for the Algolia Crawler API?
The OpenAPI spec does not publish per-endpoint rate limits for the Crawler API. Algolia enforces them server-side; respect 4xx responses and back off when triggering rapid reindex or test calls in succession.
How do I update an Algolia crawler's configuration through Jentic?
Search Jentic for 'update algolia crawler config', load the schema for PATCH /1/crawlers/{id}/config, and execute it with the updated URL patterns or extractor settings. Then POST /1/crawlers/{id}/reindex to apply the change to live results.
Can I download the log for a specific crawler run?
Yes. GET /1/crawlers/{id}/{logId}/download returns the run log file. Pair this with GET /1/crawlers/{id}/crawl_runs to find the run id you need before downloading.
GET STARTED