From bb59f370c3ef7bb931b4ac821918b85a10b0bb64 Mon Sep 17 00:00:00 2001 From: shixi-li Date: Sat, 1 Aug 2026 17:01:38 +0800 Subject: [PATCH 1/4] docs: add guide on building custom extensions Crawlee has four extension points, and each already documents its own contract in the guide that owns it: crawlers in the HTTP crawlers guide, HTTP clients and storage clients in theirs, and browser plugins in the Playwright crawler guide. What was missing is the map: a page that names the extension points, says what each contract covers, and points at the guide that goes deep. That page is also what a third-party integration can link to, which is the case #1936 was opened for: the integration hosts its own guide and references a stable statement of the interface it implements. Refs #1936 --- docs/guides/extending_crawlee.mdx | 92 +++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 docs/guides/extending_crawlee.mdx diff --git a/docs/guides/extending_crawlee.mdx b/docs/guides/extending_crawlee.mdx new file mode 100644 index 0000000000..49317f3e82 --- /dev/null +++ b/docs/guides/extending_crawlee.mdx @@ -0,0 +1,92 @@ +--- +id: extending-crawlee +title: Extending Crawlee +description: Learn which parts of Crawlee are designed to be extended, what contract each extension point defines, and where to find the detailed guide for each one. +--- + +import ApiLink from '@site/src/components/ApiLink'; + +Crawlee is built around a small number of abstract base classes that you can subclass to plug in your own behavior. This guide is the map: it lists every extension point, states the contract each one defines, and links to the guide that covers it in depth. + +If you maintain a third-party integration, such as an alternative browser backend or a storage adapter, you can build it against these contracts and host the integration guide in your own project. This page gives your users a stable reference for the interface your integration implements. + +## Extension points + +Crawlee currently has four extension points. + +```mermaid +--- +config: + class: + hideEmptyMembersBox: true +--- + +classDiagram + +class BasicCrawler { + <> +} + +class HttpClient { + <> +} + +class StorageClient { + <> +} + +class BrowserPlugin { + <> +} + +BasicCrawler --> HttpClient : uses +BasicCrawler --> StorageClient : uses +BasicCrawler --> BrowserPlugin : uses +``` + +### Crawlers + +Subclass a crawler when you need a parsing strategy or a request-handler context that the built-in crawlers do not provide. + +For HTTP-based crawling, `AbstractHttpCrawler` is the base class. A custom crawler supplies a parser that turns an HTTP response into your parsed type, a context type that exposes that parsed data to request handlers, and the crawler class that ties the two together. Everything else, including retries, concurrency, session management, and storage, is inherited from `BasicCrawler`. + +See [HTTP crawlers guide](./http-crawlers) for a worked example built on `selectolax`, and [Architecture overview](./architecture-overview) for how crawlers relate to the other components. + +### HTTP clients + +Subclass an HTTP client when you want crawlers to talk to servers through a different HTTP library, or through a proxy or transport layer that the bundled clients do not cover. + +`HttpClient` is the base class. Implementations must be async-compatible and must manage their own connection lifecycle and cleanup, because a single client instance is shared across concurrent requests. + +See [HTTP clients guide](./http-clients) for the full interface and the built-in implementations. + +### Storage clients + +Subclass a storage client when you want Crawlee's storages to be backed by a system that is not covered by the built-in memory, file system, SQL, and Redis clients. + +`StorageClient` is the base class. It is a factory: it opens the per-storage clients for datasets, key-value stores, and request queues, and those clients implement the actual create, read, update, and delete operations. + +See [Storage clients guide](./storage-clients) for the interface, a custom client example, and how clients are registered and resolved. + +### Browser plugins + +Subclass a browser plugin when an integration launches browsers through an API other than the standard Playwright one. Configuration options on `PlaywrightBrowserPlugin` cover the cases where the standard launch API is enough, so reach for a subclass only when the launch path itself differs. + +A plugin's `new_browser()` launches the browser and returns a `PlaywrightBrowserController`; `BrowserPool` initializes the plugin, forwards browser context options when creating pages, and manages the controller's lifecycle. + +See [Playwright crawler guide](./playwright-crawler) for the contract and the responsibilities a subclass has to preserve, and the [Camoufox example](../examples/playwright-crawler-with-camoufox) for a complete integration. + +## Choosing an extension point + +Match the layer to what actually differs in your integration. + +- The response is fetched the usual way but parsed differently: extend a **crawler**. +- The response is fetched over a different HTTP library or transport: extend an **HTTP client**. +- Requests and results should live somewhere other than the built-in backends: extend a **storage client**. +- Browsers are launched through a different API: extend a **browser plugin**. + +Prefer configuration over a subclass wherever the built-in class already exposes the knob you need. Subclassing ties your integration to a contract that only changes with Crawlee's major versions, while configuration keeps you on the maintained path. + +## Conclusion + +Crawlee's extension points are crawlers, HTTP clients, storage clients, and browser plugins. Each is an abstract base class with a documented contract and a guide that covers it in depth. If you are building an integration, start from the contract that matches the layer you are replacing, and keep the rest of the pipeline on the built-in path. From aa273b7b292d65da8dee2fc53446e8c95f898a6c Mon Sep 17 00:00:00 2001 From: shixi-li Date: Wed, 5 Aug 2026 19:25:06 +0800 Subject: [PATCH 2/4] docs: route the browser edge in the extension-points diagram through BrowserPool --- docs/guides/extending_crawlee.mdx | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/guides/extending_crawlee.mdx b/docs/guides/extending_crawlee.mdx index 49317f3e82..15542f56bc 100644 --- a/docs/guides/extending_crawlee.mdx +++ b/docs/guides/extending_crawlee.mdx @@ -27,6 +27,8 @@ class BasicCrawler { <> } +class PlaywrightCrawler + class HttpClient { <> } @@ -35,13 +37,17 @@ class StorageClient { <> } +class BrowserPool + class BrowserPlugin { <> } BasicCrawler --> HttpClient : uses BasicCrawler --> StorageClient : uses -BasicCrawler --> BrowserPlugin : uses +BasicCrawler --|> PlaywrightCrawler +PlaywrightCrawler --> BrowserPool : uses +BrowserPool --> BrowserPlugin : manages ``` ### Crawlers From 9a235147370081b4b9957d152d77324e83f29cbc Mon Sep 17 00:00:00 2001 From: shixi-li Date: Thu, 6 Aug 2026 11:45:35 +0800 Subject: [PATCH 3/4] docs: rework the extension-points guide around component contracts --- docs/guides/extending_crawlee.mdx | 79 +++++++++++++++++++++---------- 1 file changed, 55 insertions(+), 24 deletions(-) diff --git a/docs/guides/extending_crawlee.mdx b/docs/guides/extending_crawlee.mdx index 15542f56bc..844e1d640e 100644 --- a/docs/guides/extending_crawlee.mdx +++ b/docs/guides/extending_crawlee.mdx @@ -1,18 +1,18 @@ --- id: extending-crawlee title: Extending Crawlee -description: Learn which parts of Crawlee are designed to be extended, what contract each extension point defines, and where to find the detailed guide for each one. +description: The extension points Crawlee exposes, the contract each one defines, and how to choose between them. --- import ApiLink from '@site/src/components/ApiLink'; -Crawlee is built around a small number of abstract base classes that you can subclass to plug in your own behavior. This guide is the map: it lists every extension point, states the contract each one defines, and links to the guide that covers it in depth. +Crawlee covers the common cases out of the box, but sooner or later you'll hit something it doesn't do: a parser for a format no built-in crawler understands, an HTTP backend your company mandates, a database you want your data in, or a browser that isn't launched the standard Playwright way. Rather than forking, you can plug your own implementation into the layer that differs and keep everything else. -If you maintain a third-party integration, such as an alternative browser backend or a storage adapter, you can build it against these contracts and host the integration guide in your own project. This page gives your users a stable reference for the interface your integration implements. +This guide is the map. It covers the extension points, the contract each one defines, and links to the guide that goes deeper on it. If you maintain a third-party integration, you can build it against these contracts and host your own guide for it. ## Extension points -Crawlee currently has four extension points. +The four extension points below are the main ones, and they're where most integrations plug in. They aren't the only abstract classes you can subclass - `RequestLoader`, `FingerprintGenerator`, and `RenderingTypePredictor` are extensible too. ```mermaid --- @@ -27,6 +27,10 @@ class BasicCrawler { <> } +class AbstractHttpCrawler { + <> +} + class PlaywrightCrawler class HttpClient { @@ -35,64 +39,91 @@ class HttpClient { class StorageClient { <> + create_dataset_client() + create_kvs_client() + create_rq_client() +} + +class DatasetClient { + <> +} + +class KeyValueStoreClient { + <> +} + +class RequestQueueClient { + <> } class BrowserPool class BrowserPlugin { <> + new_browser() } -BasicCrawler --> HttpClient : uses -BasicCrawler --> StorageClient : uses +class BrowserController { + <> +} + +BasicCrawler --|> AbstractHttpCrawler BasicCrawler --|> PlaywrightCrawler +AbstractHttpCrawler --> HttpClient : uses +BasicCrawler --> StorageClient : uses +StorageClient --> DatasetClient : opens +StorageClient --> KeyValueStoreClient : opens +StorageClient --> RequestQueueClient : opens PlaywrightCrawler --> BrowserPool : uses BrowserPool --> BrowserPlugin : manages +BrowserPlugin --> BrowserController : returns ``` ### Crawlers -Subclass a crawler when you need a parsing strategy or a request-handler context that the built-in crawlers do not provide. +A crawler drives the whole run: it takes requests from the queue, fetches each one, builds the context object your handler receives, and manages retries, concurrency, sessions, and storage along the way. `BasicCrawler` implements all of that and stays agnostic about how a page is fetched or parsed, which is what makes it the base every other crawler builds on. -For HTTP-based crawling, `AbstractHttpCrawler` is the base class. A custom crawler supplies a parser that turns an HTTP response into your parsed type, a context type that exposes that parsed data to request handlers, and the crawler class that ties the two together. Everything else, including retries, concurrency, session management, and storage, is inherited from `BasicCrawler`. +For HTTP-based crawling, `AbstractHttpCrawler` adds the fetch-and-parse layer on top. Extending it means supplying a parser that implements `AbstractHttpParser`: `parse` turns an `HttpResponse` into your parsed type, `parse_text` does the same for a string, `select` and `is_matching_selector` apply selectors to it, and `find_links` extracts the URLs used for link enqueuing. You then pair that parser with a context type that exposes the parsed data to handlers, and a crawler class that ties the two together. -See [HTTP crawlers guide](./http-crawlers) for a worked example built on `selectolax`, and [Architecture overview](./architecture-overview) for how crawlers relate to the other components. +See the [HTTP crawlers guide](./http-crawlers) for a worked example built on `selectolax`, and the [Architecture overview](./architecture-overview) for how crawlers relate to the other components. ### HTTP clients -Subclass an HTTP client when you want crawlers to talk to servers through a different HTTP library, or through a proxy or transport layer that the bundled clients do not cover. +An HTTP client is what actually performs the network calls for HTTP-based crawlers. Swapping it changes the transport - the TLS stack, connection pooling, proxy handling, and browser impersonation - without touching how pages are parsed or how the crawl is orchestrated. -`HttpClient` is the base class. Implementations must be async-compatible and must manage their own connection lifecycle and cleanup, because a single client instance is shared across concurrent requests. +The contract is `HttpClient`. `crawl` performs a request inside the crawler's pipeline and returns the result the crawler consumes, `send_request` covers standalone calls made from a handler, `stream` yields a response you read incrementally, and `cleanup` releases whatever the client holds open. Crawlee ships `ImpitHttpClient`, `HttpxHttpClient`, and `CurlImpersonateHttpClient`. -See [HTTP clients guide](./http-clients) for the full interface and the built-in implementations. +See the [HTTP clients guide](./http-clients) for the full contract and the trade-offs between the built-in clients. ### Storage clients -Subclass a storage client when you want Crawlee's storages to be backed by a system that is not covered by the built-in memory, file system, SQL, and Redis clients. +A storage client is the backend behind Crawlee's three storages. `Dataset`, `KeyValueStore`, and `RequestQueue` are the API you write against, and the storage client decides where that data actually lives. That separation is what lets you move a crawler from the local file system to a database or a cloud service without changing crawl code. -`StorageClient` is the base class. It is a factory: it opens the per-storage clients for datasets, key-value stores, and request queues, and those clients implement the actual create, read, update, and delete operations. +`StorageClient` itself is only three factory methods - `create_dataset_client`, `create_kvs_client`, and `create_rq_client`. The real work sits in what they return: `DatasetClient` handles appending and reading items, `KeyValueStoreClient` handles record get, set, delete, and iteration, and `RequestQueueClient` handles adding, fetching, and marking requests as handled. A custom backend implements all four. -See [Storage clients guide](./storage-clients) for the interface, a custom client example, and how clients are registered and resolved. +See the [Storage clients guide](./storage-clients) for the built-in implementations and a custom client example. ### Browser plugins -Subclass a browser plugin when an integration launches browsers through an API other than the standard Playwright one. Configuration options on `PlaywrightBrowserPlugin` cover the cases where the standard launch API is enough, so reach for a subclass only when the launch path itself differs. +A browser plugin is what launches browsers for `PlaywrightCrawler`. The crawler never launches one itself: it goes through `BrowserPool`, which initializes the plugins it's given, forwards browser context options when creating pages, and manages each browser's lifecycle. -A plugin's `new_browser()` launches the browser and returns a `PlaywrightBrowserController`; `BrowserPool` initializes the plugin, forwards browser context options when creating pages, and manages the controller's lifecycle. +The contract is `BrowserPlugin`. Its `new_browser` launches a browser and returns a `BrowserController`, which is what the pool then drives to open pages and tear things down. Reach for a subclass when the launch path itself differs from the standard Playwright one, since `PlaywrightBrowserPlugin`'s configuration options already cover the cases where it doesn't. -See [Playwright crawler guide](./playwright-crawler) for the contract and the responsibilities a subclass has to preserve, and the [Camoufox example](../examples/playwright-crawler-with-camoufox) for a complete integration. +See the [Playwright crawler guide](./playwright-crawler) for the responsibilities a subclass has to preserve, and the [Camoufox example](../examples/playwright-crawler-with-camoufox) for a complete integration. ## Choosing an extension point Match the layer to what actually differs in your integration. -- The response is fetched the usual way but parsed differently: extend a **crawler**. -- The response is fetched over a different HTTP library or transport: extend an **HTTP client**. -- Requests and results should live somewhere other than the built-in backends: extend a **storage client**. -- Browsers are launched through a different API: extend a **browser plugin**. +- The response format is one no built-in crawler parses - subclass `AbstractHttpCrawler` with your own parser. +- The transport differs, but parsing doesn't - implement `HttpClient` and pass it to any HTTP crawler. +- Data needs to live somewhere Crawlee doesn't support yet - implement `StorageClient` and its three per-storage clients. +- Browsers need to be launched through a different API - implement `BrowserPlugin` and hand it to `BrowserPool`. -Prefer configuration over a subclass wherever the built-in class already exposes the knob you need. Subclassing ties your integration to a contract that only changes with Crawlee's major versions, while configuration keeps you on the maintained path. +When more than one fits, pick the narrowest. A custom HTTP client works with every HTTP crawler, so it's easier to maintain than a crawler subclass that hard-codes the same transport. ## Conclusion -Crawlee's extension points are crawlers, HTTP clients, storage clients, and browser plugins. Each is an abstract base class with a documented contract and a guide that covers it in depth. If you are building an integration, start from the contract that matches the layer you are replacing, and keep the rest of the pipeline on the built-in path. +Each extension point is a small abstract class with a documented contract, and everything above it keeps working once you implement one. If you're building a third-party integration, these contracts are also the stable surface to write your own documentation against. + +If anything here is unclear or you hit a case the contracts don't cover, open an issue on [GitHub](https://github.com/apify/crawlee-python) or join our [Discord](https://discord.com/invite/jyEM2PRvMU). From 1c16751c190fc6ef9341e4867630e7488995e45d Mon Sep 17 00:00:00 2001 From: shixi-li Date: Sun, 9 Aug 2026 17:37:10 +0800 Subject: [PATCH 4/4] docs: address extension guide review feedback --- docs/guides/extending_crawlee.mdx | 80 ++++++++++++++++++------------- 1 file changed, 48 insertions(+), 32 deletions(-) diff --git a/docs/guides/extending_crawlee.mdx b/docs/guides/extending_crawlee.mdx index 844e1d640e..429aaccfa8 100644 --- a/docs/guides/extending_crawlee.mdx +++ b/docs/guides/extending_crawlee.mdx @@ -6,13 +6,13 @@ description: The extension points Crawlee exposes, the contract each one defines import ApiLink from '@site/src/components/ApiLink'; -Crawlee covers the common cases out of the box, but sooner or later you'll hit something it doesn't do: a parser for a format no built-in crawler understands, an HTTP backend your company mandates, a database you want your data in, or a browser that isn't launched the standard Playwright way. Rather than forking, you can plug your own implementation into the layer that differs and keep everything else. +Crawlee covers the common cases out of the box, but sooner or later you'll hit something it doesn't do: a parser for a format no built-in crawler understands, an HTTP backend your company mandates, a database you want your data in, or a browser that isn't launched through the standard Playwright path. Rather than forking, you can plug your own implementation into the layer that differs and keep everything else. This guide is the map. It covers the extension points, the contract each one defines, and links to the guide that goes deeper on it. If you maintain a third-party integration, you can build it against these contracts and host your own guide for it. ## Extension points -The four extension points below are the main ones, and they're where most integrations plug in. They aren't the only abstract classes you can subclass - `RequestLoader`, `FingerprintGenerator`, and `RenderingTypePredictor` are extensible too. +The four component families below contain the main extension points, and they're where most integrations plug in. This isn't a complete list of Crawlee's extensible classes. Other examples include `RequestLoader`, `FingerprintGenerator`, and `RenderingTypePredictor`. The diagram marks the classes in these four families that you can extend or implement as an `extension point`. ```mermaid --- @@ -24,72 +24,85 @@ config: classDiagram class BasicCrawler { - <> + <> } class AbstractHttpCrawler { - <> + <> } -class PlaywrightCrawler +class AbstractHttpParser { + <> +} + +class PlaywrightCrawler { + <> +} + +class StagehandCrawler class HttpClient { - <> + <> } class StorageClient { - <> - create_dataset_client() - create_kvs_client() - create_rq_client() + <> } class DatasetClient { - <> + <> } class KeyValueStoreClient { - <> + <> } class RequestQueueClient { - <> + <> } class BrowserPool class BrowserPlugin { - <> - new_browser() + <> } class BrowserController { - <> + <> +} + +class PlaywrightBrowserPlugin { + <> } BasicCrawler --|> AbstractHttpCrawler BasicCrawler --|> PlaywrightCrawler -AbstractHttpCrawler --> HttpClient : uses +AbstractHttpCrawler --> AbstractHttpParser : parses with +PlaywrightCrawler --|> StagehandCrawler +BasicCrawler --> HttpClient : uses BasicCrawler --> StorageClient : uses StorageClient --> DatasetClient : opens StorageClient --> KeyValueStoreClient : opens StorageClient --> RequestQueueClient : opens PlaywrightCrawler --> BrowserPool : uses BrowserPool --> BrowserPlugin : manages -BrowserPlugin --> BrowserController : returns +BrowserPlugin --|> PlaywrightBrowserPlugin +BrowserPlugin --> BrowserController : new_browser() returns ``` ### Crawlers -A crawler drives the whole run: it takes requests from the queue, fetches each one, builds the context object your handler receives, and manages retries, concurrency, sessions, and storage along the way. `BasicCrawler` implements all of that and stays agnostic about how a page is fetched or parsed, which is what makes it the base every other crawler builds on. +A crawler drives the whole run. It takes requests from the queue, fetches each one, builds the context object your handler receives, and manages retries, concurrency, sessions, and storage along the way. `BasicCrawler` implements that orchestration and stays agnostic about how a page is fetched or parsed, which is what makes it the base every other crawler builds on. -For HTTP-based crawling, `AbstractHttpCrawler` adds the fetch-and-parse layer on top. Extending it means supplying a parser that implements `AbstractHttpParser`: `parse` turns an `HttpResponse` into your parsed type, `parse_text` does the same for a string, `select` and `is_matching_selector` apply selectors to it, and `find_links` extracts the URLs used for link enqueuing. You then pair that parser with a context type that exposes the parsed data to handlers, and a crawler class that ties the two together. +For HTTP-based crawling, `AbstractHttpCrawler` adds the fetch-and-parse layer. Its contract pairs a parser, a crawling context type, and a crawler class. The parser implements `AbstractHttpParser`. Its `parse` method turns an `HttpResponse` into your parsed type, `parse_text` does the same for a string, `select` and `is_matching_selector` apply selectors, and `find_links` extracts URLs for link enqueuing. The context exposes the parsed data to handlers, and the crawler ties the parser and context together. -See the [HTTP crawlers guide](./http-crawlers) for a worked example built on `selectolax`, and the [Architecture overview](./architecture-overview) for how crawlers relate to the other components. +Browser crawlers use the same orchestration with a browser-backed context. Extend `PlaywrightCrawler` when an integration needs crawler-level browser behavior or a different handler context. `StagehandCrawler` is an example. It extends `PlaywrightCrawler` with a Stagehand-specific context and browser behavior. If only browser launch or lifecycle differs, a browser plugin is the narrower extension point. + +See the [HTTP crawlers guide](./http-crawlers) for a worked example built on `selectolax`. The [Architecture overview](./architecture-overview) explains how HTTP and browser crawlers relate to the other components. ### HTTP clients -An HTTP client is what actually performs the network calls for HTTP-based crawlers. Swapping it changes the transport - the TLS stack, connection pooling, proxy handling, and browser impersonation - without touching how pages are parsed or how the crawl is orchestrated. +An HTTP client performs network calls for crawlers. Swapping it changes the transport, including the TLS stack, connection pooling, proxy handling, and browser impersonation. It doesn't change how pages are parsed or how the crawl is orchestrated. The contract is `HttpClient`. `crawl` performs a request inside the crawler's pipeline and returns the result the crawler consumes, `send_request` covers standalone calls made from a handler, `stream` yields a response you read incrementally, and `cleanup` releases whatever the client holds open. Crawlee ships `ImpitHttpClient`, `HttpxHttpClient`, and `CurlImpersonateHttpClient`. @@ -99,31 +112,34 @@ See the [HTTP clients guide](./http-clients) for the full contract and the trade A storage client is the backend behind Crawlee's three storages. `Dataset`, `KeyValueStore`, and `RequestQueue` are the API you write against, and the storage client decides where that data actually lives. That separation is what lets you move a crawler from the local file system to a database or a cloud service without changing crawl code. -`StorageClient` itself is only three factory methods - `create_dataset_client`, `create_kvs_client`, and `create_rq_client`. The real work sits in what they return: `DatasetClient` handles appending and reading items, `KeyValueStoreClient` handles record get, set, delete, and iteration, and `RequestQueueClient` handles adding, fetching, and marking requests as handled. A custom backend implements all four. +The `StorageClient` contract defines three factory methods: `create_dataset_client`, `create_kvs_client`, and `create_rq_client`. The returned clients define the rest of the contract. `DatasetClient` handles appending and reading items, `KeyValueStoreClient` handles record access and iteration, and `RequestQueueClient` handles adding, fetching, and marking requests as handled. A custom backend implements all four classes. See the [Storage clients guide](./storage-clients) for the built-in implementations and a custom client example. ### Browser plugins -A browser plugin is what launches browsers for `PlaywrightCrawler`. The crawler never launches one itself: it goes through `BrowserPool`, which initializes the plugins it's given, forwards browser context options when creating pages, and manages each browser's lifecycle. +A browser plugin launches browsers for `PlaywrightCrawler`. The crawler delegates that work to `BrowserPool`. The pool initializes its plugins, forwards browser context options when creating pages, and manages each browser's lifecycle. + +The abstract contract is `BrowserPlugin`. Its `new_browser` method launches a browser and returns a `BrowserController`. The pool uses that controller to open pages and tear down the browser. Implement this base contract directly when the launch and lifecycle are too specific for Crawlee's Playwright integration. -The contract is `BrowserPlugin`. Its `new_browser` launches a browser and returns a `BrowserController`, which is what the pool then drives to open pages and tear things down. Reach for a subclass when the launch path itself differs from the standard Playwright one, since `PlaywrightBrowserPlugin`'s configuration options already cover the cases where it doesn't. +Most integrations should start with `PlaywrightBrowserPlugin`. Configure it when its launch and context options cover the required browser. Extend it when you need a custom Playwright-compatible launch path while preserving its standard lifecycle and context handling. See the [Playwright crawler guide](./playwright-crawler) for the responsibilities a subclass has to preserve, and the [Camoufox example](../examples/playwright-crawler-with-camoufox) for a complete integration. ## Choosing an extension point -Match the layer to what actually differs in your integration. +Start with configuration before writing a subclass. You can parse a response with a third-party library inside an `HttpCrawler` handler, pass an existing `http_client` to any crawler, or configure `PlaywrightBrowserPlugin`. Use an extension contract only when the maintained options don't cover the required behavior. -- The response format is one no built-in crawler parses - subclass `AbstractHttpCrawler` with your own parser. -- The transport differs, but parsing doesn't - implement `HttpClient` and pass it to any HTTP crawler. -- Data needs to live somewhere Crawlee doesn't support yet - implement `StorageClient` and its three per-storage clients. -- Browsers need to be launched through a different API - implement `BrowserPlugin` and hand it to `BrowserPool`. +- If reusable HTTP parsing and the handler context both need to change, extend `AbstractHttpCrawler` and implement `AbstractHttpParser`. +- If browser-level orchestration or the handler context needs to change, extend `PlaywrightCrawler`. +- If the network transport needs to change while crawler behavior stays the same, implement `HttpClient` and pass it to the crawler. +- If the storage backend needs to change while the storage API stays the same, implement `StorageClient` and its three per-storage clients. +- If browser launch needs to change while the Playwright lifecycle stays the same, extend `PlaywrightBrowserPlugin`. Implement `BrowserPlugin` directly only when its launch and lifecycle contract needs a different implementation. When more than one fits, pick the narrowest. A custom HTTP client works with every HTTP crawler, so it's easier to maintain than a crawler subclass that hard-codes the same transport. ## Conclusion -Each extension point is a small abstract class with a documented contract, and everything above it keeps working once you implement one. If you're building a third-party integration, these contracts are also the stable surface to write your own documentation against. +Each extension point has a documented class contract, and everything above it keeps working once you implement that contract. These public abstract class contracts only change with a major release. That versioning policy makes them the stable surface for a third-party integration and its documentation. -If anything here is unclear or you hit a case the contracts don't cover, open an issue on [GitHub](https://github.com/apify/crawlee-python) or join our [Discord](https://discord.com/invite/jyEM2PRvMU). +If you have questions or need assistance, feel free to reach out on our [GitHub](https://github.com/apify/crawlee-python) or join our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping!