Skip to main content

API Reference

info

Contact Marc Plouhinec to get an authentication token.

Table Of Content​

Scraping Tasks​

Run Scraping Tasks​

HTTP Method and URL

POST https://continuous-scraper.common.chartedapi.com/scraping-tasks/${scraper}/run?autoCancelAfterSec=1200&includeAllFields=false&waitForCompletion=true

Path Parameters

  • scraper: shopee or lazada.

Query Parameters

  • autoCancelAfterSec (default = 600): scraping execution timeout (only enabled if waitForCompletion=true).
  • includeAllFields (default = false): if false, only the most important fields are returned.
  • waitForCompletion (default = true): if true, wait for the scraping tasks to terminate. If false, return immediately (use this API with the uuids parameter to get the results).
warning

waitForCompletion=true requires maintaining an open HTTP connection to receive results after the scraping tasks complete. If this connection is closed prematurely, the ongoing scraping tasks will be cancelled.

Request Headers

  • Authorization: Bearer ${token}

Request Body Example

{
  "requests": [
    {
      "url": "https://shopee.co.id/adidas-Runfalcon-3.0-Men's-Running-Shoes---Core-Black-i.234490784.25462132870?sp_atk=67836f0a-c0c8-40d4-a0a7-bd1998943cc9&xptdk=67836f0a-c0c8-40d4-a0a7-bd1998943cc9"
    },
    {
      "url": "https://shopee.co.id/SEPATU-SNEAKERS-CASUAL-ADIDS-SPEZIAL-HANDBALL-BNIB-i.197850298.20595626884?sp_atk=ff613749-9cf1-4b40-ae8e-f76dd2279ba8&xptdk=ff613749-9cf1-4b40-ae8e-f76dd2279ba8"
    },
    {
      "url": "https://shopee.co.th/api/v4/item/get_list",
      "method": "POST",
      "payload": {
        "shop_item_ids": [
          {
            "item_id": 5289970960,
            "shop_id": 237980476
          },
          {
            "item_id": 23556386969,
            "shop_id": 644345618
          }
        ],
        "source": "microsite_individual_product"
      }
    }
  ]
}

Request body fields

  • requests: An object with the following fields:
    • url: The API or page URL of the resource to scrap.
    • referrer (optional): The referrer to pass as a request header.
    • method (optional): The request method (GET or POST).
    • payload (optional): The request body (if method is POST).
  • cacheMaxAgeInSec (default = 0): If it exists and is fresh enough, return the successful scraping task result with the same URL.

In addition, you can also add scraper-specific parameters, for example:

{
  "requests": [
    { 
      "url": "https://shopee.co.id/api/v4/search/search_items?keyword=tshirt"
    }
  ],
  "productSearch_enrichUrlQuery_pageSize": 60,
  "productSearch_crawlNextPages": true,
  "productSearch_crawlNextPages_maxPages": 3,
  "productSearch_crawlProductDetails": true
}

Response Body Example

[
  {
    "url": "https://shopee.co.id/SEPATU-SNEAKERS-CASUAL-ADIDS-SPEZIAL-HANDBALL-BNIB-i.197850298.20595626884?sp_atk=ff613749-9cf1-4b40-ae8e-f76dd2279ba8&xptdk=ff613749-9cf1-4b40-ae8e-f76dd2279ba8",
    "status": "SUCCESS",
    "responseBody": "{}"
  },
  {
    "url": "https://shopee.co.id/adidas-Runfalcon-3.0-Men's-Running-Shoes---Core-Black-i.234490784.25462132870?sp_atk=67836f0a-c0c8-40d4-a0a7-bd1998943cc9&xptdk=67836f0a-c0c8-40d4-a0a7-bd1998943cc9",
    "status": "ERROR",
    "responseBody": null,
    "errorMessage": "... explanation ..."
  }
]

CURL Command Example

curl --header "Content-Type: application/json" \
  --header "Authorization: Bearer $token" \
  --request POST \
  --data '{"requests":[{"url": "https://shopee.co.id/adidas-Runfalcon-3.0-Mens-Running-Shoes---Core-Black-i.234490784.25462132870?sp_atk=67836f0a-c0c8-40d4-a0a7-bd1998943cc9&xptdk=67836f0a-c0c8-40d4-a0a7-bd1998943cc9"}, {"url": "https://shopee.co.id/SEPATU-SNEAKERS-CASUAL-ADIDS-SPEZIAL-HANDBALL-BNIB-i.197850298.20595626884?sp_atk=ff613749-9cf1-4b40-ae8e-f76dd2279ba8&xptdk=ff613749-9cf1-4b40-ae8e-f76dd2279ba8"}]}' \
  https://continuous-scraper.common.chartedapi.com/requests/shopee/run?autoCancelAfterSec=1200&includeAllFields=false

Run A Single Scraping Task​

HTTP Method and URL

POST https://continuous-scraper.common.chartedapi.com/scraping-tasks/${scraper}/run-single?autoCancelAfterSec=1200&includeAllFields=false

Path Parameters

  • scraper: shopee or lazada.

Query Parameters

  • autoCancelAfterSec (default = 600): scraping execution timeout (only enabled if waitForCompletion=true).
  • includeAllFields (default = false): if false, only the most important fields are returned.
  • waitForCompletion (default = true): if true, wait for the scraping tasks to terminate. If false, return immediately (use this API with the uuids parameter to get the results).
warning

waitForCompletion=true requires maintaining an open HTTP connection to receive results after the scraping tasks complete. If this connection is closed prematurely, the ongoing scraping tasks will be cancelled.

Request Headers

  • Authorization: Bearer ${token}

Request Body Example

{
  "url": "https://shopee.co.id/adidas-Runfalcon-3.0-Men's-Running-Shoes---Core-Black-i.234490784.25462132870?sp_atk=67836f0a-c0c8-40d4-a0a7-bd1998943cc9&xptdk=67836f0a-c0c8-40d4-a0a7-bd1998943cc9"
}

Request body fields

  • url: The API or page URL of the resource to scrap.
  • referrer (optional): The referrer to pass as a request header.
  • method (optional): The request method (GET or POST).
  • payload (optional): The request body (if method is POST).
  • cacheMaxAgeInSec (default = 0): If it exists and is fresh enough, return the successful scraping task result with the same URL.

In addition, you can also add scraper-specific parameters, for example:

{
  "url": "https://shopee.co.id/api/v4/search/search_items?keyword=tshirt",
  "productSearch_enrichUrlQuery_pageSize": 60,
  "productSearch_crawlNextPages": true,
  "productSearch_crawlNextPages_maxPages": 3,
  "productSearch_crawlProductDetails": true
}

Response Body Example

{
  "url": "https://shopee.co.id/SEPATU-SNEAKERS-CASUAL-ADIDS-SPEZIAL-HANDBALL-BNIB-i.197850298.20595626884?sp_atk=ff613749-9cf1-4b40-ae8e-f76dd2279ba8&xptdk=ff613749-9cf1-4b40-ae8e-f76dd2279ba8",
  "status": "SUCCESS",
  "responseBody": "{}"
}

CURL Command Example

curl --header "Content-Type: application/json" \
  --header "Authorization: Bearer $token" \
  --request POST \
  --data '{"url": "https://shopee.co.id/adidas-Runfalcon-3.0-Mens-Running-Shoes---Core-Black-i.234490784.25462132870?sp_atk=67836f0a-c0c8-40d4-a0a7-bd1998943cc9&xptdk=67836f0a-c0c8-40d4-a0a7-bd1998943cc9"}' \
  https://continuous-scraper.common.chartedapi.com/requests/shopee/run-single?autoCancelAfterSec=1200&includeAllFields=false

Read Scraping Tasks​

HTTP Method and URL

GET https://continuous-scraper.common.chartedapi.com/scraping-tasks/${scraper}?limit=20&offset=0&uuids=398a3b4d-241c-4297-9e81-5c4c6ecce340,785ebeaf-ae2c-4453-908b-c7dac008fbf4

Path Parameters

  • scraper: shopee or lazada.

Query Parameters

  • limit (optional, default = 20): Maximum number of scraping tasks to return.
  • offset (default = 0): Number of scraping tasks to skip.
  • uuids (optional): Comma-separated list of scraping task UUIDs. Filter by the given scraping task UUIDs. Example: uuids=398a3b4d-241c-4297-9e81-5c4c6ecce340,785ebeaf-ae2c-4453-908b-c7dac008fbf4
  • urls (optional): Comma-separated list of scraping task URLs. Filter by the given scraping task URLs. Example: urls=https://shopee.co.id/api/v4/search/search_items?keyword=nike,https://shopee.co.id/api/v4/search/search_items?keyword=huawei
  • minPendingAt (optional): Only return scraping tasks that have been submitted after this date (ISO 8601 format).
  • maxPendingAt (optional): Only return scraping tasks that have been submitted before this date (ISO 8601 format).
  • includeAllFields (default = false): if false, only the most important fields are returned.
  • includeFields (optional): Comma-separated list of scraping task fields to return. Not that it overrides includeAllFields. Example: includeFields=uuid,url,status

Request Headers

  • Authorization: Bearer ${token}

Response Body Example

[
  {
    "uuid": "398a3b4d-241c-4297-9e81-5c4c6ecce340",
    "url": "https://shopee.co.id/SEPATU-SNEAKERS-CASUAL-ADIDS-SPEZIAL-HANDBALL-BNIB-i.197850298.20595626884?sp_atk=ff613749-9cf1-4b40-ae8e-f76dd2279ba8&xptdk=ff613749-9cf1-4b40-ae8e-f76dd2279ba8",
    "status": "SUCCESS",
    "responseBody": "{}"
  },
  {
    "uuid": "785ebeaf-ae2c-4453-908b-c7dac008fbf4",
    "url": "https://shopee.co.id/adidas-Runfalcon-3.0-Men's-Running-Shoes---Core-Black-i.234490784.25462132870?sp_atk=67836f0a-c0c8-40d4-a0a7-bd1998943cc9&xptdk=67836f0a-c0c8-40d4-a0a7-bd1998943cc9",
    "status": "ERROR",
    "responseBody": null,
    "errorMessage": "... explanation ..."
  }
]

CURL Command Example

curl https://continuous-scraper.common.chartedapi.com/scraping-tasks/shopee?limit=20&offset=0 \
  --header "Authorization: Bearer $token"

Count Scraping Tasks​

HTTP Method and URL

GET https://continuous-scraper.common.chartedapi.com/scraping-tasks/${scraper}/count?statuses=PENDING,WAITING_FOR_PAGE,RUNNING,BLOCKED,ERROR

Path Parameters

  • scraper: shopee or lazada.

Query Parameters

  • statuses (optional, default = all statuses): Comma-separated list of scraping task statuses. If omitted, the API returns counts for all available statuses. Acceptable values: PENDING, WAITING_FOR_PAGE, RUNNING, BLOCKED, BLOCKED_TOO_MANY_TIMES, ERROR, ERROR_TOO_MANY_TIMES, REPLACED, CANCELLED, SUCCESS.

    Example 1: statuses=PENDING,WAITING_FOR_PAGE,RUNNING,BLOCKED,ERROR to count all pending and running tasks.

    Example 2: statuses=BLOCKED_TOO_MANY_TIMES,ERROR_TOO_MANY_TIMES,REPLACED,CANCELLED,SUCCESS to count all completed or non-recoverable tasks.

Request Headers

  • Authorization: Bearer ${token}

Response Body Example

[
  {
    "country": "ID",
    "status": "BLOCKED_TOO_MANY_TIMES",
    "count": 4846
  },
  {
    "country": "ID",
    "status": "SUCCESS",
    "count": 452746545
  },
  {
    "country": "MY",
    "status": "SUCCESS",
    "count": 3054565
  }
]

CURL Command Example

curl https://continuous-scraper.common.chartedapi.com/scraping-tasks/shopee/count?statuses=PENDING,WAITING_FOR_PAGE,RUNNING,BLOCKED,ERROR \
  --header "Authorization: Bearer $token"

Cancel Scraping Tasks​

HTTP Method and URL

POST https://continuous-scraper.common.chartedapi.com/scraping-tasks/${scraper}/cancel

Path Parameters

  • scraper: shopee or lazada.

Request Headers

  • Authorization: Bearer ${token}

Request Body Example

{
  "uuids": [
    "2199a83d-3ebe-442c-aa82-afd3a9903967",
    "1ce16c7b-26f1-4851-a281-e09f79ce2de9",
    "34d3449d-b5b9-46a5-a4e0-d524d2ab3222"
  ],
  "countries": ["ID", "MY", "PH", "SG", "TH", "VN"]
}

Request body fields

  • uuids: Array of UUIDs of the scraping tasks to cancel.
  • countries: Array of countries to cancel.
info

Sub-scraping tasks are cancelled as well.

Response Body Example

{
  "uuids": [
    "2199a83d-3ebe-442c-aa82-afd3a9903967", 
    "1ce16c7b-26f1-4851-a281-e09f79ce2de9", 
    "34d3449d-b5b9-46a5-a4e0-d524d2ab3222", 
    "37744b6d-35fb-47c0-9c73-e9b7feb9f089", 
    "29d6b918-745b-4029-bdba-5f6489c43146"
  ],
  "countries": ["ID"]
}

Response body fields

  • uuids: Array of cancelled scraping tasks (and sub-scraping tasks) UUIDs.
  • countries: Array of the countries related to cancelled scraping tasks.

CURL Command Example

curl --header "Content-Type: application/json" \
  --header "Authorization: Bearer $token" \
  --request POST \
  --data '{"uuids":["2199a83d-3ebe-442c-aa82-afd3a9903967", "1ce16c7b-26f1-4851-a281-e09f79ce2de9"}' \
  https://continuous-scraper.common.chartedapi.com/requests/shopee/cancel

Proxy Configuration​

By default, scrapers use proxies managed by Charted Sea. However, organizations can configure their own proxy settings if they wish to:

  • Expand their pool of available IP addresses for higher scraping speed or better success rate.
  • Leverage their own proxy provider deals to reduce costs or access specific networks.

If an organization chooses to use its own proxies, Charted Sea will not charge for the usage of managed proxies, allowing customers to fully control their proxy costs.

This section explains how to configure custom proxies, including the use of dynamic placeholders to generate proxy URLs with sticky sessions.

Update Proxy Configuration​

Replace the entire proxy configuration with a new set of proxy URLs.

HTTP Method and URL

PUT https://continuous-scraper.common.chartedapi.com/proxy-configs

Request Headers

  • Authorization: Bearer ${token}

Request Body Example

[
  {
    "scraper": "SHOPEE",
    "proxySettings": [
      {
        "type": "RESIDENTIAL",
        "urls": [
          "http://uuu__cr.${country(case=LOWER)}:ppp@gw.dataimpulse.com:${rand(range=[10000,19999])}",
          "http://uuu-residential-country_${country(case=UPPER)}-r_10m-s_${rand(len=10,use=[lowerCaseChar,upperCaseChar,digit])}:ppp@gate.nstproxy.io:24125"
        ]
      },
      {
        "type": "RESIDENTIAL",
        "country": "TW",
        "urls": [
          "http://uuu__cr.${country(case=LOWER)}:ppp@gw.dataimpulse.com:${rand(range=[10000,19999])}",
          "http://uuu:ppp@gw.dataimpulse.com:${rand(range=[10000,19999])}",
          "http://uuu-residential-country_${country(case=UPPER)}-r_10m-s_${rand(len=10,use=[lowerCaseChar,upperCaseChar,digit])}:ppp@gate.nstproxy.io:24125",
          "http://uuu-residential-country_ANY-r_10m-s_${rand(len=10,use=[lowerCaseChar,upperCaseChar,digit])}:ppp@gate.nstproxy.io:24125"
        ]
      }
    ]
  },
  {
    "scraper": "LAZADA",
    "proxySettings": [
      {
        "type": "DATACENTER",
        "urls": [
          "http://vvv__cr.${country(case=LOWER)}:qqq@gw.dataimpulse.com:${rand(range=[10000,19999])}"
        ]
      },
      {
        "type": "RESIDENTIAL",
        "urls": [
          "http://uuu__cr.${country(case=LOWER)}:ppp@gw.dataimpulse.com:${rand(range=[10000,19999])}"
        ]
      }
    ]
  }
]

Request body fields

  • scraper: Either "SHOPEE" or "LAZADA".
  • proxySettings: An array of objects with the following fields:
    • type: Either "DATACENTER" or "RESIDENTIAL".
    • urls: Array of proxy URLs. Note that you can use placeholders to dynamically generate proxy URLs, this is particularly useful for sticky sessions.
    • country (optional): If specified, this proxy setting is dedicated to this country. Otherwise, the proxy setting is applied to all countries with no dedicated setting.
warning

The configured proxies MUST be sticky. Rotating proxies assign a new IP address per request, which leads to detection and blocking by the target website.

Using Placeholders in Proxy URLs

Placeholders in proxy URLs are enclosed in ${}. They allow dynamic generation of parts of the URL based on specified functions and arguments.

Supported functions:

  • ${rand(...)}: Generates a random string based on provided arguments:
    • len: Length of the string to generate. Mandatory unless range is provided.
    • use: Array specifying types of characters to include. Allowed values:
      • lowerCaseChar: abcdefghijklmnopqrstuvwxyz
      • upperCaseChar: ABCDEFGHIJKLMNOPQRSTUVWXYZ
      • digit: 0123456789
      • symbol: [],$&+:;=?@#|"'><.^*()%! -
    • subset: A string of allowed characters. This is an alternative to use. The characters $, " and \ must be escaped with \ (i.e. \$, \" and \\).
    • range: Specifies a range [min, max] to generate a random number within.
  • ${country(...)} with the arguments:
    • case: Either UPPER or LOWER.

Example proxy URLs with placeholders:

{
    "urls": [
        // Generate 9999 DataImpulse URLs
        "http://uuu__cr.${country(case=LOWER)}:ppp@gw.dataimpulse.com:${rand(range=[10000,19999])}",
      
        // Generate many LunaProxy URLs
        "http://user-uuu-region-${country(case=LOWER)}-sessid-${country(case=LOWER)}${rand(len=16,use=[lowerCaseChar,digit])}-sesstime-30:ppp@as.e4f4xh3i.lunaproxy.net:12233",
      
        // Generate many IPRoyal URLs
        "http://uuu:ppp_country-${country(case=LOWER)}_session-${rand(len=8,use=[lowerCaseChar,upperCaseChar,digit])}_lifetime-30m@geo.iproyal.com:12321",
      
        // Generate many BrightData URLs
        "http://brd-customer-hl_uuu-zone-zzz-country-${country(case=LOWER)}-session-${rand(range=[1,70000000])}:ppp@brd.superproxy.io:22225",

        // Generate many Oxylabs URLs
        "http://customer-uuu-cc-${country(case=UPPER)}-sessid-${rand(len=10,use=[lowerCaseChar,digit])}-sesstime-30:ppp@pr.oxylabs.io:7777",

        // Generate many Massive URLs (note how the characters @, ?, & and = are URI encoded)
        "http://john.doe%40example.com%3Fcountry%3D${country(case=UPPER)}%26session%3D${rand(len=10,use=[lowerCaseChar,upperCaseChar,digit])}:ppp@network.joinmassive.com:65534",

        // Generate many Nstproxy URLs
        "http://uuu-residential-country_${country(case=UPPER)}-r_10m-s_${rand(len=10,use=[lowerCaseChar,upperCaseChar,digit])}:ppp@gate.nstproxy.io:24125",

        // Generate many Mango Proxy URLs
        "http://uuu-zone-custom-region-${country(case=LOWER)}-session-${rand(len=9,use=[lowerCaseChar,digit])}-sessTime-30:ppp@p1.mangoproxy.com:2333",

        // Generate many Netnut URLs
        "http://uuu-res-${country(case=UPPER)}-sid-${rand(len=9,use=[digit])}:ppp@gw.ntnt.io:5959",

        // Generate many Zenrows URLs
        "http://uuu:ppp_country-${country(case=LOWER)}_ttl-30m_session-${rand(len=12,use=[lowerCaseChar,upperCaseChar,digit])}@superproxy.zenrows.com:1337",

        // Generate many Soax Proxy URLs
        "http://package-123456-country-${country(case=LOWER)}-sessionid-${rand(len=10,use=[lowerCaseChar,digit])}-sessionlength-10:ppp@proxy.soax.com:5000"
    ]
}

Response Body Example

Same as the Read Proxy Configuration response.

CURL Command Example

curl --header "Content-Type: application/json" \
  --header "Authorization: Bearer $token" \
  --request PUT \
  --data '[{"scraper":"SHOPEE","proxySettings":[{"type":"RESIDENTIAL","urls":["http://uuu__cr.${country(case=LOWER)}:ppp@gw.dataimpulse.com:${rand(range=[10000,19999])}"]}]},{"scraper":"LAZADA","proxySettings":[{"type":"RESIDENTIAL","urls":["http://uuu__cr.${country(case=LOWER)}:ppp@gw.dataimpulse.com:${rand(range=[10000,19999])}"]}]}]' \
  https://continuous-scraper.common.chartedapi.com/proxy-configs

Read Proxy Configuration​

HTTP Method and URL

GET https://continuous-scraper.common.chartedapi.com/proxy-configs

Request Headers

  • Authorization: Bearer ${token}

Request Body Example

[
  {
    "organizationSlug": "organization-abc",
    "scraper": "SHOPEE",
    "proxySettings": [
      {
        "type": "RESIDENTIAL",
        "urls": [
          "http://uuu__cr.${country(case=LOWER)}:ppp@gw.dataimpulse.com:${rand(range=[10000,19999])}",
          "http://uuu-residential-country_${country(case=UPPER)}-r_10m-s_${rand(len=10,use=[lowerCaseChar,upperCaseChar,digit])}:ppp@gate.nstproxy.io:24125"
        ]
      },
      {
        "country": "TW",
        "type": "RESIDENTIAL",
        "urls": [
          "http://uuu__cr.${country(case=LOWER)}:ppp@gw.dataimpulse.com:${rand(range=[10000,19999])}",
          "http://uuu:ppp@gw.dataimpulse.com:${rand(range=[10000,19999])}",
          "http://uuu-residential-country_${country(case=UPPER)}-r_10m-s_${rand(len=10,use=[lowerCaseChar,upperCaseChar,digit])}:ppp@gate.nstproxy.io:24125",
          "http://uuu-residential-country_ANY-r_10m-s_${rand(len=10,use=[lowerCaseChar,upperCaseChar,digit])}:ppp@gate.nstproxy.io:24125"
        ]
      }
    ]
  },
  {
    "organizationSlug": "organization-abc",
    "scraper": "LAZADA",
    "proxySettings": [
      {
        "type": "DATACENTER",
        "urls": [
          "http://vvv__cr.${country(case=LOWER)}:qqq@gw.dataimpulse.com:${rand(range=[10000,19999])}"
        ]
      },
      {
        "type": "RESIDENTIAL",
        "urls": [
          "http://uuu__cr.${country(case=LOWER)}:ppp@gw.dataimpulse.com:${rand(range=[10000,19999])}"
        ]
      }
    ]
  }
]

Response body fields are the same as the Update Proxy Configuration request body fields.

CURL Command Example

curl https://continuous-scraper.common.chartedapi.com/proxy-configs \
  --header "Authorization: Bearer $token"