Skip to main content

Shopee Scraper

What is Shopee Scraper and how does it work?​

Shopee Scraper allows you to directly invoke the unofficial API of Shopee, the leading ecommerce marketplace in South East Asia and also present in Central and South America.

This unofficial API is used by Shopee for its frontend, that's why it provides all the data displayed on the website. In fact, it actually provides more information (e.g. sales and stock levels per product variant) and is more flexible (e.g. configurable page size).

The Shopee Scraper takes care of all the "low-level" details such as setting the right HTTP headers, to allow you to easily and efficiently requests data.

Quick Start​

Run the following scraping tasks:

{
  "requests": [
    { "url": "https://shopee.sg/Pants-cat.11012963.11027757" },
    { "url": "https://shopee.sg/search?keyword=tshirt" },
    { "url": "https://shopee.sg/Japanese-Samurai-T-Shirt-New-Samurai-T-Shirt-Distro-T-Shirts-i.414243778.23600133176?sp_atk=43c5e72a-9613-4ee1-9cd7-e29ebcdbd4b9&xptdk=43c5e72a-9613-4ee1-9cd7-e29ebcdbd4b9" },
    { "url": "https://shopee.sg/lorealparissg" }
  ]
}

This performs four distinct requests:

RequestURL
Search products in the category "Men's Wear > Pants"https://shopee.sg/Pants-cat.11012963.11027757
Search for products with the keyword "tshirt"https://shopee.sg/search?keyword=tshirt
Retrieve detailed product informationhttps://shopee.sg/Japanese-Samurai-T-Shirt-New-Samurai-T-Shirt-Distro-T-Shirts-i.414243778.23600133176?sp_atk=43c5e72a-9613-4ee1-9cd7-e29ebcdbd4b9&xptdk=43c5e72a-9613-4ee1-9cd7-e29ebcdbd4b9
Retrieve detailed information about a shophttps://shopee.sg/lorealparissg

After few seconds the Shopee Scraper returns the following results:

While the example above demonstrates the ease of using page URLs directly from your browser, advanced users might find greater value in tapping into the Shopee unofficial API for enhanced control and access to extended features not available through the website interface.

For example, leveraging the API allows for:

  • Retrieving up to 50 product ratings in a single request, bypassing the standard limit of 6 results per page.
  • Accessing a broader range of up to 440 popular keyword suggestions, far exceeding the usual cap of 11.

For a comprehensive understanding of how to utilize the Shopee Scraper effectively, including more examples and detailed explanations, please consult the How to Scrape Shopee? section below.

What data can you extract?​

Shopee Scraper seamlessly interacts with Shopee's unofficial API to extract a wide range of data, including:

  • Product searches by keyword and category.
  • Detailed product information (description, price, stock levels, sales data, and variants).
  • Buyer ratings, complete with comments and media.
  • Hierarchical category trees and detailed category information.
  • Lists of official shops, filtered by category if needed.
  • Searches for shops by keyword.
  • Comprehensive shop details (description, opening date, product counts, and performance metrics).
  • Shop products.
  • Keyword suggestions and autocompletion hints.
tip

Consider scraping both Lazada and Shopee simultaneously to strengthen your data acquisition strategy. This not only ensures a more reliable pipeline—given the low probability of both platforms introducing breaking changes simultaneously—but also leverages complementary data insights, as keyword trends from Lazada can enhance product searches on Shopee, which has a larger market share.

tip

Please visit the API reference section below for more details about each API.

Advanced users can execute other APIs as well, you can find them by using the DevTools "Network" tab. Please contact me if you encounter issues or need additional features.

Why scrape Shopee data?​

Discover what you can achieve with the Shopee Scraper through these example use cases:

  • Market and Competitive Analysis: Track category performance and identify emerging trends by analyzing sales data, product reviews, and competitor strategies. This can reveal insights into market demands and areas for strategic advantage.
  • Opportunity Discovery: Use aggregated product search results to uncover popular yet underserved keywords and niches in the market. This can highlight potential areas for new product development or market entry.
  • Product Review Analysis: Employ AI, such as ChatGPT, to deeply analyze product reviews. Summarize buyer feedback to gain insights into consumer satisfaction, preferences, and areas for product improvement.
  • Competitor Watch: Regularly scrape competitor data to monitor for critical changes like products going out of stock, pricing adjustments, or new product launches. This enables real-time alerts and swift response strategies.

How to scrape Shopee?​

Getting started​

Get started with Shopee Scraper by following these simple steps:

  1. Prepare Requests: Formulate the API request URLs you wish to scrape. Refer to the API reference section for guidance. Example URLs include:
    • Search for popular products by keyword: https://shopee.sg/api/v4/search/search_items?keyword=tshirt
    • Retrieve top categories: https://shopee.sg/api/v4/pages/get_category_tree
    • List products by category: https://shopee.sg/api/v4/search/search_items?match_id=11027757
    • List all official shops: https://shopee.sg/api/v4/official_shop/get_shops_by_category
    • List products by shop: https://shopee.sg/api/v4/shop/rcmd_items?shop_id=178877065
    • Access product detail: https://shopee.sg/api/v4/pdp/get_pc?shop_id=414243778&item_id=23600133176
    • Access product ratings: https://shopee.sg/api/v2/item/get_ratings?shopid=414243778&itemid=23600133176
  2. Execution: Submit your scraping tasks and wait for the results. Follow the execution via the Grafana dashboard.

Example input:

{
  // Search products with the "tshirt" keyword
  "requests": [
    { "url": "https://shopee.sg/api/v4/search/search_items?keyword=tshirt" }
  ]
}

Example JSON output from a scraping session:

{
  "url": "https://shopee.sg/api/v4/search/search_items?keyword=tshirt",
  "enrichedUrl": "https://shopee.sg/api/v4/search/search_items?by=relevancy&extra_params=%7B%22global_search_session_id%22%3A%22gs-c44e90e2-6f01-4ed4-9fe3-fd38fe17f6e9%22%2C%22search_session_id%22%3A%22ss-59a709b9-72fe-4fdf-8a46-23edd0ff3255%22%7D&keyword=tshirt&limit=5&newest=0&order=desc&page_type=search&scenario=PAGE_GLOBAL_SEARCH&version=2&view_session_id=53f46a2f-80b1-45b4-afbb-17cca078a722",
  "referrer": "https://shopee.sg/search?keyword=tshirt",
  "knownApi": "PRODUCT_SEARCH",
  "responseStatus": 200,
  "responseBody": {
    // ...
    "total_count": 3000,
    "items": [
      {
        "item_basic": {
          "itemid": 22541793907,
          "shopid": 391955424,
          "name": "Casual T-shirts Short-sleeved Tshirt Korean Round Neck Youth T-shirt Plain Color Tshirt Fashion Popular T-shirt",
          "image": "sg-11134201-7qvez-lgj50bviqvph8b",
          "currency": "SGD",
          "stock": 242,
          "ctime": 1683532055,
          "sold": 826,
          "historical_sold": 3044,
          "catid": 100011,
          "price": 850000,
          "price_before_discount": 1990000,
          "discount": "57%",
          "item_rating": {
            "rating_star": 4.702664796633941,
            "rating_count": [
              713,
              11,
              9,
              33,
              75,
              585
            ],
            "rcount_with_context": 254,
            "rcount_with_image": 164
          },
          // ...
        },
        "itemid": 22541793907,
        "shopid": 391955424,
        // ...
      },
      // ...
    ],
    // ...
  }
}

This output includes fields like the original URL (url), the enriched URL sent to Shopee (enrichedUrl), and the HTTP response (responseStatus and responseBody), among others.

tip

Enable 'Enrich URLs' and 'Generate referrers' for enhanced scraping. Check the input parameters for more details.

Input parameters​

The Shopee Scraper offers a range of input parameters, allowing you to tailor your scraping process according to your needs. These parameters can be broadly categorized into common parameters applicable to all APIs and specific parameters unique to certain APIs (detailed in the API reference section).

Key common parameters include:

  1. requests: This is where you specify the list of API URLs you intend to scrape. Additionally, you have the option to include the following optional attributes:
    • userData.referrer: a Referer header with each request. If generateReferrer is enabled, the scraper automatically generates a referrer for each request.
    • userData.language: Language set in the "language" cookie and the "x-shopee-language" header, only for this request.
    • method (default: GET): Request HTTP method.
    • payload: Request body in JSON format.
    An example configuration is as follows: 
    {
      "requests": [
        { 
          "url": "https://shopee.co.th/api/v4/pdp/get_pc?shop_id=146475693&item_id=25922366667",
          "userData": {
            "referrer": "https://shopee.co.th/LANEIGE-Water-Bank-Blue-Hyaluronic-Emulsion-120mL-for-Combination-to-Oily-skin--i.146475693.25922366667",
            "language": "en"
          }
        },
        {
          "url": "https://shopee.sg/api/v4/shop/rcmd_items",
          "method": "POST",
          "payload": {
            "bundle": "shop_page_category_tab_main",
            "shop_id": 178877065,
            "limit": 30,
            "offset": 0,
            "upstream": "",
            "sort_type": 1,
            "user_behaviour":{"user_behaviour_list":[]}
          }
        }
      ]
    }
  2. maxRequestsPerCrawl (Optional): Set the maximum number of requests the crawler will execute. The process stops once this limit is reached.
  3. enrichUrlQuery: Enabled by default, this option automatically adds missing parameters to URLs, such as result limits and session IDs.
  4. generateReferrer: Also enabled by default, this generates the referrer HTTP header for each API request to enhance authenticity.
  5. language (Optional): Language set in the "language" cookie and the "x-shopee-language" header, for all requests.

For any queries or assistance, feel free to reach out.

Output fields​

The data returned from the Shopee Scraper includes several important fields. Below is a description of each:

  • url: This is the original URL you submitted for scraping.

  • enrichedUrl: Represents the actual URL sent to Shopee. If you enable Enrich URLs, this URL is automatically enhanced with default values. For more details, refer to the input parameters section.

  • referrer: The Referer HTTP header sent to Shopee. It's generated automatically if Generate referrers is enabled. You have the option to set this manually as well (see input parameters for more).

  • knownApi: Enumerated value corresponding to the requested API. This helps you identify which API endpoint was used:

    APIKnownApi
    /api/v4/search/search_itemsPRODUCT_SEARCH
    /api/v4/pdp/get_pcPRODUCT_DETAIL
    /api/v2/item/get_ratingsPRODUCT_RATINGS
    /api/v4/pages/get_category_treeFE_CATEGORY_TREE
    /api/v4/search/get_fe_category_detailFE_CATEGORY_DETAIL
    /api/v4/official_shop/get_shops_by_categoryOFFICIAL_SHOPS_BY_CATEGORY
    /api/v4/search/search_userSHOP_SEARCH
    /api/v4/shop/get_shop_baseSHOP_DETAIL
    /api/v4/shop/rcmd_itemsSHOP_PRODUCTS
    /api/v4/search/search_suggestionKEYWORD_SUGGESTIONS
    /api/v4/search/search_hintKEYWORD_AUTOCOMPLETION
    -- Other API ---- not set --

  • responseStatus: Indicates the HTTP response status received from Shopee.

  • responseBody: Contains the HTTP response body from Shopee in JSON format. The specific content varies depending on the requested API.

API reference​

Product Search by Keyword or Category​

  • URL Path:

    • /api/v4/search/search_items

  • URL Parameters:

    • keyword: Search keyword, e.g., "tshirt".
    • match_id: Category ID from /api/v4/pages/get_category_tree.
    • categoryids: Comma-seperated list of category IDs.
    • fe_categoryids: Comma-seperated list of FE category IDs.
    • brandids: Comma-seperated list of FE categorybrand IDs.
      info

      keyword, match_id, categoryids, fe_categoryids and brandids are mutually exclusive.

    • by (default = "relevancy"): Sort by "relevancy", "ctime", "sales", or "price".
    • order (default = "desc"): Sort order, "asc" or "desc".
    • limit (default = set by scraper): Number of results per page.
    • newest (default = 0): Offset for results, typically a multiple of limit.
    • page_type, scenario, version: Additional parameters with default settings.
    • extra_params, view_session_id: Automatically generated, unless overridden.

  • Scraper Input Parameters

    • productSearch_enrichUrlQuery_pageSize: Number of results per request (default: 60).
    • productSearch_crawlNextPages: Set to true to crawl subsequent pages.
    • productSearch_crawlNextPages_maxPages: Limit on number of pages to crawl (default: 50).
    • productSearch_crawlNextPages_maxResults: Maximum products to crawl (default: 3000).
    • productSearch_crawlProductDetails: Set to true to crawl details of each product.

  • Example Input:

{
  "requests": [
    // Search products with the "tshirt" keyword
    { "url": "https://shopee.sg/api/v4/search/search_items?keyword=tshirt" },
    
    // Search products in the category ID 11027421
    { "url": "https://shopee.sg/api/v4/search/search_items?match_id=11027421&scenario=PAGE_CATEGORY" },
    
    // Search products from the shop ID 290739469 (scenario PAGE_SHOP_SEARCH)
    { "url": "https://shopee.sg/api/v4/search/search_items?entry_point=ShopByPDP&match_id=290739469&page_type=shop&scenario=PAGE_SHOP_SEARCH" },
    
    // Search products from the shop ID 290739469 (scenario PAGE_SHOP)
    { "url": "https://shopee.sg/api/v4/search/search_items?match_id=290739469&page_type=shop&scenario=PAGE_SHOP" },
    
    // Search products from the brand ID 4508220
    { "url": "https://shopee.sg/api/v4/search/search_items?brandids=4508220&scenario=PAGE_OTHERS" }
  ],

  // Set the "limit" URL parameter to 60
  "productSearch_enrichUrlQuery_pageSize": 60,
  // Automatically crawl the next pages
  "productSearch_crawlNextPages": true,
  // Crawl until the 3rd page
  "productSearch_crawlNextPages_maxPages": 3,
  // Alternatively, indicate the maximum number of products to crawl
  "productSearch_crawlNextPages_maxResults": 180,
  // Automatically crawl the detail of each listed product
  "productSearch_crawlProductDetails": true
}

  • Example Response:

  • Notable Response Fields:

    • items: Array of product summaries.
    • items/itemid and items/shopid: Essential for scraping product details and ratings.
    • items/item_basic/name: Product name.
    • items/item_basic/image: Image key. Convert to URL with https://down-${country}.img.susercontent.com/file/${imageKey}, for example the URL to download the image "sg-11134201-7qvez-lgj50bviqvph8b" is "https://down-sg.img.susercontent.com/file/sg-11134201-7qvez-lgj50bviqvph8b".
    • items/item_basic/ctime: Creation timestamp.
    • items/item_basic/sold: Monthly sales.
    • items/item_basic/historical_sold: Total sales.
    • items/item_basic/price: Divide by 100,000 to get actual price.
info

Unfortunately sold and historical_sold are not available anymore in non-authenticated mode. However, the Product Recommendation API still provides these fields.

Product Details​

  • URL Path:

    • /api/v4/pdp/get_pc

  • URL Parameters:

    • shop_id: The unique identifier of the shop.
    • item_id: The unique identifier of the product.

  • Scraper Input Parameters

    • productDetail_crawlProductRatings: Define the types of product ratings to crawl. Options include "ALL", "ONE_STAR", "TWO_STARS" to "FIVE_STARS", "WITH_COMMENTS", "WITH_MEDIA", "LOCAL_REVIEWS". Combine multiple values for a broader range, e.g., ["ONE_STAR", "TWO_STARS"].

  • Example Input:

{
  // Get the details of a product
  "requests": [
    { "url": "https://shopee.sg/api/v4/pdp/get_pc?shop_id=414243778&item_id=23600133176" }
  ],
  
  // Automatically crawl ratings of any stars
  "productDetail_crawlProductRatings": ["ALL"],

  // Set the rating "limit" URL parameter to 50
  "productRatings_enrichUrlQuery_pageSize": 50,
  // Automatically crawl the next rating pages
  "productRatings_crawlNextPages": true,
  // Crawl until the 3rd rating page
  "productRatings_crawlNextPages_maxPages": 3,
  // Alternatively, indicate the maximum number of ratings to crawl
  "productRatings_crawlNextPages_maxResults": 150,
}

  • Example Response:

  • Notable Response Fields:

    • data/item/item_id and data/item/shop_id: Essential identifiers for the product and shop.
    • data/item/title: Product name.
    • data/item/description: Product description.
    • data/item/image: Image key. Convert to URL using the provided format: https://down-${country}.img.susercontent.com/file/${imageKey}.
    • data/item/ctime: Product creation timestamp.
    • data/item/fe_categories: Category information including parent categories.
    • data/item/models: Detailed SKU information.
    • data/product_review: Aggregated rating statistics.
    • data/product_shipping: Shipping details.
    • data/shop_detailed: Comprehensive shop details including name, username (slug), and image.

Product Ratings​

  • URL Path:

    • /api/v2/item/get_ratings

  • URL Parameters:

    • shopid & itemid: Unique identifiers for the shop and product.
    • limit (default = set by scraper): Number of results per page.
    • offset (default = 0): Offset for results, typically a multiple of limit.
    • type (default = 0, max = 5): Filter by number of stars (e.g. 2 = 2-stars ratings only). 0 = all ratings.
    • filter (default = 0): Filter by content: 1 = with comments, 3 = with media, 9 = local ratings, 0 = all ratings.
    • exclude_filter, filter_size, flag, fold_filter, relevant_reviews, request_source, tag_filter, variation_filters: Additional parameters with default settings.

  • Scraper Input Parameters

    • productRatings_enrichUrlQuery_pageSize: Number of results per request (default: 6).
    • productRatings_crawlNextPages: Set to true to crawl subsequent pages.
    • productRatings_crawlNextPages_maxPages: Limit on number of pages to crawl (default: 500).
    • productRatings_crawlNextPages_maxResults: Maximum products to crawl (default: 3000).
    • productRatings_autofixError10002: Automatically corrects error 10002 by splitting requests (default: true).

  • Example Input:

{
  // Get the ratings of a product
  "requests": [
    { "url": "https://shopee.sg/api/v2/item/get_ratings?shopid=414243778&itemid=23600133176" }
  ],

  // Set the rating "limit" URL parameter to 50
  "productRatings_enrichUrlQuery_pageSize": 50,
  // Automatically crawl the next rating pages
  "productRatings_crawlNextPages": true,
  // Crawl until the 3rd rating page
  "productRatings_crawlNextPages_maxPages": 3,
  // Alternatively, indicate the maximum number of ratings to crawl
  "productRatings_crawlNextPages_maxResults": 150,
  // Automatically fix the error 10002
  "productRatings_autofixError10002": true,
}

  • Example Response:

  • Notable Response Fields:

    • data/ratings/cmtid: Unique identifier for each rating.
    • data/ratings/ctime: Timestamp of the rating.
    • data/ratings/rating_star: Star rating from 1 to 5.
    • data/ratings/comment: Buyer's comment.
    • data/ratings/author_username: Buyer's username (sometimes anonymized).
    • data/ratings/author_portrait: Buyer's profile image.
    • data/ratings/images & videos: Media associated with the rating.
    • data/ratings/product_items: SKU details of the rated product.
Data Privacy Note

Always be mindful of buyers' personal information and comply with local data protection laws. Avoid storing personal data unless necessary.

Product Recommendation​

  • URL Path:

    • POST /api/v4/recommend/product_detail_page

  • Request Body Parameters:

    • shopid (optional): Shop identifier.
    • itemid (optional): Product identifier.
    • catid (optional): Category identifier.
    • limit (default = 48): Number of results per page.
    • offset (default = 0): Offset for results, typically a multiple of limit.

  • Example Input:

{
  "requests": [
    {
      "url": "https://shopee.sg/api/v4/recommend/product_detail_page",
      "method": "POST",
      "payload": {
        "offset": 0,
        "limit": 48,
        "shopid": 119013379,
        "itemid": 20395639234,
        "catid": 100634
      }
    }
  ]
}

  • Example Response:

  • Notable Response Fields:

    • data/sections/units: Array of product summaries.
    • data/sections/units/itemid and shopid: Essential for scraping product details and ratings.
    • data/sections/units/name: Product name.
    • data/sections/units/image: Image key. Convert to URL with https://down-${country}.img.susercontent.com/file/${imageKey}, for example the URL to download the image "sg-11134201-7qvez-lgj50bviqvph8b" is "https://down-sg.img.susercontent.com/file/sg-11134201-7qvez-lgj50bviqvph8b".
    • data/sections/units/ctime: Creation timestamp.
    • data/sections/units/sold: Monthly sales.
    • data/sections/units/historical_sold: Total sales.
    • data/sections/units/price: Divide by 100,000 to get actual price.
info

Unlike with other APIs, the sold and historical_sold values are still available in non-authenticated mode.

Product Recommendation (bis)​

  • URL Path:

    • POST /api/v4/recommend/recommend_post

  • Request Body Parameters:

    • shopid (optional): Shop identifier.
    • itemid (optional): Product identifier.
    • catid (optional): Category identifier.
    • limit (default = 25): Number of results per page.
    • offset (optional): Number of skipped results.
    • section, item_card, bundle: Additional parameters with default settings.
note

This API is called in the "From The Same Shop" section at the bottom of a product page. Shopee usually invokes it with limit=25 and no offset, or with limit=48 and offset=0 (or 48, 96, ...)

  • Example Input:
{
  "requests": [
    {
      "url": "https://shopee.sg/api/v4/recommend/recommend_post",
      "method": "POST",
      "payload": {
        "limit": 25,
        "section": "from_same_shop",
        "shopid": 18254273,
        "itemid": 809240155,
        "catid": 100001,
        "item_card": 3,
        "bundle": "product_detail_page_ftss"
      }
    }
  ]
}

  • Example Response:

  • Notable Response Fields:

    • data/sections[0]/data/item: Array of product summaries.
    • data/sections[0]/data/item/itemid and shopid: Essential for scraping product details and ratings.
    • data/sections[0]/data/item/name: Product name.
    • data/sections[0]/data/item/image: Image key. Convert to URL with https://down-${country}.img.susercontent.com/file/${imageKey}, for example the URL to download the image "sg-11134201-7qvez-lgj50bviqvph8b" is "https://down-sg.img.susercontent.com/file/sg-11134201-7qvez-lgj50bviqvph8b".
    • data/sections[0]/data/item/ctime: Creation timestamp.
    • data/sections[0]/data/item/sold: Monthly sales.
    • data/sections[0]/data/item/historical_sold: Total sales.
    • data/sections[0]/data/item/price: Divide by 100,000 to get actual price.
info

Unlike with other APIs, the sold and historical_sold values are still available in non-authenticated mode.

Shopee Category Tree​

  • URL Path:

    • /api/v4/pages/get_category_tree

  • Example Input:

{
  // Get the category tree
  "requests": [
    { "url": "https://shopee.sg/api/v4/pages/get_category_tree" }
  ]
}

  • Example Response:

  • Notable Response Fields:

    • data/category_list/catid: Unique identifier for each category.
    • data/category_list/name: Category name.
    • data/category_list/children: Sub-categories.
info

This API only provides the two top category levels.

Category Details​

  • URL Path:

    • /api/v4/search/get_fe_category_detail

  • URL Parameters:

    • catids: Unique identifiers for the category.

  • Example Input:

{
  // Retrieve details for a specific category
  "requests": [
    { "url": "https://shopee.sg/api/v4/search/get_fe_category_detail?catids=11027710" }
  ]
}

  • Example Response:

  • Notable Response Fields:

    • data/categories/catid: The unique identifier for the category.
    • data/categories/parent_cat_id: Identifier for the parent category, helping to understand category hierarchy.
    • data/categories/display_name: The category name as displayed in different languages across the platform.

Official Shops​

  • URL Path:

    • /api/v4/official_shop/get_shops_by_category

  • URL Parameters:

    • category_id (default = -1): Specify a category ID to retrieve official shops within that category. Use -1 to list all official shops.
    • need_zhuyin: An additional parameter, typically set to default.
info

Only shops associated with registered brands are considered "official" on Shopee.

  • Example Input:
{
  // Retrieve a list of all official shops
  "requests": [
    { "url": "https://shopee.sg/api/v4/official_shop/get_shops_by_category" }
  ]
}

  • Example Response:

  • Notable Response Fields:

    • data/brands/brand_ids/shopid: The unique identifier for each official shop.
    • data/brands/brand_ids/brand_name: The name of the brand associated with the shop.
    • data/brands/brand_ids/username: The shop's username or slug, useful for direct queries or referencing.

Shop Search By Keyword​

  • URL Path:

    • /api/v4/search/search_user

  • URL Parameters:

    • keyword (minimum: 3 characters): Enter a search term, such as "adi", to find relevant shops.
    • limit (default = set by scraper): Number of results per page.
    • offset (default = 0): Offset for results, typically a multiple of limit.
    • page, with_search_cover: Additional parameters with default settings.
info

This API covers all shops on Shopee, both official and unofficial.

tip

For a comprehensive shop list, consider searching with various trigrams.

  • Scraper Input Parameters

    • shopSearch_enrichUrlQuery_pageSize: Number of results per request (default: 6).
    • shopSearch_crawlNextPages: Set to true to crawl subsequent pages.
    • shopSearch_crawlNextPages_maxPages: Limit on number of pages to crawl (default: 500).
    • shopSearch_crawlNextPages_maxResults: Maximum shops to crawl (default: 30000).
    • shopSearch_crawlShopDetails: Set to true to crawl details of each shop.

  • Example Input:

{
  // Search shop with the "adi" keyword
  "requests": [
    { "url": "https://shopee.sg/api/v4/search/search_user?keyword=adi" }
  ],

  // Set the "limit" URL parameter to 50
  "shopSearch_enrichUrlQuery_pageSize": 50,
  // Automatically crawl the next pages
  "shopSearch_crawlNextPages": true,
  // Crawl until the 3rd page
  "shopSearch_crawlNextPages_maxPages": 3,
  // Alternatively, indicate the maximum number of shops to crawl
  "shopSearch_crawlNextPages_maxResults": 150,
  // Automatically crawl the detail of each listed shop
  "shopSearch_crawlShopDetails": true
}

  • Example Response:

  • Notable Response Fields:

    • data/users/shopid: Unique identifier for the shop.
    • data/users/shopname: Name of the shop.
    • data/users/username: The shop's username or slug, useful for direct queries or referencing.
    • data/users/portrait: Shop image key. Convert to URL with https://down-${country}.img.susercontent.com/file/${imageKey}, for example the URL to download the image "6d4ff02a5978ee10a7b4527c25396b0b" is "https://down-sg.img.susercontent.com/file/6d4ff02a5978ee10a7b4527c25396b0b".

Shop Detail​

  • URL Path:

    • /api/v4/shop/get_shop_base

  • URL Parameters:

    • username: Specify the shop's username (not the shop ID).
    • shopid: Specify the shop's ID.
      info

      Use either username or shopid, but not both at the same time.

    • entry_point, need_cancel_rate, request_source, version: Additional parameters with default settings.

  • Scraper Input Parameters

    • shopDetail_crawlShopProducts: Set to true to enable automatic crawling of the shop's products.

  • Example Input:

{
  // Retrieve detailed information about a specific shop
  "requests": [
    { "url": "https://shopee.sg/api/v4/shop/get_shop_base?username=xiaomiofficialstore.sg" }
  ],
  
  // Automatically crawl shop products
  "shopSearch_crawlShopDetails": true,

  // Set the product "limit" URL parameter to 30
  "shopProducts_enrichUrlQuery_pageSize": 30,
  // Automatically crawl the next product pages
  "shopProducts_crawlNextPages": true,
  // Crawl until the 3rd product page
  "shopProducts_crawlNextPages_maxPages": 3,
  // Alternatively, indicate the maximum number of products to crawl
  "shopProducts_crawlNextPages_maxResults": 90,
}

  • Example Response:

  • Notable Response Fields:

    • data/shopid: The unique identifier for the shop.
    • data/name: Official name of the shop.
    • data/description: Brief description of the shop.
    • data/cover: Image key for the shop's profile picture. Convert to URL using the provided format: https://down-${country}.img.susercontent.com/file/${imageKey}.
    • data/account/username: The shop's unique username or slug.
    • data/ctime: Timestamp marking the shop's creation.
    • data/shop_rating: Aggregated rating details of the shop.
    • data/seller_metrics: Various metrics providing insights into the shop's performance.
    • data/item_count: Total number of products offered by the shop.

Shop Products​

  • URL Path:

    • api/v4/shop/rcmd_items

  • URL Parameters:

    • shop_id: Unique identifier for the shop.
    • limit (default = set by scraper): Number of results per page.
    • offset (default = 0): Offset for results, typically a multiple of limit.
    • sort_type (default = 1): Sorting option for products. 1 = popular first, 2 = latest first, 13 = top sales first, 8 = price low to high, 4 = price high to low
    • cat_id (optional): Filter products by a category.
    • bundle, upstream: Additional parameters with default settings.

  • Scraper Input Parameters

    • shopProducts_enrichUrlQuery_pageSize: Number of results per request (default: 30).
    • shopProducts_crawlNextPages: Set to true to crawl subsequent pages.
    • shopProducts_crawlNextPages_maxPages: Limit on number of pages to crawl (default: 100).
    • shopProducts_crawlNextPages_maxResults: Maximum products to crawl (default: 3000).
    • shopProducts_crawlNextPages_minSales: Minimum sales threshold for page crawling (optional). Crawling stops when a product with sales equal to or less than this value appears, assuming products are sorted by sales in descending order.
    • shopProducts_crawlProductDetails: Set to true to automatically crawl product details (default: false).

  • Example Input:

{
  // Get the products of a shop
  "requests": [
    { "url": "https://shopee.sg/api/v4/shop/rcmd_items?shop_id=178877065" }
  ],

  // Set the product "limit" URL parameter to 30
  "shopProducts_enrichUrlQuery_pageSize": 30,
  // Automatically crawl the next product pages
  "shopProducts_crawlNextPages": true,
  // Crawl until the 3rd product page
  "shopProducts_crawlNextPages_maxPages": 3,
  // Alternatively, indicate the maximum number of products to crawl
  "shopProducts_crawlNextPages_maxResults": 150,
  // Automatically crawl product details
  "shopProducts_crawlProductDetails": true
}
info

This API is scraped by using the POST method.

  • Notable Response Fields:
    • data/items: Array of product summaries.
    • data/items/itemid and items/shopid: Essential for scraping product details and ratings.
    • data/items/name: Product name.
    • data/items/image: Image key. Convert to URL with https://down-${country}.img.susercontent.com/file/${imageKey}.
    • data/items/ctime: Creation timestamp.
    • data/items/stock: Stock level.
    • data/items/sold: Monthly sales.
    • data/items/historical_sold: Total sales.
    • data/items/price: Divide by 100,000 to get actual price.
info

Unfortunately sold and historical_sold are not available anymore in non-authenticated mode. However, the Product Recommendation API still provides these fields.

warning

This API seems not available anymore in non-authenticated mode. However, the /api/v4/search/search_items API provides the same information with the following parameters:

{
  "requests": [
    // match_id corresponds to the shop_id
    { "url": "https://shopee.sg/api/v4/search/search_items?by=sales&entry_point=ShopByPDP&match_id=182624414&page_type=shop&scenario=PAGE_SHOP_SEARCH" }
  ]
}

Keyword Suggestion​

  • URL Path:
    • /api/v4/search/search_suggestion
info

The keyword suggestions are normally displayed under the top search bar.

  • URL Parameters:

    • limit (default = set by scraper): Number of results per page.
    • offset (default = 0): Offset for results, typically a multiple of limit.
    • bundle: An additional parameter, usually set to default.

  • Scraper Input Parameters

    • keywordSuggestions_enrichUrlQuery_pageSize: Number of keyword suggestions per request (default: 8).
    • keywordSuggestions_crawlNextPages: Set to true to crawl subsequent pages.
    • keywordSuggestions_crawlNextPages_maxPages: Limit on number of pages to crawl (default: 55).
    • keywordSuggestions_crawlNextPages_maxResults: Maximum keyword suggestions to crawl (default: 440).

  • Example Input:

{
  // Retrieve the 50 top suggested keywords
  "requests": [
    { "url": "https://shopee.sg/api/v4/search/search_suggestion?limit=50" }
  ],

  // Set the "limit" URL parameter to 50
  "keywordSuggestions_enrichUrlQuery_pageSize": 50,
  // Automatically crawl the next pages
  "keywordSuggestions_crawlNextPages": true,
  // Crawl until the 3rd page
  "keywordSuggestions_crawlNextPages_maxPages": 3,
  // Alternatively, indicate the maximum number of keyword suggestions to crawl
  "keywordSuggestions_crawlNextPages_maxResults": 150
}

Keyword Autocompletion​

  • URL Path:

    • /api/v4/search/search_hint

  • URL Parameters:

    • keyword: The initial characters of a potential search term.
    • extra_params, search_type, version: Additional parameters with default settings.

  • Example Input:

{
  // Retrieve autocomplete suggestions for the keyword prefix "n"
  "requests": [
    { "url": "https://shopee.sg/api/v4/search/search_hint?keyword=n" }
  ]
}

  • Example Response:

  • Notable Response Fields:

    • keywords/keyword: Suggested keyword based on the input characters.

  • URL Path:

    • /api/v4/search/search_mart_items

  • URL Parameters:

    • by (default = "relevancy"): Sort by "relevancy", "discount", "sales", or "price".
    • order (default = "desc"): Sort order, "asc" or "desc".
    • limit (default = 25): Number of results per page.
    • newest (default = 0): Offset for results, typically a multiple of limit.
    • keyword (optional): Search keyword, e.g., "cheese".

  • Example Input:

{
  "requests": [
    // First page of the "all products" page
    { "url": "https://shopee.sg/api/v4/search/search_mart_items?by=relevancy&limit=25&newest=0&order=desc" },

    // Search for the "cheese" keyword
    { "url": "https://shopee.sg/api/v4/search/search_mart_items?by=relevancy&keyword=cheese&limit=25&newest=0&order=desc" }
  ]
}

  • Example Response:

  • Notable Response Fields:

    • items: Array of product summaries.
    • items/itemid and items/shopid: Essential for scraping product details and ratings.
    • items/item_basic/name: Product name.
    • items/item_basic/image: Image key. Convert to URL with https://down-${country}.img.susercontent.com/file/${imageKey}, for example the URL to download the image "sg-11134201-7qvez-lgj50bviqvph8b" is "https://down-sg.img.susercontent.com/file/sg-11134201-7qvez-lgj50bviqvph8b".
    • items/item_basic/ctime: Creation timestamp.
    • items/item_basic/sold: Monthly sales.
    • items/item_basic/historical_sold: Total sales.
    • items/item_basic/price: Divide by 100,000 to get actual price.
info

Unlike with other APIs, the sold and historical_sold values are still available in non-authenticated mode.

It is legal to scrape publicly available data such as product descriptions, prices, or ratings. Read Apify's blog post on the legality of web scraping to learn more.

Your feedback​

Please don't hesitate to share your feedback (improvement ideas, bug, ...etc.). You can reach me on Discord (username "marcplouhinec").

Thanks​

Shopee Scraper is built with Crawlee, a great JavaScript framework to accelerate scraper development.

It also uses ungoogled-chromium via the Chrome DevTools Protocol. These powerful technologies have been instrumental in "opening" the website despite anti-bot protections.

Finally, a lot of knowledge that enabled the development of Shopee Scraper comes from The Web Scraping Club and its Discord Community.