Scrape Amazon's products with variant sizes and colours with JavaScript.

Updated: Mar 24

This article aims to explain how to scrape the products with variant sizes. Indeed, as shown in the below picture, you don't have the price. You have to select first the size. And after you can choose the colour wished.

For example, for the following => https://www.amazon.com/dp/B08RG75YHB.














After the size selection, you can see the price,


Now, let's go to the halios-data API to implement the solution!


Create an account to RapidAPI

Today, you need to create an account to RAPIDAPI to consume the API, but don't worry, you can use your Google or Github account ;).



Subscribe to the API


After the login, you are generally redirected to the API hub. Else click here

You can test the API for free with 200 requests per month. Click on the subscribe button and choose the Basic plan.















Create the project with NestJS

In your terminal, use the following command.


git clone https://github.com/nestjs/typescript-starter.git tutorial-amazon-variant

go to the folder and use npm install



 npm i

After the installation, we can see the following structure.


  • app.controller.ts: we will develop access to our API, i.e. our endpoint;

  • app.module.ts: like Angular framework, the module represents the partitioning;

  • app.service.ts: we will develop our logic of scraping engine consuming and create our custom object









Let's go to scrape!


Implementation of solution.

Implementation of service

the service aim is to realize the communication with the halios-data API and return the result to the controller.


Now, we implement the communication with the API thanks to RAPIDAPI. To Realize the communication, we propose Axios. You can simply add it to your package.json with the following command.

npm i -S axios

Below, the communication implementation with RAPIDAPI

To do that, we create a new method in your app.service.ts


getProduct() {
    const response = {};
    return response;
  }

Inside we add the folowing implementation

const options: any = {
      method: 'GET',
      url: 'https://amazon24.p.rapidapi.com/api/product/B08RG75YHB',
      params: { country: 'US' },
      headers: {
        'X-RapidAPI-Host': 'amazon24.p.rapidapi.com',
        'X-RapidAPI-Key': 'YOUR_TOKEN',
      },
    };

    const response = await axios.request(options);

    return response.data;

You need to add your token to the X-RapidAPI-Key query. You can find him in the RAPIDAPI HMI



Now we can call our method inside our controller, we create our new method inside the app.controller.ts and we call the method that returns the product.


@Get('product')
  async getProduct() {
    return await this.appService.getProduct();
  }

It's time to test our endpoint witch curl or postman.

http://localhost:3000/product

And you can see the result after a few seconds.


{
    "variantAsin": {
        "White": {
            "asin": "B07KKZ9L6H"
        },
        "White\/Black": {
            "asin": "B0059CL74W"
        },
        "black": {
            "asin": "B0059CL1P2"
        },
        "Black\/White\/Grey": {
            "asin": "B07HXVNG4Q"
        },
        "Black": {
            "asin": "B0059CL20G"
        }
    },
    "variantImages": {
        "White": [...

You can see that the price is 0 like the website because we need to select the size


 "price_information": {
        "app_sale_price": null,
        "currency": "$",
        "original_price": 0,
        "discount": 0,
        "discount_percentage": 0
    }

Filter by the size

To realize the operation, we need to get the information thanks to variantSizes attribute in the root of the object.


"variantSizes": [
        {
            "product_id": "B0059CL20G",
            "size": "Large Fits Shoe Size: 5Y-7T Sock Size 9-...",
            "url": "https://www.amazon.com/dp/B0059CL20G?psc=1"
        },
        {
            "product_id": "B08RG75YHB",
            "size": "Large",
            "url": "https://www.amazon.com/dp/B08RG75YHB?psc=3"
        },
        {
            "product_id": "B07QGHW71J",
            "size": "5-7",
            "url": "https://www.amazon.com/dp/B07QGHW71J?psc=4"
        }
    ],

It's the same result that the website















If you want the information about the size of the large fits. You have to realize the following implementation


 const options: any = {
      method: 'GET',
      url: 'https://amazon24.p.rapidapi.com/api/product/0',
      params: {
        country: 'US',
        productURL: 'https://www.amazon.com/dp/B0059CL20G?psc=1',
      },
      headers: {
        'X-RapidAPI-Host': 'amazon24.p.rapidapi.com',
        'X-RapidAPI-Key': 'YOUR_TOKEN',
      },
    };

    const response = await axios.request(options);

    return response.data;

We have replaced the productID on the path by 0 and added the productURL in the params attribute of the options object.

The productURL corresponds to the URL in the variantSizes attribute.


"variantSizes": [
        {
            "product_id": "B0059CL20G",
            "size": "Large Fits Shoe Size: 5Y-7T Sock Size 9-...",
            "url": "https://www.amazon.com/dp/B0059CL20G?psc=1"
        },

And your service returns now the price of your product.

"price_information": {
        "app_sale_price": 32.82,
        "currency": "$",
        "original_price": 32.82,
        "discount": null,
        "discount_percentage": null
    }

Filter by variant ASIN

In our case, we can select the different colours, for example, white.

{
    "variantAsin": {
        "White": {
            "asin": "B07KKZ9L6H"
        },
        "White\/Black": {
            "asin": "B0059CL74W"
        },
        "black": {
            "asin": "B0059CL1P2"
        },
        "Black\/White\/Grey": {
            "asin": "B07HXVNG4Q"
        },
        "Black": {
            "asin": "B0059CL20G"
        }
    },
    "variantImages": {
        "White": [...

You have to add it in the URL of the request


const options: any = {
      method: 'GET',
      url: 'https://amazon24.p.rapidapi.com/api/product/B07KKZ9L6H',
      params: {
        country: 'US',
      },
      headers: {
        'X-RapidAPI-Host': 'amazon24.p.rapidapi.com',
        'X-RapidAPI-Key': 'YOUR_TOKEN',
      },
    };

    const response = await axios.request(options);

    return response.data;

The service returns the result


You can see that for the white colour, you have just one size if we check the website


And as before, we have to select the size to display the price.




 "variantSizes": [
        {
            "product_id": "B07KKZ9L6H",
            "size": "5-7",
            "url": "https://www.amazon.com/dp/B07KKZ9L6H?psc=4"
        }
    ],


const options: any = {
      method: 'GET',
      url: 'https://amazon24.p.rapidapi.com/api/product/0',
      params: {
        country: 'US',
        productURL: 'https://www.amazon.com/dp/B07KKZ9L6H?psc=4',
      },
      headers: {
        'X-RapidAPI-Host': 'amazon24.p.rapidapi.com',
        'X-RapidAPI-Key': 'YOUR_TOKEN',
      },
    };

    const response = await axios.request(options);

    return response.data;

And the service returns

 "price_information": {
        "app_sale_price": 20.98,
        "currency": "$",
        "original_price": 20.98,
        "discount": null,
        "discount_percentage": null
    }

To upgrade the code, you can add a new method that realizes the scraping for the size of one product.


Thank you and good scraping with us!










66 views0 comments

Discover Halios-Data solution for the web scraping

More Halios-Data

Never miss an update

Thanks for submitting!