How to scrape sneakers from Amazon® with the Halios-data's API.

Amazon page analyze

For the example, we decide to scrape the following product https://www.amazon.com/dp/B07DBHWRJ9

As indicated, you have to select the size to see the price.

When you have selected the size, the URL is updated with the following query parameters "th"& "psc"

https://www.amazon.com/dp/B07DBM6WG4?th=1&psc=1

Scrap with the API and some JavaScript code

For this tutorial, we use the NX project with NestJS for the server-side. You can clone the project thanks to GitHub https://github.com/HaliosData/amazon-retrieve-shoes.git.

Today, you need to create an account to RAPIDAPI to consume the API, but don't worry, you can use your Google or Github account ;).

Subscribe to the API

After the login, you are generally redirected to the API hub. Else click here

You can test the API for free with 200 requests per month. Click on the subscribe button and choose the Basic plan.

Communicate with the API

To communicate with the API, in the example, we will use Axios.

First, install it thanks to npm.

Now inside the service, we implement the communication with the API in the app.service.ts with Axios library.

We inject the library in the top of our service

import { Axios } from 'axios';

We create the Axios instance in the constructor

private axios: Axios = null; constructor() { this.axios = new Axios({}); this.axios.interceptors.response.use(r => r, this.handleAxiosError); }

Now we can create a private method that handles the request thanks to Axios and get the Amazon object.

private async getProduct(asin: string) { try { var amazonProduct = await this.axios.request({ method: 'GET', url: 'https://amazon24.p.rapidapi.com/api/product/'.concat(asin), headers: { 'x-rapidapi-host': 'amazon24.p.rapidapi.com', 'x-rapidapi-key': 'YOUR_TOKEN' } }) } catch (e) { throw e; } return JSON.parse(amazonProduct.data); }

We can call our private method and test it with Postman or another tool.

async getData(): Promise<{ message: any; }> { try { var amazonProduct = await this.getProduct('B07DBM6WG4'); } catch (e) { throw e; } return { message: amazonProduct }; }

In my case, I call the following method HTTP://localhost:3333/api

Below, the JSON returned by the service

Like Amazon's website, the price is not present in the object returned by the service.

You have to select the size, to do this, you have to select it in the variantSizes attribute.

Scrap with the size variant URL

To do the scraping you need to realize the following implementation.

We need to change the method signature and add a second parameter "productURL".

private async getProduct(asin: string, productURL: string)

We also need to add another attribute to the Axios request "params"

params: { productURL: productURL }

Now, we can realize the test to scrap the shoes with the size 7, but before we need to add "0" to the asin and https://www.amazon.com/dp/B07DBHRNXB?psc=6 to productURL parameters.

Below is the code updated.

var amazonProduct = await this.getProduct('0', 'https://www.amazon.com/dp/B07DBHRNXB?psc=6');

The service returns

Thanks and good scraping :).