Puppeteer – URL Sequence (steps) monitor

With puppeteer, we normally visit a website, that website has a form for us to submit the data. In the api world, there is no form for us to submit, when you visit you must embed the request data in the request. the data could be username or password or some special headers. This is easy done with curl. How can we do that with puppeteer? We can do that, Puppeteer allows us to modify the request before sending it to the server. Here is a sample code:

await page.setRequestInterception(true);
var accessToken ="";
page.on('request', request =>
{
  const overrides = {};
  if (request.url().indexOf('api/step1')>0) {
    const headers = request.headers();
    headers['content-type'] = 'application/x-www-form-urlencoded';
    overrides.method = 'POST';
    overrides.postData = 'username=a&password=b';
     overrides.headers=headers;
  }
  if (request.url().indexOf('api/step2')>0) {
    const headers = request.headers();
    headers['APIKEY'] = 'myapi-key';
    headers['Authorization'] = 'Bearer '+accessToken;
    headers['Content-Type'] = 'application/json';
    overrides.headers=headers;
  }
  
  request.continue(overrides);
});
await page.goto('http://www.example.com/api/step1');
result = await page.content();
await page.screenshot({ path: '/puppeteer/step1.png' });

if (result.indexOf("accessToken")==-1)
{
   console.log("There is no [accessToken] in the response");
   console.log(result);
}else
{
   accessToken = result.split("\"accessToken\":\"").pop().split("\"")[0];
   console.log("Found [accessToken]: " + accessToken );
   await page.goto('http://www.example.com/api/step2');
   result = await page.content();
   console.log(result);
   await page.screenshot({ path: '/puppeteer/step2.png' });
}