Uploading a file URL to AWS pre-signed URL using Streaming

You can write a UploadService file.

The below code (Nodejs) is designed to upload a file from a given URL to an AWS S3 bucket using a pre-signed URL. The process is done through streaming, which is efficient for handling large files.

The uploadFileToPresignedUrl(presignedUrl, fileUrl) function is defined. This function is responsible for the actual uploading of the file.

• It first makes a GET request to the fileUrl using axios, with the response type set to 'stream'. This means that the file is read as a stream of data, which is useful for handling large files that may not fit into memory.
• The response from this request, which is a stream of the file data, is then used as the data in a PUT request to the presignedUrl. The headers of this request include the content type and length of the file.
• The PUT request also uses a httpsAgent with keepAlive set to true, which keeps the connection open until all the data has been sent.
• The maxContentLength is set to 5GB and maxBodyLength is set to Infinity, allowing for large files to be uploaded.

Here is the full source code:

"use strict";
const axios = require("axios");
const https = require("https");

async function uploadFileToPresignedUrl(presignedUrl, fileUrl) {
  const response = await axios({
    method: "get",
    url: fileUrl,
    responseType: "stream",
    httpsAgent: new https.Agent({ keepAlive: true }),
  });

  const fileStream = response.data;
  const uploadResponse = await axios({
    method: "put",
    url: presignedUrl,
    data: fileStream,
    headers: {
      "Content-Type": response.headers["content-type"],
      "Content-Length": response.headers["content-length"],
    },
    httpsAgent: new https.Agent({ keepAlive: true }),
    maxContentLength: 5 * 1024 * 1024 * 1024, // 1GB
    maxBodyLength: Infinity,
  });

  return uploadResponse;
}

class UploadService {
  constructor() {
    this.uploadStream = this.uploadStream.bind(this);
  }

  async uploadStream(presignedUrl, fileUrl) {
    try {
      const response = await uploadFileToPresignedUrl(presignedUrl, fileUrl);
      return true;
    } catch (error) {
      throw error;
    }
  }
}

module.exports = UploadService;