Using the Multi-Part File Upload API

A file may be uploaded to an AppResult to which your application has WRITE access. There are two APIs for uploading files to an AppResult. This guide for multipart file upload describes a robust alternative for file upload over the simpler POST:appresults/{id}/files. Some of the advantages of using the multi-part upload API include:

  • Increased reliability as each part may be retried without needing to restart the full upload in case of a connection or application problem
  • Increased performance as parts may be uploaded in parallel
  • Ability to pause and resume uploads
  • Ability to start uploading before knowing the complete file size

API usage summary

  1. Initiate the upload into an AppResult by providing the filename, directory, and content-type
  2. Upload the file parts
  3. Complete the upload

Initiate the file upload

As part of initiating the file upload, you're telling the API some basics about the file including its filename, Content-Type, and directory.

Curl example to initiate upload

curl -v -H "x-access-token: {access token}" \
-H "Content-Type: application/zip" \
-X POST https://api.basespace.illumina.com/v1pre3/appresults/38040/files?name=reportarchive.zip\&multipart=true

This will initiate a multipart upload of a file named reportarchive.zip having a mimetype of application/zip to app result id 38040. Make sure you include the query parameter multipart=true to initiate the multipart upload. An optional directory query parameter may also be included.

If successful, a 200 OK is returned with a JSON file object. The UploadStatus should be pending.

HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 18 Jul 2012 14:51:50 GMT
Content-Length: 304
{
   "Response":{
      "UploadStatus":"pending",
      "HrefContent":"v1pre3/files/7094087/content",
      "Id":"7094087",
      "Href":"v1pre3/files/7094087",
      "Name":"reportarchive.zip",
      "ContentType":"application/zip",
      "Size":0,
      "Path":"reportarchive.zip",
      "DateCreated":"2012-07-18T14:51:50.0475813Z"
   },
   "ResponseStatus":{},
   "Notifications":[]
}

Upload the file parts

After initiating the upload, you may begin uploading the individual file parts. These must be numbered 1 to 10000 and will be reassembled by ascending part number when the upload is marked as completed later. Each part must be at least 5MB in size except for the final one. You may upload the parts in parallel and each part may be retried if the connection fails. Currently a part may not be larger than 25MB.

If you wish to calculate an MD5 checksum of the part prior to upload, you may provide your computed value to ensure that the checksum matches after upload. If the server-calculated value does not match the provided value, the upload is rejected. The client provides this checksum via the Content-MD5 header as a base64 encoded 128bit MD5 digest of the part content (excluding request headers). Although it's optional, its use is strongly recommended to ensure end-to-end data integrity.

Curl example to upload file parts

curl -v -H "x-access-token: {access token}" \
-H "Content-MD5: 9mvo6qaA+FL1sbsIn1tnTg==" \
-T reportarchive.zipaa \
-X PUT https://api.basespace.illumina.com/v1pre3/files/7094087/parts/1

This uploads the first part of the file as part number 1. Note that the above example shows a file that has already been split into parts on the filesystem (first part being reportarchive.zipaa as generated by split command) but you're free to use any method you want to upload a stream of the part segment.

If successful, a 200 OK is returned with a JSON file part object.

HTTP/1.1 200 OK
Content-Type: application/json
Date: Wed, 18 Jul 2012 15:21:12 GMT
Content-Length: 90

{
    "Response":{
        "Number":1,
        "ETag":"f66be8eaa680f852f5b1bb089f5b674e",
        "Size":10496561
    },
    "ResponseStatus":{},
    "Notifications":[]
}

The ETag included in the response is a hex encoded MD5 digest of the uploaded part data.

Complete the upload

After every file part has been uploaded, you'll need to make another API call to mark the upload as complete. This is done by passing query parameter uploadstatus=complete as a POST to the file.

Curl example to complete the upload

curl -v -H "x-access-token: {access token}" \
-X POST https://api.basespace.illumina.com/v1pre3/files/7094087?uploadstatus=complete

If successful, a 201 Created is returned with the JSON file object. The UploadStatus should be complete.

HTTP/1.1 201 Created
Content-Type: application/json
Date: Wed, 18 Jul 2012 16:20:00 GMT
Content-Length: 304
{
   "Response":{
      "UploadStatus":"complete",
      "HrefContent":"v1pre3/files/7094087/content",
      "Id":"7094087",
      "Href":"v1pre3/files/7094087",
      "Name":"reportarchive.zip",
      "ContentType":"application/zip",
      "Size":527623673,
      "Path":"reportarchive.zip",
      "DateCreated":"2012-07-18T14:51:50.0475813Z"
   },
   "ResponseStatus":{},
   "Notifications":[]
}

Upon getting this response, the file upload is complete.