Files, blobs and direct uploads

System Design · lesson 18 of 32 · 4 min read

Keep big files out of the database and off your app servers, and serve them from the edge.

Open this lesson in the learning hub

Key points

  • Never keep images or video in the database. Rows balloon, backups crawl, and you pay to stream every byte through your app.
  • Object storage (S3, GCS, Azure Blob) is cheap, effectively unlimited and already replicated. Your table stores only the key.
  • Upload straight from the browser with a presigned URL: your app signs a short-lived PUT and never sees the bytes.
  • Anything past about 100 MB uses a multipart upload, so a dropped connection retries one part instead of the whole file.
  • An object-created event feeds a queue, and workers make thumbnails or transcode. The upload response never waits for them.
  • Serve reads through the CDN. Keys are immutable, so a long max-age is safe; sign the URL when the file is private.

Example

// The app never handles the bytes: it only signs a short-lived PUT.
@PostMapping("/uploads")
UploadTicket create(@RequestBody NewUpload req, Principal user) {

    String key = "u/" + user.getName() + "/" + UUID.randomUUID() + "." + req.ext();

    PutObjectRequest put = PutObjectRequest.builder()
            .bucket("media-prod")
            .key(key)
            .contentType(req.contentType())
            .build();

    PresignedPutObjectRequest signed = presigner.presignPutObject(
            PutObjectPresignRequest.builder()
                    .signatureDuration(Duration.ofMinutes(5))   // a narrow window
                    .putObjectRequest(put)
                    .build());

    media.save(new MediaRow(key, user.getName(), Status.PENDING));  // the key only
    return new UploadTicket(signed.url().toString(), key);
}

// S3 fires ObjectCreated -> SQS -> here. The POST above did not wait for any of it.
@SqsListener("media-created")
void onUploaded(ObjectCreated event) {
    thumbnails.generate(event.key(), 200, 600, 1200);
    media.markReady(event.key());
}

Sign a short-lived URL, let the client talk to object storage, and do the processing off an event.

This is a reading copy. The full lesson — with the visual explainer, the interactive lab and a Run button for the code — lives in the System Design course, and every lesson in it is listed on the System Design contents page.