Storage

See how to store, read and serve files in Athenna.

Introduction

Athenna provides a filesystem abstraction on top of "disks". A disk is a named configuration bound to a driver, so the same code can write to the local filesystem in development and to an Amazon S3 bucket in production. The Storage facade of @athenna/storage exposes one API for every driver, plus an in-memory fake for tests.

Installation

First of all you need to install @athenna/storage package and configure it. Artisan provides a very simple command to install and configure the storage library in your project. Simply run the following:

node artisan install @athenna/storage

The storage configurer will do the following operations in your project:

Create the storage.ts configuration file.
Add the storage provider in your .athennarc.json file.
Add storage environment variables to .env, .env.test and .env.example.

tip

The configurer sets STORAGE_DISK=fs in all the .env files. Remember to change it to fake in your .env.test file, as explained in the testing section.

Configuration

Disks are configured in your application's

Path.config('storage.ts')

configuration file. Every disk has a driver and the options of that driver:

import { Env } from '@athenna/config'
import { Path } from '@athenna/common'

export default {
  default: Env('STORAGE_DISK', 'fs'),

  disks: {
    fs: {
      driver: 'fs',
      root: Path.storage(),
      url: Env('STORAGE_FS_URL')
    },
    fake: {
      driver: 'fake'
    },
    s3: {
      driver: 's3',
      bucket: Env('AWS_S3_BUCKET_NAME', ''),
      region: Env('AWS_REGION', ''),
      credentials: {
        accessKeyId: Env('AWS_ACCESS_KEY_ID', ''),
        secretAccessKey: Env('AWS_SECRET_ACCESS_KEY', '')
      },
      url: Env('AWS_S3_PUBLIC_URL')
    }
  }
}

Available storage drivers

Driver name	Description
`fs`	Local filesystem, files are stored under `root`.
`s3`	Amazon S3 and S3 compatible services (MinIO, R2, LocalStack).
`fake`	In-memory disk for tests, selected with `STORAGE_DISK=fake`.

The s3 driver accepts every option of the AWS SDK S3Client (region, credentials, endpoint, forcePathStyle, ...) plus the bucket and the optional public url. Old configuration files that declare key/secret keep working, the driver maps them to credentials for you.

One disk per bucket

Declare one disk per bucket instead of switching buckets at runtime:

disks: {
  s3: { driver: 's3', bucket: 'public-bucket', url: 'https://cdn.athenna.io', ... },
  s3_private: { driver: 's3', bucket: 'private-bucket', ... }
}

Then pick the disk with Storage.disk('s3_private'). Runtime options can still be merged on top of the disk configuration:

Storage.disk('s3', { bucket: 'another-bucket' })

Driver instances are cached by disk name and options, so calling Storage.disk('s3') many times does not create a new S3 client every time.

Writing files

import { Storage } from '@athenna/storage'

await Storage.put('avatars/user.png', buffer)
await Storage.disk('s3').put('reports/report.json', JSON.stringify(report), {
  contentType: 'application/json',
  cacheControl: 'max-age=3600',
  metadata: { owner: 'user-1' }
})

The content type defaults to the mime type of the key extension and falls back to application/octet-stream. The following options are available:

Option	Description
`contentType`	The `Content-Type` of the file.
`contentDisposition`	The `Content-Disposition` of the file.
`contentEncoding`	The `Content-Encoding` of the file.
`cacheControl`	The `Cache-Control` of the file.
`metadata`	User defined metadata as a `Record<string, string>`.
`acl`	The S3 canned ACL (`private`, `public-read`, `bucket-owner-full-control`, ...).

The fs driver ignores headers and metadata.

Streams

Big files should be written as streams. putStream() accepts the same options of put(). The s3 driver uses a multipart upload and reports progress:

await Storage.disk('s3').putStream('videos/movie.mp4', readable, {
  partSize: 10 * 1024 * 1024,
  queueSize: 4,
  onProgress: ({ loaded, total }) => console.log(loaded, total)
})

From a URL

putFromUrl() downloads the URL as a stream and writes it to the disk. The content type defaults to the content-type response header and a non 2xx response rejects with the HttpClient error:

await Storage.putFromUrl('avatars/user.png', 'https://example.com/user.png')

Reading files

const text = await Storage.get('notes.txt')
const buffer = await Storage.getBuffer('avatars/user.png')
const stream = await Storage.getStream('videos/movie.mp4')

tip

get() always decodes the file as UTF-8. Use getBuffer() for binary content.

Both getBuffer() and getStream() accept a byte range, which is handy to sniff a file signature without downloading the whole file:

const header = await Storage.getBuffer('videos/movie.mp4', {
  range: { start: 0, end: 11 }
})

Metadata

stat() returns the size, content type, last modification date, etag and user metadata of a file and throws FileNotFoundException when it is missing. exists() never throws:

const { size, contentType, lastModified, etag, metadata } = await Storage.stat('avatars/user.png')

if (await Storage.exists('avatars/user.png')) {
  // ...
}

Listing files

list() returns one page of files matching a prefix and a cursor to fetch the next page. Each page has up to 1000 files by default, use the limit option to change it:

let cursor: string | null = null

do {
  const page = await Storage.disk('s3').list('avatars/', { limit: 100, cursor })

  for (const file of page.files) {
    console.log(file.key, file.size, file.lastModified)
  }

  cursor = page.nextCursor
} while (cursor)

Copying, moving and deleting

await Storage.copy('avatars/user.png', 'avatars/user-backup.png')
await Storage.move('tmp/user.png', 'avatars/user.png')
await Storage.delete('avatars/user-backup.png')
await Storage.deleteAll('tmp/')

warning

Calling deleteAll() without a prefix removes every file of the disk. In the fs driver, the whole root folder is removed and created again empty.

Between disks

Pass toDisk to copy or move a file to another disk. Copies between two s3 disks happen server side (CopyObject), every other combination streams the file and forwards its content type and metadata:

await Storage.disk('s3_private').copy('videos/1.mp4', 'videos/1.mp4', {
  toDisk: 's3'
})

URLs

Every disk can build the public URL of a key and resolve a key back from a URL:

Storage.disk('s3').url('avatars/user 1.png')
// https://cdn.athenna.io/avatars/user%201.png

Storage.disk('s3').parseUrl('https://cdn.athenna.io/avatars/user%201.png?w=200')
// avatars/user 1.png

Storage.disk('s3').parseUrl('https://other-bucket.s3.amazonaws.com/a.png')
// null

For the s3 driver, url() uses the configured url when present, then the custom endpoint in path-style, then the virtual-hosted AWS URL (https://{bucket}.s3.{region}.amazonaws.com/{key}). parseUrl() accepts the configured url, virtual-hosted and path-style AWS URLs (with or without region) and custom endpoints, always checking that the bucket matches. Values that are not URLs are treated as relative keys, so parseUrl('/avatars/user.png') returns avatars/user.png.

The fs driver only builds URLs when its url option is configured and throws NotImplementedDriverMethodException otherwise. The fake driver builds URLs using its url option or https://fake.storage.

Signed URLs

Signed URLs let clients read or upload files directly against S3:

const { url, expiresAt } = await Storage.disk('s3').getSignedUrl('uploads/user.png', {
  method: 'put',
  expiresIn: 300,
  contentType: 'image/png'
})

The result has the signed url, the method, the key and the expiresAt date as an ISO string. The following options are available:

Option	Default	Description
`method`	`get`	`get` to read the file or `put` to upload it.
`expiresIn`	`300`	The number of seconds until the URL expires.
`contentType`	-	The `Content-Type` the client must send when uploading (`put`).
`responseContentType`	-	Override the `Content-Type` of the response when reading (`get`).
`contentDisposition`	-	Override the `Content-Disposition` of the response (`get`).

The fs driver doesn't support signed URLs and throws NotImplementedDriverMethodException. The fake driver returns a deterministic URL like https://fake.storage/put/uploads/user.png?expires=300.

Exceptions

The following exceptions are exported by @athenna/storage:

Exception	When
`FileNotFoundException`	Reading or stating a file that doesn't exist.
`NotImplementedConfigException`	Using a disk that is not configured in `storage.disks`.
`NotFoundDriverException`	Using a disk with a driver that is not registered.
`NotImplementedDriverMethodException`	Calling a method that the driver doesn't support.

Custom drivers

You can create your own driver by extending the Driver class and implementing its abstract methods. The constructor receives the disk configuration merged with the runtime options:

import type { Readable } from 'node:stream'
import { Driver } from '@athenna/storage'

export class GcsDriver extends Driver {
  public constructor(private options: any) {
    super()
  }

  public async put(key: string, content: string | Buffer) {
    // ...

    return this
  }

  public async putStream(key: string, content: Readable) {
    // ...

    return this
  }

  // exists(), stat(), get(), getBuffer(), getStream(), copy(), move(),
  // delete(), deleteAll(), list(), url(), parseUrl(), getSignedUrl()
}

Then register it with the DriverFactory and use it in your disks:

import { DriverFactory } from '@athenna/storage'

DriverFactory.createDriver('gcs', GcsDriver)

disks: {
  gcs: {
    driver: 'gcs',
    bucket: 'my-bucket'
  }
}

putFromUrl() is already implemented by the Driver class on top of your putStream(). Driver instances are cached by disk name and options, use DriverFactory.clearCache() if you need to create them again.

Testing

The fake driver keeps every file in memory, so tests never touch the network or the filesystem. Select it as the default disk in your .env.test file, the same way you do for the database and mail libraries:

.env.test
STORAGE_DISK=fake

Storage.put(), Storage.get(), Storage.exists(), Storage.stat(), Storage.list() and the other methods behave like a real disk on top of the FakeDriver.files map. Clear it between tests and assert on it directly:

import { FakeDriver, Storage } from '@athenna/storage'
import { Test, AfterEach, type Context } from '@athenna/test'

export default class UploadServiceTest {
  @AfterEach()
  public afterEach() {
    FakeDriver.clear()
  }

  @Test()
  public async shouldUploadTheAvatar({ assert }: Context) {
    await new UploadService().uploadAvatar(user, buffer)

    assert.isTrue(await Storage.exists('avatars/user-1.png'))
    assert.deepEqual(FakeDriver.getFile('avatars/user-1.png').contentType, 'image/png')
    assert.deepEqual(FakeDriver.files.size, 1)
  }
}

Code that names a disk explicitly (Storage.disk('s3')) does not go through the default disk. Point that disk at the fake driver in your test configuration, or stub the driver with the Mock class, exactly like the fake drivers of the other libraries:

import { FakeDriver } from '@athenna/storage'
import { Mock } from '@athenna/test'

Mock.when(FakeDriver, 'getSignedUrl').resolve({ url: 'https://fake.storage/put/a.png' })

See mocking storage for more information.

Introduction​

Installation​

Configuration​

Available storage drivers​

One disk per bucket​

Writing files​

Streams​

From a URL​

Reading files​

Metadata​

Listing files​

Copying, moving and deleting​

Between disks​

URLs​

Signed URLs​

Exceptions​

Custom drivers​

Testing​