Linkhut.Archiving.Storage behaviour (linkhut v0.1.4)

View Source

Behaviour and abstraction layer for archive storage backends.

Crawlers produce content in various forms. store/3 persists it to the configured backend (local filesystem, S3, etc.) and returns a storage_key and metadata that identifies the stored content.

Summary

Types

serve_instruction()

How the stored content should be served to the client.

source()

The source of archive content to store.

store_meta()

Metadata returned by the storage backend after storing content.

Callbacks

delete(storage_key)

Deletes the content identified by the given storage key.

resolve(storage_key)

Resolves a storage key into a serve instruction for the controller.

resolve(storage_key, opts)

Resolves a storage key with additional options (e.g. content disposition for downloads).

storage_used(opts)

Returns the total bytes stored on the backend, optionally scoped to a user.

store(source, snapshot, opts)

Persists archive content to the storage backend.

Functions

delete(key)
resolve(key)

Resolves a storage key by dispatching to the backend that produced it.

resolve(key, opts)

Resolves a storage key with additional options.

storage_used(opts \\ [])
store(source, snapshot, opts \\ [])

Types

serve_instruction()

@type serve_instruction() :: {:file, Path.t()} | {:redirect, String.t()}

How the stored content should be served to the client.

  • {:file, path} — serve directly from the local filesystem
  • {:redirect, url} — redirect the client to an external URL (e.g. presigned S3 URL)

source()

@type source() :: {:file, Path.t()} | {:data, binary()} | {:stream, Enumerable.t()}

The source of archive content to store.

  • {:file, path} — a local file path (e.g., from SingleFile CLI output)
  • {:data, binary} — raw binary content
  • {:stream, enumerable} — a stream of binary chunks

store_meta()

@type store_meta() :: %{
  file_size_bytes: non_neg_integer(),
  encoding: String.t() | nil
}

Metadata returned by the storage backend after storing content.

  • file_size_bytes — size of the stored file on disk (may differ from original if compressed)
  • encoding — content encoding applied during storage (e.g. "gzip"), or nil if none

Callbacks

delete(storage_key)

@callback delete(storage_key :: String.t()) :: :ok | {:error, term()}

Deletes the content identified by the given storage key.

resolve(storage_key)

@callback resolve(storage_key :: String.t()) ::
  {:ok, serve_instruction()} | {:error, term()}

Resolves a storage key into a serve instruction for the controller.

resolve(storage_key, opts)

(optional) 
@callback resolve(storage_key :: String.t(), opts :: keyword()) ::
  {:ok, serve_instruction()} | {:error, term()}

Resolves a storage key with additional options (e.g. content disposition for downloads).

storage_used(opts)

@callback storage_used(opts :: keyword()) :: {:ok, non_neg_integer()} | {:error, term()}

Returns the total bytes stored on the backend, optionally scoped to a user.

store(source, snapshot, opts)

@callback store(source(), snapshot :: Linkhut.Archiving.Snapshot.t(), opts :: keyword()) ::
  {:ok, storage_key :: String.t(), store_meta()} | {:error, term()}

Persists archive content to the storage backend.

Returns a storage_key and metadata that can later be used to retrieve the content.

Functions

delete(key)

resolve(key)

Resolves a storage key by dispatching to the backend that produced it.

Uses StorageKey.parse/1 to determine which module handles resolution, regardless of the currently configured storage backend.

resolve(key, opts)

Resolves a storage key with additional options.

Currently only S3 supports options (e.g. :disposition for downloads). Other backends fall back to resolve/1.

storage_used(opts \\ [])

store(source, snapshot, opts \\ [])