std::storage
Object storage: files kept whole under keys, in a directory, an S3
bucket or memory. Code that stores files calls the Store interface,
and each backend implements it, so one function serves the local
directory in development, a bucket in production and a test’s
std::storage::testing.Memory:
fn save_avatar(user_id: Int, data: Bytes) -> Result<Url, storage.Error> key = "avatars/${user_id}.png" Store.put(App.store, key, data, "image/png")? Store.url(App.store, key) end
storage.local(root) keeps objects as plain files under a directory,
and storage.handler(store) serves any store’s objects through the app.
lib/storage/s3 is a backend for any server that speaks the S3
protocol: DigitalOcean Spaces, MinIO, Cloudflare R2, Backblaze B2 and
AWS. Boot code picks one:
store = if production then s3.store(config)? else storage.local("tmp/files", public_base: Some(Url"/files/")) end
std::storage::testing has Memory, a store kept in memory for tests,
Failing, a store whose every call fails, and conformance, the suite
every backend passes.
Import with import std::storage.
Exports
Section titled “Exports”Every backend enforces one key grammar (storage.check_key), so a key
that works on the local store works on S3:
- 1 to 1,024 bytes of UTF-8, in segments separated by
/, none of them empty; - each segment at most 255 bytes, and none starting with
.; - no control character and no
\.
Anything else is allowed: spaces, punctuation such as (, !, %,
# and ?, and letters of any script, so a store can address the
objects a bucket already holds. ../etc/passwd, /abs and a\b are
refused before any backend builds a path or a URL from them.
Listing
Section titled “Listing”Store.list returns one page of the keys under a prefix, in order,
with each object’s size and time. With a delimiter, keys that go on
past it are folded into the common prefix they share, as a directory
listing shows a folder once; a page’s next is passed back as
continuation for the page after it. storage.list_all follows the
pages to the end:
page = Store.list(App.store, ListOptions{prefix: "avatars/", delimiter: "/"})? everything = storage.list_all(App.store, prefix: "avatars/")?
Objects move whole, as Bytes; there is no streaming or multipart
upload.
enum Error
Section titled “enum Error”enum Error NotFound { key: String } InvalidKey { key: String, reason: String } InvalidContentType { content_type: String } NoPublicUrl Denied { message: String } Unavailable { message: String } Backend { code: String, message: String } end
Why a store call failed.
Variants
-
NotFoundNo object has the key.
getreturns it;inforeturnsOk(None)instead, anddeletereturnsOk. -
InvalidKeyThe key breaks the key rules (
check_key).reasonsays which. -
InvalidContentTypeThe content type is empty, longer than 255 characters, or holds a character outside printable ASCII.
-
NoPublicUrlThe store was made with no public URL for its objects.
-
DeniedThe backend refused the credentials or the permission.
-
UnavailableThe backend could not be reached, timed out, or said it is overloaded. The same call may succeed later.
-
BackendAnything else, with the backend’s own code: an S3 error code such as
NoSuchBucket, or anio.Errorvariant’s name for the local store.
assert "${Error.NotFound{key: "a.txt"}}" == "no object has the key a.txt"
Error.to_string
Section titled “Error.to_string”impl Display for Error
fn to_string(error: Error) -> String
impl Display.to_string
Error.equal
Section titled “Error.equal”impl Equatable for Error
fn equal(a: Error, b: Error) -> Bool
impl Equatable.equal
Error.inspect
Section titled “Error.inspect”impl Debug for Error
fn inspect(value: Error) -> String
impl Debug.inspect
struct Info
Section titled “struct Info”struct Info size: Int content_type: String modified: Instant end
What a store knows about an object without reading it.
Info.equal
Section titled “Info.equal”impl Equatable for Info
fn equal(a: Info, b: Info) -> Bool
impl Equatable.equal
Info.inspect
Section titled “Info.inspect”impl Debug for Info
fn inspect(value: Info) -> String
impl Debug.inspect
struct Object
Section titled “struct Object”struct Object data: Bytes info: Info end
An object read whole: its bytes and its Info.
Object.equal
Section titled “Object.equal”impl Equatable for Object
fn equal(a: Object, b: Object) -> Bool
impl Equatable.equal
Object.inspect
Section titled “Object.inspect”impl Debug for Object
fn inspect(value: Object) -> String
impl Debug.inspect
struct Entry
Section titled “struct Entry”struct Entry key: String size: Int modified: Instant end
An object as a listing shows it: its key, its size in bytes, and when it was last stored.
Entry.equal
Section titled “Entry.equal”impl Equatable for Entry
fn equal(a: Entry, b: Entry) -> Bool
impl Equatable.equal
Entry.inspect
Section titled “Entry.inspect”impl Debug for Entry
fn inspect(value: Entry) -> String
impl Debug.inspect
struct ListOptions
Section titled “struct ListOptions”struct ListOptions prefix: String = "" delimiter: String = "" continuation: Maybe<String> = None limit: Int = 1000 end
Which keys Store.list lists, and which page of them.
ListOptions.equal
Section titled “ListOptions.equal”impl Equatable for ListOptions
fn equal(a: ListOptions, b: ListOptions) -> Bool
impl Equatable.equal
ListOptions.inspect
Section titled “ListOptions.inspect”impl Debug for ListOptions
fn inspect(value: ListOptions) -> String
impl Debug.inspect
struct Listing
Section titled “struct Listing”struct Listing objects: List<Entry> = [] prefixes: List<String> = [] next: Maybe<String> = None end
One page of a listing: the objects and the common prefixes on it, each
in key order, and the continuation for the page after it, or None on
the last page.
Listing.equal
Section titled “Listing.equal”impl Equatable for Listing
fn equal(a: Listing, b: Listing) -> Bool
impl Equatable.equal
Listing.inspect
Section titled “Listing.inspect”impl Debug for Listing
fn inspect(value: Listing) -> String
impl Debug.inspect
interface Store
Section titled “interface Store”interface Store fn put(store: self, key: String, data: Bytes, content_type: String) -> Result<Unit, Error> fn get(store: self, key: String) -> Result<Object, Error> fn info(store: self, key: String) -> Result<Maybe<Info>, Error> fn delete(store: self, key: String) -> Result<Unit, Error> fn url(store: self, key: String) -> Result<Url, Error> fn list(store: self, options: ListOptions) -> Result<Listing, Error> fn exists(store: self, key: String) -> Result<Bool, Error> end
A place objects live, each under a key. storage.local, lib/storage/s3
and std::storage::testing.Memory implement it.
Every backend follows these rules, and std::storage::testing’s
conformance suite checks each:
- Every function checks its key with
check_keyfirst, and returns itsErr(.InvalidKey)before doing anything else.putthen checks the content type withcheck_content_type. putreplaces any object under the key. A reader sees the old object or the new one, never part of one.getandinforeturn the bytes and content type the lastputstored, and its size.getof a missing key isErr(.NotFound);infoof one isOk(None);deleteof one isOk.urldoes no I/O. It returns a URL whether or not the object exists.listreturns the keys under its prefix in byte order, its prefix checked withcheck_prefixfirst, folds keys that go on past the delimiter into their common prefix, and pages asListOptionssays;pageis that rule over a list of entries.
import std::storage::testing.Memory
store: Store = Memory.new()
assert Store.put(store, "notes/a.txt", String.to_bytes("hi"), "text/plain") == Ok(Unit)
assert Store.get(store, "notes/a.txt")!.data == String.to_bytes("hi")
assert Store.exists(store, "notes/b.txt") == Ok(False)
Store.put
Section titled “Store.put”fn put(store: self, key: String, data: Bytes, content_type: String) -> Result<Unit, Error>
interface Store
Stores data under key with content_type, replacing any object
there.
Store.get
Section titled “Store.get”fn get(store: self, key: String) -> Result<Object, Error>
interface Store
The object under key, read whole, or Err(.NotFound).
Store.info
Section titled “Store.info”fn info(store: self, key: String) -> Result<Maybe<Info>, Error>
interface Store
What the store knows about the object under key, or None when
there is none.
Store.delete
Section titled “Store.delete”fn delete(store: self, key: String) -> Result<Unit, Error>
interface Store
Removes the object under key. A missing key is Ok.
Store.url
Section titled “Store.url”fn url(store: self, key: String) -> Result<Url, Error>
interface Store
Where a reader fetches the object under key, with no I/O, or
Err(.NoPublicUrl) from a store made without one.
Store.list
Section titled “Store.list”fn list(store: self, options: ListOptions) -> Result<Listing, Error>
interface Store
One page of the objects whose keys start with options.prefix.
Store.exists
Section titled “Store.exists”fn exists(store: self, key: String) -> Result<Bool, Error>
interface Store
Whether an object has the key.
storage.check_key
Section titled “storage.check_key”fn check_key(key: String) -> Result<Unit, Error>
Ok when key is a valid key, else Err(.InvalidKey) saying why. A
key is 1 to 1,024 bytes of segments separated by /. Each segment is 1
to 255 bytes, does not start with ., and holds no control character
(U+0000 to U+001F, U+007F to U+009F) and no \. Any other character
is allowed, as S3 allows it.
assert check_key("posts/2024-05-01/a_1.jpg") == Ok(Unit)
assert check_key("posts/Noah_-_3010(1).jpg") == Ok(Unit)
assert check_key("100% café/a b#1?.txt") == Ok(Unit)
assert check_key("../etc/passwd") == Err(Error.InvalidKey{
key: "../etc/passwd",
reason: "segment .. starts with a dot",
})
assert check_key("a//b") is Err(.InvalidKey{})
assert check_key("a\\b") is Err(.InvalidKey{})
assert check_key("a\nb") is Err(.InvalidKey{})
assert check_key("") is Err(.InvalidKey{})
storage.check_prefix
Section titled “storage.check_prefix”fn check_prefix(prefix: String) -> Result<Unit, Error>
Ok when prefix can start a key, as list takes it: at most 1,024
bytes, with no control character and no \. It may be empty, end
with / or stop inside a segment.
assert check_prefix("") == Ok(Unit)
assert check_prefix("avatars/") == Ok(Unit)
assert check_prefix("a\\b") is Err(.InvalidKey{})
storage.page
Section titled “storage.page”fn page(entries: Iter<Entry>, options: ListOptions) -> Listing
The page of entries that options asks for, as every backend lists:
the entries whose keys start with the prefix, in key order, each key
that holds the delimiter after the prefix folded into its common
prefix, after the continuation, at most limit of them. The
continuation is the last key or prefix of the page before, so a
backend that holds its keys can list with this alone.
at = Instant.from_seconds(0)
entry = fn key: String -> Entry{key:, size: 1, modified: at}
entries = [entry("p/b/2"), entry("p/a"), entry("p/b/1"), entry("q"), entry("p/c")]
assert page(entries, ListOptions{prefix: "p/"}).objects
|> Iter.map(fn e -> e.key)
|> Iter.to_list() == ["p/a", "p/b/1", "p/b/2", "p/c"]
assert page(entries, ListOptions{prefix: "p/", delimiter: "/", limit: 2}) == Listing{
objects: [entry("p/a")],
prefixes: ["p/b/"],
next: Some("p/b/"),
}
assert page(
entries,
ListOptions{prefix: "p/", delimiter: "/", limit: 2, continuation: Some("p/b/")},
) == Listing{objects: [entry("p/c")]}
storage.list_all
Section titled “storage.list_all”fn list_all(store: Store, prefix: String = "", delimiter: String = "", page_size: Int = 1000) -> Result<Listing, Error>
Every object and common prefix under prefix, following each page’s
next to the last page.
import std::storage::testing.Memory
store = Memory.new()
_ = Store.put(store, "a/1", String.to_bytes("x"), "text/plain")
_ = Store.put(store, "a/2/z", String.to_bytes("y"), "text/plain")
all = list_all(store, prefix: "a/", delimiter: "/", page_size: 1)!
assert all.objects |> Iter.map(fn e -> e.key) |> Iter.to_list() == ["a/1"]
assert all.prefixes == ["a/2/"]
assert all.next == None
storage.join_key
Section titled “storage.join_key”fn join_key(base: Url, key: String) -> Url
base with the segments of key joined to its path, each escaped, so
that a %, ? or # in a key stays part of the path. A backend’s
url returns it for its public base. key is not checked.
u = join_key(Url"https://cdn.example.com/files/", "a b/50%(1)#2.jpg")
assert u.path == "/files/a b/50%(1)#2.jpg"
assert "${u}" == "https://cdn.example.com/files/a%20b/50%25%281%29%232.jpg"
storage.check_content_type
Section titled “storage.check_content_type”fn check_content_type(content_type: String) -> Result<Unit, Error>
Ok when content_type can be stored: 1 to 255 characters of
printable ASCII, space to ~. It refuses CR and LF, which could
otherwise add a header, but does not parse the media type: image/jpg
is stored as written.
assert check_content_type("image/jpeg") == Ok(Unit)
assert check_content_type("text/html; charset=utf-8") == Ok(Unit)
assert check_content_type("") is Err(.InvalidContentType{})
assert check_content_type("text/plain\r\nX-Evil: 1") is Err(.InvalidContentType{})
storage.local
Section titled “storage.local”fn local(root: String, public_base: Maybe<Url> = None) -> Store
A store over the directory root. Each object is a plain file at
<root>/<key>, so the directory can be backed up, inspected or served
by anything; its content type is kept beside it in
<root>/.nomi-storage/types/<key>, which no key can name. root is
created on the first put.
A few characters of a key are written on disk as % and two
hexadecimal digits, because some file system cannot hold them in a
name: % itself, <, >, :, ", |, ? and *; a . or space
that ends a segment, which Windows drops; and the first letter of a
segment Windows keeps for a device (CON, nul.txt, COM1). So the
key a:b.txt is the file a%3Ab.txt, and a file copied in by hand is
found under the key its name spells only when its name holds none of
these.
public_base is where the app serves the directory, as
Some(Url"/files/") with a route to storage.handler; url joins the
key to it. With None, url returns Err(.NoPublicUrl).
A put writes the type file, then the object, each to a temporary file
renamed over the old one, so a reader sees the old object or the new
one. A file someone copied in with no type file is read with the type
http.detect_content_type gives its bytes. Errors from the file system
map to NotFound, Denied for PermissionDenied, and Backend with
the io.Error variant’s name for the rest.
It differs from S3 in three ways:
- Keys
aanda/bcannot both exist, sinceacannot be a file and a directory. The secondputisErr(.Backend). - Two concurrent
puts to one key may pair one put’s bytes with the other’s type. Keys that never collide, as ones built from a timestamp or a UUID, are safe. - On a case-insensitive file system, as macOS and Windows have by
default, keys that differ only in case name one file, and on macOS
keys that differ only in Unicode normalization (
éas one code point or two) do too.
The calls go through std::fs, so std::fs::testing.run sandboxes the store in
tests, and fs.within applies.
import std::fs::testing
ran = testing.run(testing.Sandbox{}, fn ->
store = local("files", public_base: Some(Url"/files/"))
Store.put(store, "a/b.txt", String.to_bytes("hi"), "text/plain")?
Ok((Store.get(store, "a/b.txt")?.info.content_type, "${Store.url(store, "a/b.txt")?}"))
end)
assert ran.value == Ok(("text/plain", "/files/a/b.txt"))
assert testing.text(ran, "/work/files/a/b.txt") == Some("hi")
assert testing.text(ran, "/work/files/.nomi-storage/types/a/b.txt") == Some("text/plain")
storage.handler
Section titled “storage.handler”fn handler(store: Store, param: String = "key") -> (Request) -> Response
A handler that serves GET and HEAD with the object whose key is the
request’s path value param, with its stored content type. It handles
Range and conditional requests as Response.content does. A missing
object or an invalid key is a 404, Denied a 403, Unavailable a 503,
and any other error a 500.
files = storage.handler(App.store) routes = [Route.to_with("GET /files/{key...}", fn r, _ -> files(r))]
It sets no Cache-Control: an app whose keys never change wraps the
handler and adds one. It reads the whole object into memory first,
which suits images and documents, not video.
import std::http.{Method, Status}
import std::http::testing
import std::storage::testing.Memory
store = Memory.new()
_ = Store.put(store, "a/b.css", String.to_bytes("p{}"), "text/css")
serve = handler(store)
found = serve(
Request.with_path_value(testing.request(Method.Get, "/files/a/b.css"), "key", "a/b.css")
)
assert found.status == Status.ok
assert Headers.get(found.headers, "Content-Type") == Some("text/css")
assert Response.text(found) == Ok("p{}")
missing = serve(Request.with_path_value(testing.request(Method.Get, "/files/x"), "key", "x"))
assert missing.status == Status.not_found