APIEncoding

Encoding

Character encodings: converting between text and bytes.

UTF-8 and Latin-1 are supported. Neither encoding normalizes text, detects an encoding, removes a byte order mark, or converts newlines; a UTF-8 byte order mark decodes to U+FEFF. Byte offsets are zero-based and relative to the bytes given. Bytes.toStringLossy decodes UTF-8 with replacement instead of failing.

AsciiSet

type AsciiSetSource

A set of ASCII characters, built once by asciiSet and used by spanUntil to find where a run of text stops.

Encoding

type EncodingSource

A character encoding: utf8 or latin1.

Error

type Error    = InvalidSequence Int    | IncompleteSequence Int    | Unrepresentable CharSource

Why text could not be decoded or encoded.

Constructors

InvalidSequence

InvalidSequence : Int -> ErrorSource

A malformed sequence starts at this byte offset. For UTF-8 that includes overlong forms, surrogates, and values above U+10FFFF.

IncompleteSequence

IncompleteSequence : Int -> ErrorSource

A valid but unfinished sequence starts at this byte offset at the end of the input; more bytes may complete it.

Unrepresentable

Unrepresentable : Char -> ErrorSource

This character has no encoding in the selected encoding.

Instances:

Eq Error
Show Error

asciiSet

asciiSet : (Char -> Bool) -> AsciiSetSource

The ASCII characters that satisfy member. It calls member once per ASCII code while building the set and never afterwards, so build a set outside a scanning loop. Characters at or above U+0080 are never members.

decode

decode : Encoding -> Bytes -> Result Error StringSource

The text the bytes encode, or the first error in them.

import Bytes
import Encoding exposing (Error(..))
import Result exposing (Result(..))

Encoding.decode Encoding.utf8 (Bytes.fromList [104, 105]) == Ok "hi"
Encoding.decode Encoding.utf8 (Bytes.fromList [104, 255]) == Err (InvalidSequence 1)
Encoding.decode Encoding.utf8 (Bytes.fromList [104, 226, 130]) == Err (IncompleteSequence 1)
Encoding.decode Encoding.latin1 (Bytes.fromList [233]) == Ok "é"

decodeAt

decodeAt : Encoding -> Bytes -> Int -> Result Error (Maybe (Char, Int))Source

The character encoded at byte offset and its width in bytes, without consuming anything; Ok Nothing when the offset is outside the bytes. An offset inside a UTF-8 sequence is invalid. An IncompleteSequence may decode once more bytes are appended.

import Bytes
import Encoding exposing (Error(..))
import Result exposing (Result(..))

Encoding.decodeAt Encoding.utf8 (Bytes.fromString "é!") 0 == Ok (Just ('é', 2))
Encoding.decodeAt Encoding.utf8 (Bytes.fromString "é!") 1 == Err (InvalidSequence 1)
Encoding.decodeAt Encoding.utf8 (Bytes.fromString "a") 1 == Ok Nothing

encode

encode : Encoding -> String -> Result Error BytesSource

The bytes encoding the text, or Unrepresentable with the first character the encoding cannot represent.

import Bytes
import Encoding exposing (Error(..))
import Result exposing (Result(..))

Encoding.encode Encoding.latin1 "é" == Ok (Bytes.fromList [233])
Encoding.encode Encoding.latin1 "a€" == Err (Unrepresentable '€')

latin1

latin1 : EncodingSource

Latin-1 (ISO 8859-1): each byte is the character with the same code. Every byte sequence decodes; only U+0000 to U+00FF encode.

matchAt

matchAt : Encoding -> Bytes -> Int -> String -> Maybe IntSource

The encoded width of text when its encoding occurs in bytes at offset, or Nothing. Text the encoding cannot represent, and an offset outside the bytes, never match; empty text matches at any offset from zero through the length. Bytes after the match are not inspected.

import Bytes
import Encoding exposing (Error(..))
import Result exposing (Result(..))

Encoding.matchAt Encoding.utf8 (Bytes.fromString "key: v") 0 "key" == Just 3
Encoding.matchAt Encoding.utf8 (Bytes.fromString "key: v") 1 "key" == Nothing

message

message : Error -> StringSource

A sentence describing the error, for reporting.

import Bytes
import Encoding exposing (Error(..))
import Result exposing (Result(..))

Encoding.message (InvalidSequence 3) == "Invalid encoded sequence at byte 3"

spanUntil

spanUntil : Encoding -> AsciiSet -> Bytes -> Int -> IntSource

The end of the longest run of complete, valid characters from offset that contains no member of stops. A malformed or incomplete sequence ends the run, as does the end of the bytes; an offset outside the bytes is returned unchanged.

import Bytes
import Encoding

stops = Encoding.asciiSet { c -> c == ',' }

Encoding.spanUntil Encoding.utf8 stops (Bytes.fromString "ab,c") 0 == 2
Encoding.spanUntil Encoding.utf8 stops (Bytes.fromString "ab,c") 3 == 4

utf8

utf8 : EncodingSource

UTF-8, the encoding adapters use when none is given. Decoding rejects malformed sequences; encoding always succeeds, since a String is valid text.