Encodings and text I/O.

Decode transport bytes explicitly, then read and write Unicode characters.

Use byte readers for framing and text adapters for characters. Their units differ: bytes set the size of a transport frame, while text counts refer to Unicode scalars.

API reference: Encoding · Text.Reader · Text.Writer · Text.Builder

Encoding.

import Bytes
import Encoding

main() =
    print (Encoding.decode Encoding.utf8 (Bytes.fromString "café"))
    print (Encoding.decode Encoding.latin1 (Bytes.fromList [233]))

The results are Ok "café" and Ok "é". UTF-8 decoding rejects malformed and incomplete sequences. Latin-1 maps each byte to the scalar with the same code, so encoding a scalar above U+00FF fails.

Encoding is deliberately plain. It doesn’t detect formats, normalize text, remove a BOM, or change newlines. To report an error, use Encoding.message, and reach for Bytes.toStringLossy only when replacing bad bytes is really what you want.

Text.Reader.

import Text.Reader as Text

main() =
    result = Text.withString "hello" { reader -> Text.tryReadChar reader }
    print result

This prints Ok (Just 'h'). withString reads valid text that’s already in memory. over wraps a byte reader with UTF-8, and overWith lets you choose the encoding.

Ordinary reads can raise Fail Encoding.Error. If you’d rather get encoding errors back as Result values, use tryReadChar and the other try operations, which keep the source’s effects. The end of input is Nothing, which isn’t the same thing as an unfinished encoded character at the end.

readExactly and the delimited reads stage their consumption: if there’s too little input, an overflow, or a decoding failure, that call leaves its input unconsumed. readLine removes the LF and a preceding CR. Text line limits count scalars and include the terminator.

While a text adapter is active, advance the byte parent only through it. For a message framed in bytes, put Reader.limited outside the text scope.

Text.Writer.

A text writer accepts characters and strings, not bytes that are already encoded. We’ll start by collecting into memory:

import Text.Writer as Text

main() =
    (_, contents) = Text.collecting { output ->
        Text.write output "caf"
        Text.writeChar output 'é'
    }
    print contents

café

collecting returns the callback’s value and the string it accumulated. Over a byte writer, over chooses UTF-8 and overWith chooses another encoding. Each string is encoded before it’s emitted, so if one character can’t be represented, none of that string is written. Earlier writes aren’t rolled back.

Normal completion flushes an over adapter. With borrow, the final flush is left to the caller, and an abort skips it.

Text.Builder.

import Text.Builder as Builder

main() =
    first = Builder.empty |> Builder.append "hello"
    second = first |> Builder.append " world"
    print (Builder.toString second)
    print (Builder.toString first)

This prints hello world and hello. Builders are immutable values, and the append functions take the builder last so they chain with |>. Appending in sequence is efficient, and if you extend an older builder, its contents are preserved by copying as needed. Convert to a string once the text is assembled.

Buffered scanning.

A window is a snapshot you can use to inspect buffered text without moving the reader. Only commit the position you chose once the inspection has succeeded:

import Text.Reader as Text

main() =
    result = Text.withString "ab" { reader ->
        start = Text.window reader
        afterA = Text.skipAscii start
        Text.commitWindow reader afterA
        Text.tryReadChar reader
    }
    print result

Ok (Just 'b')

This uses skipAscii because we know the next character is ASCII. For general text, use the window’s decoding operations and handle incomplete or malformed characters. A window only describes the current buffer, so reaching its end may just mean the reader needs a refill, not that the input is over.

Keep positions from one snapshot together. Don’t mix ordinary reader calls with a scan you haven’t committed, and don’t reuse a window after the buffer changes. Most application code is clearer with readChar, readExactly, or readLine. Windows are meant for scanners that gain from batching inspection and consumption.