Approximate Numerics in Applied Transition Environments

Reference

Approximate Numerics in Applied Transition Environments

This document lists every public call in aniate, with its arguments and return values. For guidance on which call suits a task, see tree.md.

aniate works with finite models whose states can be listed in memory. It solves them exactly, simulates them, and tests external simulators against them.

pip install aniate            # numpy and scipy
pip install 'aniate[vis]'     # adds matplotlib, required for plots
pip install 'aniate[gym]'     # adds gymnasium, required for aniate.gym

Verifying the installation: ant

ant(stream=None)

Prints the Fibonacci drawing, the installed versions of Python, numpy, scipy, matplotlib, pytest, hypothesis and gymnasium, and the result of a self-test. The self-test builds, solves, simulates and checks a five-state model, and writes a PDF plot to a temporary directory when matplotlib is installed. Returns True when every required component works. stream defaults to standard output. python -m aniate and the aniate command run the same check and exit with status 0 or 1.

from aniate import ant
ant()

Complete workflow

import aniate as an
from aniate import nmdp, vis

# 1. define
m = an.from_functions(
    states=range(5), actions=["left", "right"],
    transition=lambda s, a: {max(s - 1, 0): 1.0} if a == "left" else {min(s + 1, 4): 0.8, s: 0.2},
    reward=lambda s, a, s_next: 10.0 if s_next == 4 else -1.0,
    gamma=0.9, terminal=[4], initial=0,
)
print(m.summary())
m.describe()

# 2. run
env = m.env(seed=0)
obs, info = env.reset()
obs, reward, terminated, truncated, info = env.step("right")
runs = an.simulate(m, policy=None, episodes=10_000)     # uniform random policy

# 3. check, against the model's own environment or an external simulator passed as env=
an.check(m)

# 4. solve
sol = m.solve()

# 5. plot
vis.save(vis.overview(an.simulate(m, sol, episodes=10_000)), "overview.pdf")

1. Defining a model

MDP(P, R, gamma, *, states=None, actions=None, initial=None, terminal=None, horizon=None, validate=True)

Builder(gamma=0.95, horizon=None)

b = an.Builder(gamma=0.9)
b.transition("ok", "run", "ok", prob=0.9, reward=10)      # one call per outcome
b.transition("ok", "run", "broken", prob=0.1, reward=10)
b.transition("ok", "sell", "done", reward=50)
b.transition("broken", "run", "broken", reward=0)
b.transition("broken", "sell", "done", reward=5)
b.terminal("done")
b.initial("ok")                       # or {"ok": 0.7, "broken": 0.3}
m = b.build(missing="error")          # missing="stay" makes undefined actions leave the state unchanged

from_functions(states, actions, transition, reward=None, gamma=0.95, *, initial, terminal, horizon)

transition(s, a) returns a dict {s2: probability}. reward(s, a, s2) or reward(s, a) returns a number. terminal may be a list of states or a predicate.

Model attributes and methods

Every public call accepts labels. An integer is interpreted as a position only when no label is an integer.

Validation

The following conditions are errors. The model is not built, and the message names every affected state and action.

The following conditions produce logged warnings: unreachable_state, rewarding_absorbing_state, reward_on_impossible_transition.

These checks, together with the test suite, verify the stated properties at runtime. They are not formal proofs.

2. History-dependent problems

In these problems, the reward or a constraint depends on the history of the episode, for example "visit A before B" or "refuel at most three times". The user declares events and an automaton. aniate builds the exact product model, and its memory states keep the names given by the automaton.

world  = an.problems.gridworld(rows=5, cols=5, goals=((0, 4),), start=(4, 0))
labels = nmdp.Labels(world).at("r4c4", "A").at("r0c4", "B")
task   = an.NMDP(world, labels, nmdp.Ordering(["A", "B"], violation_penalty=-1))
pi     = task.solve()

Labels(world)

The events of a step s -a-> s2 are the union of on(s, a) and at(s2). No events are read at time 0 or after the episode has ended.

Automata

Every automaton provides .summary(), .describe(), .step(q, events) and .run(trace).

NMDP(world, labels, spec, *, combine="add", violation_terminal=False)

• combine="add" adds the world reward and the automaton output. combine="replace" uses the automaton output alone.
• violation_terminal=True ends the episode when a constraint can no longer be satisfied.

MemoryPolicy

3. Running

Env(model, *, seed=None, max_steps=None, labels=True, hide_memory=False, log=True)

The environment follows the Gym interface and samples from the model.

env = an.problems.gridworld().env(seed=0)
obs, info = env.reset()                                        # or reset(seed=..., state=...)
obs, reward, terminated, truncated, info = env.step("north")   # an action label or position
env.render()        # 't=1  r3c0 --north--> r2c0  reward -0.04  return -0.04'

info contains episode, t, state, action, return (discounted, accumulated so far) and discount (gamma to the power t). History-dependent models add world_state, memory and events. labels=False returns integer observations.

simulate(model, policy=None, episodes=100, *, max_steps=None, seed=0, start=None)

Runs all episodes in parallel and returns an Episodes object.

Accepted policy forms

A policy may be None (uniform random), a Solution, an array of action indices of shape (S,) or (T, S), a probability array of shape (S, A), a dict {state: action}, a function f(state) -> action, a MemoryPolicy, or any object with a step(state) method. The last form can be simulated but not evaluated exactly.

4. Checking a simulator

check(model, env=None, policy=None, *, episodes=300, max_steps=None, seed=0, state=None, action=None, alpha=1e-3)

Runs episodes through a simulator, tests every step against the model, and returns a CheckReport.

• env= accepts any object with the Gym interface. step may return 5 or 4 values, and reset() may return obs or (obs, info).
• state=lambda obs: label translates observations, and action=lambda position: ... translates actions.
• For a history-dependent model, the simulator observes world states and aniate tracks the memory state with the automaton.

The report provides report.ok, report.issues, report.stats and report.describe().

5. Solving

6. Plots

Plots use black Computer Modern text, the default LaTeX typeface, on a white background, without titles. Every plot accepts ax= and returns the Axes, and no function calls show().

7. Logs

Every operation writes one line to standard error:

aniate | model    | 5 states | 2 actions | gamma 0.9 | horizon inf | 14 transitions | valid
aniate | solve    | policy_iteration | 4 iterations | 2.8 ms | start value 3.20876 | bound 8.9e-15
aniate | simulate | 50% of 10,000 episodes finished by step 5
aniate | simulate | 10,000 episodes | 49,768 steps | 4 ms | mean return 3.23989 +/- 0.014 | 10,000 terminated | 0 truncated
aniate | check    | PASSED | 300 episodes | 1,505 steps | 4 pairs tested | return 3.1863 vs model 3.2088
aniate | env      | episode 20 done | 6 steps | return 3.439 | terminated | mean return 3.12
aniate | vis      | saved overview.pdf

Env logs completed episodes 1 to 10, then every 10th episode up to 100, then every 100th up to 1,000, and so on.

8. Built-in problems

an.problems.gridworld(rows, cols, goals, traps, walls, slip, gamma, start), chain(), river_swim(), inventory().

9. Worked examples

Each folder contains a model module, two tests, a plot.py script that writes PDF files, and a README with the derivation and results.

pytest                         # all tests
python tests/stsp/plot.py      # writes tests/stsp/route.pdf and tests/stsp/overview.pdf

10. Import banner

import aniate prints a drawing to standard error once per process: squares of side 1, 1, 2, 3, 5 and 8 tiling a 13 × 8 rectangle, with the golden spiral drawn through them. python -m aniate, or the aniate command, prints it together with the version.

11. Notebook display

Evaluating one of the following objects as the last expression of a Jupyter cell displays it as an HTML table. The tables use inline styles and inherit the text colour of the notebook, and a table longer than 20 rows states how many rows were omitted.

aniate.notebook.to_html(obj) returns the same markup as a string.

12. Gymnasium adapter: aniate.gym

The adapter requires pip install 'aniate[gym]'.

gym.make(model, *, max_steps=None, hide_memory=True, render_mode=None, log=False)

Returns an AniateEnv, a subclass of gymnasium.Env.

Transitions are sampled from the model with the seeded generator of Gymnasium, so equal seeds give equal episodes. info carries the keys described for Env in section 3, except the episode counter.

gym.register(id, model, **kwargs)

Registers the model with Gymnasium, so that gymnasium.make(id) builds it. model is a model, or a function without arguments that returns one; kwargs are passed to AniateEnv.