Approximate Numerics in Applied Transition Environments

Guide

This guide shows which part of aniate to use for a given task, how the package is organised, and where new functionality belongs.

0. Verifying the installation

from aniate import ant
ant()                 # or, from a shell: python -m aniate

ant() lists Python, the required dependencies (numpy, scipy) and the optional ones (matplotlib, pytest, hypothesis, gymnasium) with their installed versions. It then builds, solves, simulates and checks a five-state model, and writes a PDF plot when matplotlib is present. The last line is either aniate is working. or a statement that a component failed, and the function returns True or False accordingly.

1. Decision tree

What is known about the problem?
│
├── The transition probabilities and rewards are known
│   │
│   ├── How is the model written?
│   │   ├── as Python functions s, a -> {s2: p} ............ an.from_functions
│   │   ├── as a list of individual transitions ............ an.Builder
│   │   └── as arrays or sparse matrices ................... an.MDP(P, R, gamma)
│   │
│   ├── Does the reward or a constraint depend on history?
│   │   ├── no .............................................. use the MDP directly
│   │   └── yes ............................................. nmdp.Labels + an automaton + an.NMDP
│   │       ├── "A before B before C" ....................... nmdp.Ordering
│   │       ├── "at most k uses of a resource" .............. nmdp.Budget
│   │       ├── "finish within T steps" ..................... nmdp.Deadline
│   │       ├── any other finite-memory rule ................ nmdp.Mealy
│   │       └── several rules at once ....................... pass a list, or nmdp.compose
│   │
│   └── What is needed?
│       ├── the optimal policy .............................. m.solve()
│       ├── the exact value of a given policy ............... an.evaluate(m, policy)
│       ├── values at each time step ........................ an.mdp.values_by_time(m, policy, T)
│       ├── sampled episodes and their statistics ........... an.simulate(m, policy, episodes)
│       ├── one step at a time, Gym style ................... m.env()
│       ├── a Gymnasium environment ......................... aniate.gym.make(m)
│       ├── a table in a notebook ........................... end a Jupyter cell with the object
│       └── a picture ....................................... aniate.vis (section 5)
│
└── Only a simulator exists (a program, a Gym environment, a physical system)
    └── Write down the model believed to describe it, then call
        an.check(model, env=simulator, policy=...)
        A failed check names the state-action pairs whose behaviour disagrees with the model.

2. Defining a model

States and actions may be any hashable labels, for example "ok", 3 or (2, 1). Every public function accepts labels.

3. Solving

A method can be forced with m.solve(method="value_iteration"). Solution.value is always the exact value of Solution.policy, and Solution.bound limits the loss of any state relative to the optimum.

4. Running and checking

Accepted policy forms: None (uniform random), a Solution, an array of action indices (S,) or (T, S), a probability array (S, A), a dict {state: action}, a function f(state) -> action, a MemoryPolicy, or any object with a step(state) method. The last form can be simulated but not evaluated exactly.

5. Plotting

All plots require pip install 'aniate[vis]'. Each accepts ax= and returns the Axes. vis.save(figure_or_ax, "name.pdf") writes the file.

6. Reading errors and check results

7. Package layout

aniate/
├── __init__.py        public names: ant, MDP, NMDP, Builder, from_functions, Env, simulate, check, solve, evaluate
├── selfcheck.py       ant: installation report and self-test
├── __main__.py        python -m aniate
├── art.py             import banner
├── log.py             log lines, verbosity, capture
├── notebook.py        HTML tables for notebooks, used by _repr_html_
├── gym.py             Gymnasium adapter: make, register, AniateEnv
├── mdp/
│   ├── space.py       label <-> index mapping
│   ├── core.py        MDP: storage, lookups, describe
│   ├── builder.py     Builder, from_functions
│   ├── validate.py    construction-time checks and their codes
│   ├── solve.py       evaluate, policy iteration, value iteration, backward induction
│   ├── solution.py    Solution
│   ├── policies.py    conversion of every accepted policy form
│   ├── env.py         Env, simulate, Episodes, martingale
│   ├── check.py       check, CheckReport
│   └── text.py        text summaries
├── nmdp/
│   ├── labels.py      events attached to states and actions
│   ├── machine.py     Mealy automata, compose
│   ├── builders.py    Ordering, Budget, Deadline
│   ├── product.py     product of a world model and an automaton
│   ├── policy.py      MemoryPolicy
│   └── task.py        NMDP
├── problems/          gridworld, chain, river_swim, inventory
└── vis/
    ├── style.py       fonts, colours, save
    ├── stats.py       returns, paths, martingale, convergence, overview
    ├── diagrams.py    graph, automaton
    └── grid.py        grid
tests/
├── test_*.py          unit and property tests for each module
├── stsp/              worked example: stochastic travelling salesman
├── gaussian/          worked example: Gaussian random walk
└── heavy_tail/        worked example: heavy-tailed durations and the value martingale

8. Extending aniate

The public interface is the set of names exported by aniate, aniate.nmdp and aniate.vis. Other modules are internal.

Conventions that every addition follows:

1. Public functions accept state and action labels as well as integer positions.
2. Every operation writes one line through aniate.log.
3. Every result object has a describe() method that returns a JSON-compatible dict.
4. A new feature ships with a test that fails when the feature is broken.

9. Scope

aniate works with finite models whose states can be listed in memory. Transition matrices are stored in sparse form, and the worked examples include a model with 1,011 states. Continuous state spaces must be discretised before use. Training agents with function approximation is outside the scope of the current version.

[View the source on GitHub]