aniate — Approximate Numerics in Applied Transition Environments

About

[GitHub][PyPI][Guide][Reference][Playground][Cite]

Version 2.1.0 · MIT Licence

aniate is a Python library for finite decision problems: Markov decision processes, as well as non-Markovian problems, whose reward depends on the history of the episode.

A problem is specified by its states, actions, transition probabilities, rewards and discount factor. The specification is validated at construction, and a model is rejected when a probability row does not sum to one, when a terminal state continues to move or to accrue reward, or when its value is unbounded.

The library computes optimal policies together with their exact values, simulates episodes in batches, and compares any external simulator with the model through per-transition chi-square tests and the value martingale. History-dependent objectives are expressed with event labels and a Mealy automaton, and the exact product model is solved.

Keywords: Markov Decision Processes, Non-Markovian Rewards, Dynamic Programming, Monte Carlo Simulation, Simulator Validation.

Installation

Install:

pip install aniate, with numpy and scipy

Plots:

pip install 'aniate[vis]', which adds matplotlib

Gymnasium:

pip install 'aniate[gym]', which adds gymnasium for aniate.gym

Requires:

Python 3.10 or later, numpy 1.24 or later, scipy 1.12 or later

Verify:

from aniate import ant; ant(), or python -m aniate from a shell

ant() prints the installed version of Python and of each dependency, followed by the result of a self-test that builds, solves, simulates and checks a small model. It returns True when aniate is working.

aniate 2.1.0   Approximate Numerics in Applied Transition Environments

python      3.12.4      ok             required, 3.10 or later
numpy       2.1.0       ok             required, 1.24 or later
scipy       1.14.1      ok             required, 1.12 or later
matplotlib  -           not installed  optional, for plots in aniate.vis
pytest      -           not installed  optional, for running the tests
hypothesis  -           not installed  optional, for property-based tests
gymnasium   -           not installed  optional, for the Gymnasium adapter in aniate.gym

self-test   model valid | solved, value 3.20876 | 2,000 episodes simulated | check passed | 13 ms
            plots unavailable; install them with: pip install 'aniate[vis]'

aniate is working.

Quick Start

import aniate as an
from aniate import vis

m = an.from_functions(
    states=range(5), actions=["left", "right"],
    transition=lambda s, a: {max(s - 1, 0): 1.0} if a == "left" else {min(s + 1, 4): 0.8, s: 0.2},
    reward=lambda s, a, s_next: 10.0 if s_next == 4 else -1.0,
    gamma=0.9, terminal=[4], initial=0,
)

sol = m.solve()                                   # optimal policy, its exact value, an error bound
runs = an.simulate(m, sol, episodes=10_000)       # 10,000 episodes, run in parallel
report = an.check(m, policy=sol)                  # compares simulated behaviour with the model
vis.save(vis.overview(runs), "overview.pdf")      # four standard plots in one PDF
m.describe()                                      # the model as a JSON-compatible dict

Each operation writes one line to standard error:

aniate | model    | 5 states | 2 actions | gamma 0.9 | horizon inf | 14 transitions | valid
aniate | solve    | policy_iteration | 4 iterations | 2.8 ms | start value 3.20876 | bound 8.9e-15
aniate | simulate | 10,000 episodes | 49,768 steps | 4 ms | mean return 3.23989 +/- 0.014 | 10,000 terminated | 0 truncated
aniate | check    | PASSED | 300 episodes | 1,505 steps | 4 pairs tested | return 3.1863 vs model 3.2088
aniate | vis      | saved overview.pdf

Four plots of simulated episodes: the return distribution with a normal fit, the running mean return, the return accumulated along sample paths, and the value martingale — A figure produced by `vis.overview(runs)`: the return distribution, the running mean return, the return accumulated along sample episodes, and the value martingale `M_t = G_<t + γ^t V(s_t)`, whose mean equals the model value at every step when the model and the simulation agree.

Contributing

The scope of aniate is being extended. Researchers and engineers who wish to contribute, whether through new solvers, automaton patterns, standard problems or worked examples, are invited to write to kabirmurjani@gmail.com. The conventions that every addition follows are described in Extending aniate.

Validation

Construction fails, and the error names each offending state and action, if any of the following hold:

code	condition
`row_sum`	a row of transition probabilities does not sum to 1
`dead_end`	a state-action pair has no successor
`terminal_not_absorbing`	a terminal state moves to another state or pays a reward
`unbounded_value`	the discount factor is 1 and no terminal state is reachable

an.check(model, env=simulator) runs episodes through an external simulator and reports impossible_transition, transition_frequency, reward_mismatch, termination_mismatch and return_mismatch. These properties are verified by runtime checks and by the test suite; they are not formally proved.

Non-Markovian Rewards

from aniate import nmdp

world  = an.problems.gridworld(rows=5, cols=5, goals=((0, 4),), start=(4, 0))
labels = nmdp.Labels(world).at("r4c4", "A").at("r0c4", "B")
task   = an.NMDP(world, labels, nmdp.Ordering(["A", "B"], violation_penalty=-1))
policy = task.solve()        # a MemoryPolicy: policy.step(state) returns the next action

nmdp.Ordering, nmdp.Budget and nmdp.Deadline cover common patterns, and nmdp.Mealy expresses any finite-memory rule. The complete interface is listed in the reference, and the guide indicates which call suits a given task.

Notebooks and Gymnasium

In Jupyter, a model, solution, set of episodes, check report or non-Markovian task that ends a cell is displayed as a table. The same tables appear in the playground, which runs aniate in the browser without installation.

from aniate import gym

env = gym.make(m)                               # a gymnasium.Env with Discrete spaces
obs, info = env.reset(seed=0)
obs, reward, terminated, truncated, info = env.step(env.action_space.sample())

gym.register("aniate/Corridor-v0", lambda: m)   # gymnasium.make("aniate/Corridor-v0") then builds it

aniate.gym presents any model as a gymnasium.Env with Discrete spaces. Transitions are sampled from the model with the seeded generator of Gymnasium, so equal seeds give equal episodes. The adapter requires pip install 'aniate[gym]'.

–

Releases

2026

14.09 Released aniate 2.0.2 on PyPI, with the reference, the decision guide and three worked examples.

[Worked Examples][All Releases]

Citation

@software{murjani2026aniate,
  author  = {Murjani, Kabir},
  title   = {aniate: Approximate Numerics in Applied Transition Environments},
  year    = {2026},
  version = {2.1.0},
  url     = {https://github.com/Kcbir/aniate}
}

Released under the MIT Licence.