Operations relaySystems online
Work/02
Dependency-aware execution

Async Workflow Orchestrator

A DAG workflow engine with validated dependencies, concurrent async execution, plugin tasks, resumable state, and run-level observability.

EvidencePublic repository
CategoryOrchestration
RoleArchitecture → delivery
System routeRead left to right
  1. 01Workflow spec
  2. 02DAG validate
  3. 03Schedule
  4. 04Execute
  5. 05Resume
01 / Context and problem

The operational problem

Multi-step jobs need more than a task list: dependencies must be valid, failures must propagate clearly, and interrupted work must remain inspectable.

Ownership

Built the graph validator, execution manager, plugin protocol, persistent task state, FastAPI control plane, YAML examples, CLI, JSONL logging, tests, CI, and Docker runtime.

02 / Technical challenge

What made it non-trivial

Concurrency, retries, timeouts, cancellation, dependency failure, and resume behavior had to produce a coherent, persistent task history.

SQLite is the default local state store.

Built-in plugins are deliberately small demonstration tasks.

The public project does not claim a hosted production service.

03 / System design

The architecture decision

The engine validates the graph before execution, schedules topological levels under a bounded semaphore, resolves typed task plugins, persists every transition, and exposes trigger, inspect, cancel, resume, and graph endpoints.

01Python02asyncio03FastAPI04SQLAlchemy05Pydantic06YAML07Pytest08Docker
04 / Reliability

Designed for the failure path

  • 01

    Cycle and missing-dependency validation

  • 02

    Bounded concurrent execution

  • 03

    Retries and per-task timeout limits

  • 04

    Cancellation tokens and dependency-aware skipping

  • 05

    Persistent workflow and task state

  • 06

    Per-run JSONL operational logs

05 / Result and evidence

What this system proves

A compact, installable workflow engine that demonstrates graph reasoning, stateful async execution, API design, packaging, and strict CI quality gates.

Tayyab can design orchestration behavior where state transitions and failure semantics are first-class concerns.
06 / Lessons

What carries forward

01

Validate the complete graph before any side effect starts.

02

Persist state around task execution so interruption is observable.

03

Plugin boundaries keep the scheduler independent from task implementation.

Next system · 03API Integration Hub