Welcome to Grain’s documentation!

Grain parallelizes your workload across supercomputing clusters, just like Dask, Ray, etc.. Unlike the existing solutions, Grain focuses on one scenario: running external calculations (i.e. binary executables) with defined resource constraints.

Release notes: https://github.com/Contextualist/grain/releases (so far, commit messages provide better information for the features and fixes)

Contents:

• Tutorial
  • Before you begin
  • Delayed function and object
  • Resource binding
  • Local or remote execution
  • Getting real: workers
  • What’s next?
• Best practice
  • Manage sessions with Linux’s screen command
  • Quickly switch between different Grain configs
• API Reference: delayed
  • delayed()
  • DelayedFn
  • Delayed
  • each()
  • delayedval()
  • run()
• Incorporate computationally intensive Python functions (if you must)
  • subprocify()
  • subprocess_pool_scope()
  • BrokenSubprocessError
• Utilities
  • set_numpy_oneline_repr()
  • QueueLimiter()
  • open_nursery_with_capacity()
• Advanced usage
  • Worker autoscaling
  • Context module: plugin system for worker
• Resource: a language for coordination
  • Built-in resources
  • Resource interface
• Specialized worker: bring your own server
  • Do I need a specialized worker?
  • Implement a specialized worker
  • Teach users about your specialized worker
• Connection discovery and Communication
  • Bridge protocol
  • Edge protocol
  • Msgpack schema for head-worker communication

Work in progress:

• Resource: a language for coordination

• FAQ

• Low-level API reference

Indices and tables

• Index

• Module Index

• Search Page