lunapi.parallel

Process-based project execution helpers for lunapi.

Attributes

ParallelProcResult
ParallelProcError

Exceptions

FileOutputModeError	Raised when table access is attempted on a file-output-only ProcResult.
ProcError	Raised when execution fails with strict=True.

Classes

ProcResult

Result returned by proc(), silent_proc(), proc_parallel(), and procn().

Functions

coerce_strata(pairs, cmd, strata)	Resolve a list/set strata to the matching string given C-level (cmd, strata) pairs.
default_workers(→ int)
clamp_workers(→ int)
tokenize_param_line(→ list[str])
parse_param_text(→ list[tuple[str, str]])
parse_param_file(→ list[tuple[str, str]])
resolve_params(→ list[tuple[str, str]])
normalize_result_table(df, record_id)
project_eval_slices(tasks, workers[, batch_size])
normalize_sample_row(→ list[str])
run_parallel_project(→ ParallelProcResult)
list_text_tables(→ pandas.DataFrame)	List available tables in a Luna -t text-output folder.
read_text_table(→ pandas.DataFrame)	Read a concatenated text-output table from a Luna -t directory.

Module Contents

exception lunapi.parallel.FileOutputModeError

Bases: RuntimeError

Raised when table access is attempted on a file-output-only ProcResult.

class lunapi.parallel.ProcResult(_owner=None, errors=None, stdout=None, records=None, workers=1, _out_paths=None, _data=None, tables=None)

Result returned by proc(), silent_proc(), proc_parallel(), and procn().

In memory mode (default): table data lives in the C-level cache of _owner; all table queries delegate there.

In file-output mode (out_db / out_text): _owner is None and table data was written directly to disk. Metadata (errors, records, out_paths) is still available; table queries raise FileOutputModeError.

errors: pandas.DataFrame

stdout: pandas.DataFrame

records: pandas.DataFrame

workers: int = 1

property ok: bool

property tables

Provide mapping-style table access and a callable table index.

keys()

items()

values()

get(key, default=None)

table(cmd, strata='BL')

Return one result table by Luna command and strata.

strata()

Return available command/strata pairs as a DataFrame.

commands()

Return available commands as a DataFrame.

has_table(cmd, strata='BL')

Return whether a command/strata table is present.

copy()

Return a frozen copy of the current results.

The returned ProcResult owns its data independently of the project cache, so subsequent proc() calls do not affect it. Useful when calling proc() in a loop:

res = {}
for stage in ['N1', 'N2', 'N3', 'R']:
    proj.var('stage', stage)
    res[stage] = proj.silent_proc(cmdstr).copy()
res['N1']['STATS: CH_STAGE']
res['N1'].table('STATS', 'CH_STAGE')

table_index()

Return {command: [strata_factor_list, ...]} for available tables.

lunapi.parallel.ParallelProcResult

exception lunapi.parallel.ProcError(message: str, result: ProcResult)

Bases: RuntimeError

Raised when execution fails with strict=True.

result

lunapi.parallel.ParallelProcError

lunapi.parallel.coerce_strata(pairs, cmd, strata)

Resolve a list/set strata to the matching string given C-level (cmd, strata) pairs.

pairs is the list of (cmd, strata) tuples returned by eng.strata() / edf.strata(). If strata is already a string it is returned unchanged. A list or set is matched order-independently against the _-delimited factor tokens in each available strata string.

lunapi.parallel.default_workers(cpu_count=None) → int

lunapi.parallel.clamp_workers(value, total_records=None, cpu_count=None) → int

lunapi.parallel.tokenize_param_line(line: str, keep_quotes: bool = True) → list[str]

lunapi.parallel.parse_param_text(text: str) → list[tuple[str, str]]

lunapi.parallel.parse_param_file(path) → list[tuple[str, str]]

lunapi.parallel.resolve_params(params=None, param_file=None) → list[tuple[str, str]]

lunapi.parallel.normalize_result_table(df, record_id)

lunapi.parallel.project_eval_slices(tasks, workers, batch_size=None)

lunapi.parallel.normalize_sample_row(row) → list[str]

lunapi.parallel.run_parallel_project(project, cmdstr: str, *, workers=None, batch_size=None, params=None, param_file=None, strict: bool = False, progress=None, out_db=None, out_text=None, in_memory=None, n1=None, n2=None, ids=None, skip=None) → ParallelProcResult

lunapi.parallel.list_text_tables(path, id=None) → pandas.DataFrame

List available tables in a Luna -t text-output folder.

Inspects one individual subdirectory and returns a summary of available command/strata combinations based on the filenames present.

Parameters:

• path (str or Path) – Root folder passed as out_text.

• id (str, optional) – Individual ID (subdirectory name) to inspect. Defaults to the first subdirectory in sorted order.

Returns:

Columns: command, strata, file. strata is 'BL' for the baseline (no factors) or the factor name(s) joined by '_' (e.g. 'CH', 'B_CH').

Return type:

pd.DataFrame

lunapi.parallel.read_text_table(path, cmd_or_file, factors=None) → pandas.DataFrame

Read a concatenated text-output table from a Luna -t directory.

Finds every per-individual file that matches the requested command/strata combination and concatenates them into a single DataFrame — equivalent to:

awk 'NR==1 || FNR!=1' path/*/COMMAND_FACTOR.txt

Factor ordering in the filename is handled automatically; both .txt and .txt.gz files are supported.

Parameters:

• path (str or Path) – Root folder passed as out_text.

• cmd_or_file (str, tuple, or list) –

  One of:

  • A bare command name: 'HEADERS' (baseline strata)

  • A filename: 'HEADERS_CH.txt' or 'HEADERS_CH.txt.gz'

  • A sequence: ('HEADERS', 'CH') or ['HEADERS', ['B','CH']]

• factors (str or list of str, optional) – Factor(s) when cmd_or_file is a plain command name, e.g. factors='CH' or factors=['B', 'CH'].

Return type:

pd.DataFrame