lunapi.parallel =============== .. py:module:: lunapi.parallel .. autoapi-nested-parse:: Process-based project execution helpers for lunapi. Attributes ---------- .. autoapisummary:: lunapi.parallel.ParallelProcResult lunapi.parallel.ParallelProcError Exceptions ---------- .. autoapisummary:: lunapi.parallel.FileOutputModeError lunapi.parallel.ProcError Classes ------- .. autoapisummary:: lunapi.parallel.ProcResult Functions --------- .. autoapisummary:: lunapi.parallel.coerce_strata lunapi.parallel.default_workers lunapi.parallel.clamp_workers lunapi.parallel.tokenize_param_line lunapi.parallel.parse_param_text lunapi.parallel.parse_param_file lunapi.parallel.resolve_params lunapi.parallel.normalize_result_table lunapi.parallel.project_eval_slices lunapi.parallel.normalize_sample_row lunapi.parallel.run_parallel_project lunapi.parallel.list_text_tables lunapi.parallel.read_text_table Module Contents --------------- .. py:exception:: FileOutputModeError Bases: :py:obj:`RuntimeError` Raised when table access is attempted on a file-output-only ProcResult. .. py:class:: ProcResult(_owner=None, errors=None, stdout=None, records=None, workers=1, _out_paths=None, _data=None, tables=None) Result returned by proc(), silent_proc(), proc_parallel(), and procn(). In memory mode (default): table data lives in the C-level cache of *_owner*; all table queries delegate there. In file-output mode (out_db / out_text): *_owner* is None and table data was written directly to disk. Metadata (errors, records, out_paths) is still available; table queries raise FileOutputModeError. .. py:attribute:: errors :type: pandas.DataFrame .. py:attribute:: stdout :type: pandas.DataFrame .. py:attribute:: records :type: pandas.DataFrame .. py:attribute:: workers :type: int :value: 1 .. py:property:: ok :type: bool .. py:property:: tables Provide mapping-style table access and a callable table index. .. py:method:: keys() .. py:method:: items() .. py:method:: values() .. py:method:: get(key, default=None) .. py:method:: table(cmd, strata='BL') Return one result table by Luna command and strata. .. py:method:: strata() Return available command/strata pairs as a DataFrame. .. py:method:: commands() Return available commands as a DataFrame. .. py:method:: has_table(cmd, strata='BL') Return whether a command/strata table is present. .. py:method:: copy() Return a frozen copy of the current results. The returned ``ProcResult`` owns its data independently of the project cache, so subsequent ``proc()`` calls do not affect it. Useful when calling ``proc()`` in a loop:: res = {} for stage in ['N1', 'N2', 'N3', 'R']: proj.var('stage', stage) res[stage] = proj.silent_proc(cmdstr).copy() res['N1']['STATS: CH_STAGE'] res['N1'].table('STATS', 'CH_STAGE') .. py:method:: table_index() Return ``{command: [strata_factor_list, ...]}`` for available tables. .. py:data:: ParallelProcResult .. py:exception:: ProcError(message: str, result: ProcResult) Bases: :py:obj:`RuntimeError` Raised when execution fails with ``strict=True``. .. py:attribute:: result .. py:data:: ParallelProcError .. py:function:: coerce_strata(pairs, cmd, strata) Resolve a list/set strata to the matching string given C-level (cmd, strata) pairs. ``pairs`` is the list of ``(cmd, strata)`` tuples returned by ``eng.strata()`` / ``edf.strata()``. If *strata* is already a string it is returned unchanged. A list or set is matched order-independently against the ``_``-delimited factor tokens in each available strata string. .. py:function:: default_workers(cpu_count=None) -> int .. py:function:: clamp_workers(value, total_records=None, cpu_count=None) -> int .. py:function:: tokenize_param_line(line: str, keep_quotes: bool = True) -> list[str] .. py:function:: parse_param_text(text: str) -> list[tuple[str, str]] .. py:function:: parse_param_file(path) -> list[tuple[str, str]] .. py:function:: resolve_params(params=None, param_file=None) -> list[tuple[str, str]] .. py:function:: normalize_result_table(df, record_id) .. py:function:: project_eval_slices(tasks, workers, batch_size=None) .. py:function:: normalize_sample_row(row) -> list[str] .. py:function:: run_parallel_project(project, cmdstr: str, *, workers=None, batch_size=None, params=None, param_file=None, strict: bool = False, progress=None, out_db=None, out_text=None, in_memory=None, n1=None, n2=None, ids=None, skip=None) -> ParallelProcResult .. py:function:: list_text_tables(path, id=None) -> pandas.DataFrame List available tables in a Luna ``-t`` text-output folder. Inspects one individual subdirectory and returns a summary of available command/strata combinations based on the filenames present. :param path: Root folder passed as ``out_text``. :type path: str or Path :param id: Individual ID (subdirectory name) to inspect. Defaults to the first subdirectory in sorted order. :type id: str, optional :returns: Columns: ``command``, ``strata``, ``file``. ``strata`` is ``'BL'`` for the baseline (no factors) or the factor name(s) joined by ``'_'`` (e.g. ``'CH'``, ``'B_CH'``). :rtype: pd.DataFrame .. py:function:: read_text_table(path, cmd_or_file, factors=None) -> pandas.DataFrame Read a concatenated text-output table from a Luna ``-t`` directory. Finds every per-individual file that matches the requested command/strata combination and concatenates them into a single DataFrame — equivalent to:: awk 'NR==1 || FNR!=1' path/*/COMMAND_FACTOR.txt Factor ordering in the filename is handled automatically; both ``.txt`` and ``.txt.gz`` files are supported. :param path: Root folder passed as ``out_text``. :type path: str or Path :param cmd_or_file: One of: * A bare command name: ``'HEADERS'`` (baseline strata) * A filename: ``'HEADERS_CH.txt'`` or ``'HEADERS_CH.txt.gz'`` * A sequence: ``('HEADERS', 'CH')`` or ``['HEADERS', ['B','CH']]`` :type cmd_or_file: str, tuple, or list :param factors: Factor(s) when *cmd_or_file* is a plain command name, e.g. ``factors='CH'`` or ``factors=['B', 'CH']``. :type factors: str or list of str, optional :rtype: pd.DataFrame