Older SciExp² versions

1.1.13

External compatibility breaks

New features

Improvements

Bug fixes

  • Allow sciexp2.common.utils.find_files (and related) to search for directories (paths ending with “/”).

  • Fix sciexp2.data.io.find_files when using a job descriptor file.

Documentation

  • Fix documentation of sciexp2.data.io.find_files.

  • Improve documentation of sciexp2.data.io.extract_txt and sciexp2.data.io.extract_regex regarding variable references and regular expressions.

Internals

  • Properly account progress indication in iterables.

  • Fix sciexp2.common.instance.InstanceGroup.dump to only have the exported variables. Indirectly affects sciexp2.data.io.find_files when used with a job descriptor file.

  • Improve Python 3 compatibility.

1.1.12

External compatibility breaks

New features

Improvements

  • Speedup construction of sciexp2.data.Dim objects by two orders of magnitude.

  • Forward SIGINT and SIGTERM to user-specified commands in job scripts.

  • Forward SIGINT to executing jobs with the shell backend (closes #293).

  • Forward SIGINT to executing commands with sciexp2.launchgen.Launchgen.execute (closes #293).

Bug fixes

Documentation

Internals

1.1.11

External compatibility breaks

New features

Improvements

Bug fixes

  • Fix progress message logging.

  • Fix length calculation of progress operations.

  • Export relevant variables to user-provided job commands.

Documentation

Internals

1.1.10

External compatibility breaks

  • Remove default_launchgen and its methods from module sciexp2.launchgen.env (you should explicitly instantiate sciexp2.launchgen.Launcher instead).

New features

Improvements

  • Use package tqdm to show fancier progress indicators.

  • Detect when we’re running in a IPython/Jupyter notebook and use proper progress widgets when available.

  • Make sure the output of the user’s commands in launchgen’s default templates is properly seen during interactive execution.

  • Add method sciexp2.data.Data.sort (closes #307).

Bug fixes

  • Fix construction of Launcher objects from other objects of the same type.

  • Fix handling of argument append in sciexp2.launchgen.Launcher.params when applied to an empty object.

  • Support sciexp2.data.Data dimensions with missing elements and elements with missing variable values.

Documentation

  • Extend and clarify quickstart, installation and basic concepts.

  • Extend and clarify the user guide for launchgen.

  • Extend and clarify the installation and execution guide.

Internals

  • Remove unused progress indicator sciexp2.common.progress.LVL_MARK.

1.1.9

External compatibility breaks

New features

Improvements

Bug fixes

  • Fix bug in compatibility code handling argument inspection of functions.

  • Fix compatibility code to detect unused arguments in older numpy versions.

Documentation

Internals

1.1.8

External compatibility breaks

New features

Improvements

  • Do not crash progress reports when running on an IPython notebook.

Bug fixes

  • Fix expression and instance construction in sciexp2.data.Data.flatten.

  • Fix indexing when using a filter on the last indexed dimension.

  • Fix advanced indexing with boolean arrays.

Documentation

Internals

1.1.7

External compatibility breaks

New features

Improvements

Bug fixes

  • Fix dimension indexing in sciexp2.data.DataDims.

Documentation

  • Improve introduction.

Internals

1.1.6

External compatibility breaks

New features

Improvements

Bug fixes

  • Fix parsing of the keepdims argument in sciexp2.data.wrap_reduce for newer numpy versions (which affects all reduction operations).

  • Fix setuptools dependency on Python (again).

Documentation

Internals

1.1.5

External compatibility breaks

  • Changed semantics of callables in sciexp2.data.meta.Dim.sort to be compatible with sort in Python 3.

New features

Improvements

Bug fixes

  • Properly fail launcher when an experiment errors-out and we’re in DEBUG log level.

  • Fix dependency timestamp computation in sciexp2.data.io.lazy and friends with Python 3.

Documentation

Internals

  • Remove cmp argument on calls to sort to be compatible with Python 3.

  • Improve compatibility with Python 3.

  • Always show a stable element order when pretty-printing instances.

1.1.4

External compatibility breaks

New features

Improvements

Bug fixes

  • Fix setuptools dependency on Python.

Documentation

Internals

1.1.3

External compatibility breaks

New features

  • Add method sciexp2.data.DataIndexer.as_tuple.

Improvements

  • Improve output file initialization in the shell template.

  • Ignore empty variable values in argument DEPENDS of sciexp2.launchgen.Launchgen.launcher (closes #298).

  • Do not warn when sciexp2.data.io.lazy receives only non-lazy arguments.

Bug fixes

  • Fix boolean negation operator in Filter.

  • Fix ravel.

Documentation

Internals

1.1.2

External compatibility breaks

New features

Improvements

  • Allow setting Dim ticks to None.

  • Remove most limitations of assignments to sciexp2.data.meta.Dim.expression and sciexp2.data.meta.Dim contents.

  • Generalize the following functions to work with any type of arrays: sciexp2.data.append, sciexp2.data.concatenate, sciexp2.data.copy, sciexp2.data.delete, sciexp2.data.drop_fields, sciexp2.data.imag, numpy.lib.recfunctions.merge_arrays, sciexp2.data.ravel, sciexp2.data.real, numpy.lib.recfunctions.rename_fields.

  • Improve output file initialization in the shell template.

  • Ignore empty variable values in argument DEPENDS of sciexp2.launchgen.Launchgen.launcher (closes #298).

Bug fixes

  • Properly escape sciexp2.data.meta.Dim.expression values to avoid confusing the user’s string with parts of a regular expression.

  • Fix boolean negation operator in Filter.

  • Fix ravel.

Documentation

Internals

  • Fix hard resets on sciexp2.common.instance.InstanceGroup.cache_reset.

  • Fix sciexp2.data.DataDims copies.

  • Implement consistent named axis selection (axis argument in numpy function) when using multiple array arguments.

  • Follow numpy’s exception format when using a non-existing field name.

1.1.1

External compatibility breaks

  • Remove argument filters in sciexp2.launchgen.Launchgen.expand in favour of sciexp2.launchgen.Launchgen.select (closes #300).

  • Deprecate argument export in sciexp2.launchgen.Launchgen.launcher in favour of new value EXPORTS in argument values (closes #301).

New features

Improvements

Bug fixes

  • Fix filter optimization (breaks construction from multiple filters).

  • Allow comparing variables in PFilter with strings (fixes #302, as a refinement of #278).

  • Do not complain when about empty views when using sciexp2.launchgen.Launchgen methods (fixes #296).

Documentation

  • Clarify use cases of files sub-command in launcher.

  • Clarify how variables are substituted in sciexp2.launchgen.Launchgen.params.

Internals

  • Fix representation of strings in Filter objects (was breaking escape sequences, used in regular expressions).

1.1

External compatibility breaks

  • Remove sciexp2.data.Data.dim_sort and dim_sort argument in sciexp2.data.Data.reshape in favour of sciexp2.data.meta.Dim.sort.

  • Remove unused “filter override” operator.

  • Deprecate “add” operation in Filter in favour of “and”.

  • Forbid Data indexing with intermediate results from DataIndexer.

New features

  • Allow sorting sciexp2.data.Data arrays using sciexp2.data.meta.Dim.sort (closes #279).

  • Add filter syntax to check variable existence (exists(VAR); closes #262).

  • Add --inverse argument in launcher to easily invert the job state selection (closes #287).

  • Add sciexp2.data.meta.Dim.values and sciexp2.data.meta.Dim.unique_values to easily retrieve per-variable values (closes #290).

  • Add sciexp2.launchgen.Launchgen.translate and sciexp2.launchgen.Launchgen.expand to translate and expand expressions from instances (closes #276).

  • Add sciexp2.data.Data.idata attribute to allow immediate dimension-oblivious indexing (shortcut to sciexp2.data.Data.indexer; closes #282).

Improvements

  • Auto-optimize filter matching.

  • Using sciexp2.launchgen.LaunchgenView.select_inverse works properly on nested views (only inverts the last selection).

  • Allow sciexp2.launchgen.Launchgen.pack to dereference symbolic links (closes #280).

  • Allow sciexp2.data.Data.indexer to accept PFilter instances (closes #284).

  • Allow arbitrary values to be returned by functions used through sciexp2.data.io.lazy and similar (closes #285).

  • Simplify use of variables in the files sub-command of launcher (closes #281).

  • Allow selecting multiple dimensions in sciexp2.data.DataDims and sciexp2.data.DataIndexer.

Bug fixes

  • Fix bug in string representation for sciexp2.common.filter.Filter.

  • Fix indexing in Dim when using filters as a start and/or stop slice.

  • Fix management of DONE and FAIL files in the shell template.

  • Fix merging of PFilter with strings (closes #278).

  • Fix result of “or” operation in Filter.

  • Fix array element-wise comparison (metadata is ignored for now).

  • Make indexing logic more robust (closes #283).

Documentation

Internals

1.0.2

External compatibility breaks

New features

Improvements

  • Show progress message when building new dimensions in sciexp2.data.Data.reshape.

  • Improve performance of sciexp2.data.Data.reshape.

Bug fixes

  • Fix spurious ignored AttributeError exceptions when using progress.

Documentation

Internals

1.0.1

External compatibility breaks

New features

Improvements

  • Return None in sciexp2.data.meta.Dim.index when dimension is not associated.

Bug fixes

  • Fix sciexp2.data.meta.Dim association when indexing.

Documentation

Internals

1.0

External compatibility breaks

  • Move dimension-related classes to sciexp2.data.meta (with shortened names).

  • sciexp2.data.meta.Dim constructor no longer performs copies, use sciexp2.data.meta.Dim.copy instead.

  • Remove argument copy_dims from Data.

  • Remove methods sciexp2.data.Data.dim and sciexp2.data.Data.dim_index in favour of sciexp2.data.Data.dims and sciexp2.data.meta.Dim.index, respectively.

  • Remove method sciexp2.data.Data.iteritems.

  • Remove attribute sciexp2.data.meta.Dim.contents.

  • Remove deprecated (since 0.18) argument promote in in data extraction routines (sciexp2.data.io).

New features

  • Add sciexp2.data.meta.Dim.instances attribute to access the instances of a dimension.

  • Add sciexp2.data.meta.Dim.translate and sciexp2.data.meta.Dim.extract.

  • Add sciexp2.data.DataDims to query and manipulate collections of dimension metadata objects.

  • Allow Dim objects with missing ticks or empty expression (closes #243).

  • Allow Data objects with empty dimension metadata (closes #242).

  • All views of a Data object have consistent metadata.

  • Allow element and expression assignments to Dim objects (closes #236).

  • Unhandled ndarray methods now return a Data object without metadata.

  • Add indexer to facilitate alignment of indexes to dimensions.

Improvements

  • Export lazy_wrap, lazy_wrap_realize and lazy_wrap_checkpoint through sciexp2.data.env.

  • Return a Data when using newaxis or advanced indexing.

  • Allow axis numpy.ufunc argument with multiple values (closes #274).

  • Let keepdims numpy.ufunc argument return a Data object (closes #275).

  • Return a Data object with empty metadata when broadcasting to a ndarray argument.

Bug fixes

  • Fixed indexing results on sciexp2.data.meta.Dim.instances.

Documentation

  • Add a quick example of all modules in the introduction.

  • Document array and metadata indexing and manipulation in the user guide.

Internals

  • Move free functions for Data objects into sciexp2.data._funcs.

  • Rename sciexp2.data.meta.ExpressionError as DimExpressionError.

  • Refactor dimension expression logic into sciexp2.data.meta.DimExpression.

  • Add progressable_simple to wrap container iterations with a progress indicator.

  • Sanitize sciexp2.data.meta.Dim construction.

  • Remove the EXPRESSION internal variable from dimension metadata, making it smaller at the expense of more complex expression lookups (closes #231).

  • Remove the INDEX internal variable from dimension metadata, making it smaller at the expense of more costly index lookups.

  • Allow constructing views of sciexp2.data.meta.Dim objects.

0.18.2

External compatibility breaks

New features

  • Lazy result realization also works without a file path (see lazy).

  • Add lazy_wrap, lazy_wrap_realize and lazy_wrap_checkpoint to streamline use of lazily evaluated functions.

  • Allow per-variable conversion rules in extract_regex (closes #270).

Improvements

  • Use default value if converter fails in extract_regex.

  • Show an error message (instead of throwing an exception) if job submission fails.

  • Add argument --keep-going to launcher to keep submitting jobs even if others fail.

  • Provide a crude string representation of lazy results (lazy).

Bug fixes

  • Make wrap_reduce more resilient (closes #269).

  • Apply converters in extract_txt for string fields.

  • Fix missing value handling in first line for extract_regex.

  • Apply user conversions on missing values for extract_regex (closes #268).

  • Fix dtype detection when using vars_to_fields (data extraction and reshaping).

  • Remove output file if there is an error during lazy result realization.

Documentation

  • Document converters and default values in extract_txt.

Internals

0.18.1

External compatibility breaks

New features

Improvements

Bug fixes

Documentation

Internals

  • Switch to Git repository.

0.18

External compatibility breaks

  • New argument fields_to_vars in data extraction routines (sciexp2.data.io; deprecates argument promote).

  • Rename argument columns into fields in extract_txt.

  • Rename argument rows into fields in extract_regex.

New features

  • Add generic data extraction routine extract_func (closes #233).

  • Add support for gzip-compressed source files in data extraction routines for io (closes #232).

  • Add function data_frombuffer (closes #194).

  • Add function data_memmap.

  • Add argument fields_to_vars in reshape.

  • Add argument vars_to_fields in data extraction routines (sciexp2.data.io) and reshape (closes #241).

Improvements

  • Add support for “multi-comparisons” in filters (e.g., 1 < a < 3).

  • Allow pattern binding operations with any type of arguments in filters.

  • Add support for lists in filters (e.g., [1, 2, b]).

  • Add support for list membership checks in filters (e.g., a in [1, 3, 5]).

Bug fixes

  • Fix handling of unary minus operator in filters.

  • Fix handling of override operator in filters.

Documentation

  • Improve documentation of routines in sciexp2.data.io.

  • Initial user guide for the sciexp2.data package.

Internals

  • Reimplement extract_txt and extract_regex on top of extract_func.

0.17.1

External compatibility breaks

  • Rename sciexp2.data.Data.sort as dim_sort, since numpy.sort already exists (closes #244).

  • Rename argument order in reshape as dim_sort to keep naming consistent.

New features

Improvements

  • Lazy evaluation with sciexp2.data.io.lazy detects changes to the source code of functions passed as arguments, triggering a re-computation.

Bug fixes

  • Fix command line argument parsing of launcher when using a job descriptor file as a binary.

  • Fix reductions (wrap_reduce) when the result has no dimensions (e.g., a single number).

  • Fix indexing of Data objects when using the old numeric-compatible basic slicing [1].

Documentation

Internals

0.17

External compatibility breaks

  • Slicing a Dimension always returns a new one (or None).

New features

  • Added dimension slicing helpers indexes and ticks.

  • Functions that have an axis argument can now identify a dimension by giving the name of one of its variables (closes #200).

Improvements

  • Raise exception ConverterError when the user-provided conversion functions in extract_regex fail.

  • Slicing in Data accepts anything that can be converted into a filter (e.g., dict; closes #235).

  • Slicing in Data also accepts anything that can be converted into a filter (e.g., dict) as long as it selects one element (closes #230).

  • Fixed dimension metadata when performing binary operations with Data objects (closes #54).

Bug fixes

  • Raise EmptyFileError when probing a file without matches during extract_regex.

  • Do not fail when using extract_txt on files with one single column and row (closes #238).

  • Properly handle genfromtxt-specific arguments in extract_txt (closes #239).

Documentation

Internals

  • Can pretty-print Dimension instances.

  • Provide copy constructor of Dimension instances.

  • Provide public wrap_reduce and wrap_accumulate methods to wrap existing numpy functions.

0.16

External compatibility breaks

  • Command launcher now has a saner command and option syntax.

  • New syntax for template descriptor files.

  • Remove commands monitor and reset from launcher.

  • Removed variable sciexp2.common.progress.SHOW in favour of routine level.

  • Changed the syntax of execute (it’s compatible with the common single-string argument).

New features

  • Add commands summary, variables and files to launcher.

  • Template descriptors can refer to some of the variables defined by their parent.

  • Template descriptors, launcher and launcher can define additional job submission arguments.

  • Program launcher can define additional job killing arguments.

  • Add simple begin/end progress indicator (sciexp2.common.progress.LVL_MARK).

  • Add file_contents to simplify inserting the contents of a file as the value of a variable.

  • Add support for parallel command execution in execute (closes #170).

Improvements

  • Can now run launcher scripts even if the execution system is not installed (assumes no job is currently running).

  • Improved error resilience in template scripts.

  • All file-generation routines in launchgen will retain the permission bits of their source file.

  • Be clever about interactive terminals when showing progress indicators.

  • User can set the desired progress reporting level when using launcher.

  • Program launcher now explicitly shows outdated jobs.

Bug fixes

  • Fix error when using select with a dict-based filter that contains a non-existing value.

  • Fix path computation of STDOUT and STDERR files in the gridengine template.

  • Properly handle operations through a LaunchgenView.

  • Allow creating a Launchgen from a LaunchgenView (closes #228).

  • Fix creation of a Launchgen from a Launchgen or LaunchgenView.

Documentation

  • Point out availability of examples’ input files on the source distribution.

Internals

  • Job submission no longer removes results; instead, job scripts do it themselves.

  • Do not treat template variables DONE and FAIL as glob patterns.

  • New module parallel providing simple parallelization primitives.

0.15.4

External compatibility breaks

New features

Improvements

Bug fixes

Documentation

Internals

0.15.3

External compatibility breaks

New features

Improvements

Bug fixes

  • Fix error during executable file generation in launchgen.

  • Fix test number detection in find_SPEC.

Documentation

Internals

0.15.2

External compatibility breaks

New features

Improvements

Bug fixes

  • Add missing package declaration.

Documentation

Internals

0.15.1

External compatibility breaks

New features

  • Add sciexp2.data.Data.imag, sciexp2.data.imag, sciexp2.data.Data.real and sciexp2.data.real.

Improvements

  • Allow building new Data instances by copying metadata from others.

  • Any unimplemented method in Data falls back to a numpy.ndarray and (by default) issues a warning (see WARN_UPCAST).

  • Add sciexp2.data.Data.copy and sciexp2.data.copy.

  • Add sciexp2.data.Data.ravel.

Bug fixes

Documentation

Internals

  • Provide sciexp2.data.Dimension.copy.

0.15

External compatibility breaks

  • Remove sciexp2.data.io.maybe and sciexp2.data.io.maybe_other in favour of lazy.

  • Removed sort in reshape in favour of order using the same semantics as sort.

New features

  • Simpler lazy data extraction and management infrastructure with lazy.

  • Allow sorting data dimensions with sort (closes #198).

  • Added concatenate (closes #193).

  • Added append (closes #50).

  • Added append_fields (closes #215).

  • Added append_fields, drop_fields, rename_fields and merge_arrays (closes #215).

  • Added transpose (closes #204).

  • Added flatten and ravel.

  • Added delete.

  • Added support for multi-line regular expressions in extract_regex (closes #206).

Improvements

  • Detect argument changes in results produced by lazy to force re-execution.

  • Allow lists of filters as arguments to find_files, and to all the extraction routines by extension (closes #209).

  • Allow data extraction routines to take a single input file (closes #210).

  • Properly handle immediate promote string in extract_txt and extract_regex.

  • Support both promote and count in extract_txt and extract_regex (closes #203).

  • Allow passing some arguments in extract_txt down to genfromtxt (closes #211).

Bug fixes

  • Make scripts generated by launcher executable.

  • Ensure data_array uses the appropriate dtype.

  • Fix handling of Ellipsis in data (closes #213).

  • Fix handling of data indexing with multiple filters (closes #208).

  • Fix data extraction when all fields have the same type (closes #205 and #225).

  • Fix descriptor parsing in extract_txt (closes #212 and #223).

Documentation

Internals

0.14.2

External compatibility breaks

New features

Improvements

Bug fixes

  • Fixed internal error in execute.

Documentation

Internals

0.14.1

External compatibility breaks

New features

Improvements

Bug fixes

  • Fixed internal error in find_files.

Documentation

Internals

0.14

External compatibility breaks

  • Removed prefixed underscore from user-visible variables LAUNCHER, DONE and FAIL generated by Launchgen (closes #216).

  • Removed done_expr and fail_expr arguments to launcher in favour of variables DONE and FAIL, which have a default value (closes #217).

  • By default, launcher generates the job descriptor in file jobs.jd (controlled through variable JD).

  • If specified, argument export in launcher overrides the variables that are exported by default.

New features

  • Method execute can now specify stdin, stdout and stderr (closes #168).

  • Program launcher accepts the values (or the relative path to them, when interpreted as files) of certain variables (e.g., DONE) as a short-hand to filters (closes #182).

  • Method launcher accepts a list of variables that launcher will use to establish whether a job needs reexecution (closes #175).

  • Method params can also append new contents instead of recombining them with the existing ones (closes #202).

  • Function find_files accepts job descriptor files as arguments.

Improvements

  • The user can now easily set when the gridengine template should send a mail notification (closes #126).

  • Properly handle stdout and stderr redirection in the gridengine and shell templates (closes #180).

  • Default templates can have separate values for files holding stdout/stderr (variables STDOUT and STDERR) and the DONE and FAIL files.

  • Generating or copying files with Launchgen will only update these when new contents are available (closes #174).

Bug fixes

  • Mark jobs as failed whenever the _FAIL file exists (closes #163).

  • Fix handling of job descriptor files in directories other than the output base.

  • Fixed gridengine template to establish the base directory (closes #176).

Documentation

Internals

  • Method launcher only exports the appropriate variables.

  • Method launcher makes job launcher scripts executable.

  • Added get_path to handle path expansions (used in Launchgen).

  • New implementation of the pretty-printing module pp (adds IPython as a dependency).

  • Store some metadata in job descriptor files to ensure their formatting.

0.13

External compatibility breaks

  • Variable sciexp2.launchgen.Launchgen.DEFAULT_OUT has been renamed to OUTPUT_DIR.

New features

  • Implemented the modulus operation in filters.

  • Added programmatic filters to streamline the filter writing (see PFilter; relates to #185).

  • Instances of Launchgen can be constructed with initial contents (including copies of other instances).

  • Method generate now accepts filters.

  • Added method select to return an object that operates on a subset of the contents (closes #184 and #186).

Improvements

Bug fixes

  • All methods in Launchgen performing parameter recombination accept any iterable structure (closes #164).

Documentation

  • Rewrote the user guide for launchgen, which is now more concise and tangible, as well as describes the latest features.

Internals

  • Have Filter accept a list of arguments.

  • Have match silently fail when it contains a variable not present in the source if argument allow_unknown is set to True.

0.12

External compatibility breaks

  • Dimension no longer handles named groups if the expression argument is a regular expression, as version 0.11.2 removed the feature from the io module.

  • Removed function sciexp2.data.build_dimension and method sciexp2.data.Dimension.build_instance in favour of a saner Dimension constructor and methods add and extend.

New features

  • Progress indicators now try to avoid updating the screen too often. Speed can be controlled through sciexp2.common.progress.SPEED.

  • Whether to show progress indicators on the screen can be globally controlled through sciexp2.common.progress.SHOW.

  • Add support for sciexp2.data.Data.reshape to sort axes according to their variables.

Improvements

  • Improvement of orders of magnitude on the speed of creation of new Data objects (thanks to optimized Dimension construction).

  • Improvement of orders of magnitude on the speed of reshape (thanks to optimized Dimension construction and improved algorithm).

  • Better progress indication in reshape and Data slicing.

Bug fixes

  • Fix sorting of results for file-finding routines.

Documentation

  • Reference documentation no longer shows class hierarchies.

Internals

  • Refactored progress indicators into the progress module.

  • Use context manager protocol with Counter and Spinner.

  • Progress indicator type (counter or spinner) can be automatically selected through sciexp2.common.progress.get and sciexp2.common.progress.get_pickle.

  • Split InstanceGroup caching into cache_get and cache_set.

  • Added proactive and zero-caching instance additions in InstanceGroup.

  • Small performance improvements on various operations of the instance module.

  • Move sciexp2.common.instance.Expander.Result into ExpanderResult.

  • Added progressable as a means to add progress indication in routines that where oblivious to it (e.g., adding it to an instance group to get progress indication when used with an expander).

  • Huge speedup in sort by using Python’s sorted() routine.

  • Add support for progress indicators in sort.

0.11.2

External compatibility breaks

  • Extraction routines in io do not retain the complete source expression as the dimension expression (now it just contains the variables).

New features

Improvements

  • Extraction routines in io ignore empty files by default.

  • Added expr_to_regexp to handle expression-to-regexp conversions in a single place.

  • Added expr_get_vars and regexp_get_names to handle variable/group name extraction in a single place (closes #195).

  • Failed translations show the offending substitution (closes #188).

Bug fixes

Documentation

0.11.1

External compatibility breaks

New features

  • Added “pattern binding” operators to Filter, so that filters can express matches with regular expressions.

Improvements

  • Ensure that maybe and maybe_other applied to extraction routines preserve the path used to find files as the dimension expression.

  • Properly handle argmax and argmin reductions.

  • Properly handle cumsum and cumprod accumulations.

Bug fixes

  • Handle indexing of Data objects with boolean arrays.

  • Properly handle the axis argument in reductions when not explicitly named.

  • Properly translate named regular expression groups into variable references in dimension expressions. Integrates with complex path expressions given to extraction routines as an implicit argument to find_files.

Documentation

0.11

External compatibility breaks

  • Removed top-level scripts launchgen and plotter (closes #119).

    Script launcher now is able to show the list and contents of templates (instead of the now removed launchgen).

    The old functionality of injecting a default instance and its methods is now available (and documented) in the sciexp2.launchgen.env module.

  • Removed module sciexp2.common.cmdline.

    Was used by the top-level scripts, of which only one is present now.

  • Removed modules sciexp2.common.config and sciexp2.common.doc (closes #118).

    Was used by the --describe argument of top-level scripts, which no longer exists.

  • Removed sciexp2.common.utils.check_module.

    All checks are already performed by the requirements stated in the setup.py script.

New features

  • Added initial unit-testing framework. Can be run with python ./setup test. See python ./setup test --help for additional arguments.

  • Added module sciexp2.data.env to quickly import all relevant functions and classes.

Improvements

  • Depend on Python 2.7 or later (closes #43).

    This implies that sciexp2.common.utils.OrderedDict has been removed in favour of Python’s OrderedDict, and launcher now uses argparse instead of optparse.

Bug fixes

Documentation

  • Showing the contents of a template now lists the variables that must be forcefully defined by the user.

  • Properly document how to install and run using “virtualenv” and “pip” (closes #178).

0.10

External compatibility breaks

  • Moved data extraction methods in sciexp2.data.io.RawData into routines in sciexp2.data.io (extract_txt and extract_regex).

  • Re-implemented data extraction routines (extract_txt and extract_regex) to provide a saner, simpler and more unified interface (closes #103).

  • Removed the bit-rotten “valuename” implementation from the Data object, which also helps decreasing the number of concepts (closes #192).

  • Removed sciexp2.data.load and sciexp2.data.io.extract_maybe in favour of maybe and maybe_other.

  • Removed bit-rotten module sciexp2.data.save.

  • Remove sciexp2.data.io.Source in favour of find_files.

    All data extraction utilities in sciexp2.data.io can now accept either an InstanceGroup (resulting from a call to find_files), a tuple with the arguments for find_files or simply the file expression (thus without filters) as the first argument for find_files.

  • Remove sciexp2.data.io.RawData and sciexp2.data.Data.from_rawdata in favour of extraction routines that directly return a Data object (closes #122).

New features

  • Instances of Data can be built directly by the user (see Data and data_array; closes #51 and #65).

  • Added maybe and maybe_other to simplify the process of caching the initial extraction and transformation of data (closes #177).

Improvements

  • Data extraction routines can also work with file-like objects (aside from open existing files).

  • Routine extract_regex can now perform multiple extracts per file.

  • Routine reshape now lets the user specify per-field filling values for newly-generated entries (closes #55).

Bug fixes

Documentation

0.9.7

External compatibility breaks

New features

  • Added the possibility to filter which files to process with pack.

Improvements

Bug fixes

  • Fix a strange bug where find_files returned no results even though there were files to be found.

Documentation

0.9.6

External compatibility breaks

New features

Improvements

  • Return a Data object when slicing with numpy-derived arrays.

Bug fixes

  • Allow superclasses when specifying indexing methods with sciexp2.data.with_dim_index (closes #92).

  • Allow superclasses when specifying indexing methods with sciexp2.data.with_new_dim_index.

  • Return a Data object with the appropriate metadata when using reduction-like numpy routines; if the result has no meaning as a Data object, a numpy.ndarray is returned instead.

  • Fix import path to ipython’s IPython.core.ultratb module.

  • Fix a couple of typos in variable names when building Dimension objects.

Documentation

  • The todo is temporarily disabled due to a bug in Sphinx.

Improvements

Bug fixes

0.9.5

External compatibility breaks

New features

Improvements

  • Move common initialization code to abstract Job class.

  • Only try to kill jobs in Launcher if they’re in the running state.

Bug fixes

  • Do not use sciexp2.system.Job.state to query job state in generic machinery, but use the “_STATE” variable instead.

  • Rename abstract sciexp2.system.Job.status into sciexp2.system.Job.state (closes #125).

  • Fix cleanup of progress indication stack when (un)pickling contents with progress indication.

Documentation

0.9.4

External compatibility breaks

New features

Improvements

  • Improve memory usage during from_rawdata when using extremely large inputs (closes #66).

  • Improve speed of select (closes #63). This results in improved speeds during reshape.

  • Use a plain dict during reverse lookup in InstaceGroup (closes #120). This was unnecessary and a dict might yield faster lookups.

  • Show a clearer error message when a filter contains unknown variable names during match (closes #123).

Bug fixes

  • Allow calls to sciexp2.launchgen.Launchgen.generate without any extra values.

  • Generate source distribution with description and template data files for launchgen.

Documentation

0.9.3

External compatibility breaks

  • Deleted variables QUEUE_CMD and STDINDIR in launcher’s gridengine template.

  • Job descriptor files for launcher use the .jd suffix instead of .dsc.

New features

  • Added variable QSUB_OPTS in launcher’s gridengine template.

  • Templates can provide their own default values (see launchgen -T templatename).

Improvements

  • When finding files, the same variable can appear more than once (e.g., @v1@-foo-@v2@-bar-@v1@).

  • More robust and faster file finding, including finding files using expressions without variable references.

Bug fixes

Documentation

  • Add some user-oriented summaries on the launcher template headers.

0.9.2

External compatibility breaks

  • Removed selector argument in sciexp2.launchgen.Launchgen.params.

New features

  • Let the user tinker with the SEARCH_PATH of launchgen templates.

  • Let the user tinker with the SEARCH_PATH of execution systems.

Improvements

Bug fixes

Documentation

  • User guide for launchgen.

  • User guide for launcher.

  • Auto-generated API documentation.