Older SciExp² versions
1.1.13
External compatibility breaks
New features
Improvements
Bug fixes
Allow
sciexp2.common.utils.find_files
(and related) to search for directories (paths ending with “/”).Fix sciexp2.data.io.find_files when using a job descriptor file.
Documentation
Fix documentation of sciexp2.data.io.find_files.
Improve documentation of sciexp2.data.io.extract_txt and sciexp2.data.io.extract_regex regarding variable references and regular expressions.
Internals
Properly account progress indication in iterables.
Fix
sciexp2.common.instance.InstanceGroup.dump
to only have the exported variables. Indirectly affects sciexp2.data.io.find_files when used with a job descriptor file.Improve Python 3 compatibility.
1.1.12
External compatibility breaks
New features
Improvements
Speedup construction of sciexp2.data.Dim objects by two orders of magnitude.
Forward SIGINT and SIGTERM to user-specified commands in job scripts.
Forward SIGINT to executing jobs with the shell backend (closes #293).
Forward SIGINT to executing commands with sciexp2.launchgen.Launchgen.execute (closes #293).
Bug fixes
Documentation
Internals
Properly document
sciexp2.common.utils.OrderedSet.view
andsciexp2.common.instance.InstanceGroup.view
.Add methods
sciexp2.common.utils.OrderedSet.set_view_able
andsciexp2.common.instance.InstanceGroup.set_view_able
.Improve Python 3 compatibility.
1.1.11
External compatibility breaks
New features
Improvements
Bug fixes
Fix progress message logging.
Fix length calculation of progress operations.
Export relevant variables to user-provided job commands.
Documentation
Internals
1.1.10
External compatibility breaks
Remove
default_launchgen
and its methods from module sciexp2.launchgen.env (you should explicitly instantiate sciexp2.launchgen.Launcher instead).
New features
Improvements
Use package
tqdm
to show fancier progress indicators.Detect when we’re running in a IPython/Jupyter notebook and use proper progress widgets when available.
Make sure the output of the user’s commands in launchgen’s default templates is properly seen during interactive execution.
Add method sciexp2.data.Data.sort (closes #307).
Bug fixes
Fix construction of Launcher objects from other objects of the same type.
Fix handling of argument
append
in sciexp2.launchgen.Launcher.params when applied to an empty object.Support sciexp2.data.Data dimensions with missing elements and elements with missing variable values.
Documentation
Extend and clarify quickstart, installation and basic concepts.
Extend and clarify the user guide for launchgen.
Extend and clarify the installation and execution guide.
Internals
Remove unused progress indicator sciexp2.common.progress.LVL_MARK.
1.1.9
External compatibility breaks
New features
Improvements
Bug fixes
Fix bug in compatibility code handling argument inspection of functions.
Fix compatibility code to detect unused arguments in older numpy versions.
Documentation
Internals
1.1.8
External compatibility breaks
New features
Improvements
Do not crash progress reports when running on an IPython notebook.
Bug fixes
Fix expression and instance construction in sciexp2.data.Data.flatten.
Fix indexing when using a filter on the last indexed dimension.
Fix advanced indexing with boolean arrays.
Documentation
Internals
1.1.7
External compatibility breaks
New features
Improvements
Bug fixes
Fix dimension indexing in sciexp2.data.DataDims.
Documentation
Improve introduction.
Internals
1.1.6
External compatibility breaks
New features
Improvements
Bug fixes
Fix parsing of the
keepdims
argument in sciexp2.data.wrap_reduce for newer numpy versions (which affects all reduction operations).Fix setuptools dependency on Python (again).
Documentation
Internals
1.1.5
External compatibility breaks
Changed semantics of callables in sciexp2.data.meta.Dim.sort to be compatible with sort in Python 3.
New features
Improvements
Bug fixes
Properly fail
launcher
when an experiment errors-out and we’re inDEBUG
log level.Fix dependency timestamp computation in sciexp2.data.io.lazy and friends with Python 3.
Documentation
Internals
Remove
cmp
argument on calls to sort to be compatible with Python 3.Improve compatibility with Python 3.
Always show a stable element order when pretty-printing instances.
1.1.4
External compatibility breaks
New features
Improvements
Bug fixes
Fix setuptools dependency on Python.
Documentation
Internals
1.1.3
External compatibility breaks
New features
Add method sciexp2.data.DataIndexer.as_tuple.
Improvements
Improve output file initialization in the shell template.
Ignore empty variable values in argument
DEPENDS
of sciexp2.launchgen.Launchgen.launcher (closes #298).Do not warn when sciexp2.data.io.lazy receives only non-lazy arguments.
Bug fixes
Fix boolean negation operator in
Filter
.Fix ravel.
Documentation
Internals
1.1.2
External compatibility breaks
New features
Improvements
Allow setting Dim ticks to
None
.Remove most limitations of assignments to sciexp2.data.meta.Dim.expression and sciexp2.data.meta.Dim contents.
Generalize the following functions to work with any type of arrays: sciexp2.data.append, sciexp2.data.concatenate, sciexp2.data.copy, sciexp2.data.delete, sciexp2.data.drop_fields, sciexp2.data.imag,
numpy.lib.recfunctions.merge_arrays
, sciexp2.data.ravel, sciexp2.data.real,numpy.lib.recfunctions.rename_fields
.Improve output file initialization in the shell template.
Ignore empty variable values in argument
DEPENDS
of sciexp2.launchgen.Launchgen.launcher (closes #298).
Bug fixes
Properly escape sciexp2.data.meta.Dim.expression values to avoid confusing the user’s string with parts of a regular expression.
Fix boolean negation operator in
Filter
.Fix ravel.
Documentation
Internals
Fix hard resets on
sciexp2.common.instance.InstanceGroup.cache_reset
.Fix sciexp2.data.DataDims copies.
Implement consistent named axis selection (
axis
argument in numpy function) when using multiple array arguments.Follow
numpy
’s exception format when using a non-existing field name.
1.1.1
External compatibility breaks
Remove argument
filters
in sciexp2.launchgen.Launchgen.expand in favour of sciexp2.launchgen.Launchgen.select (closes #300).Deprecate argument
export
in sciexp2.launchgen.Launchgen.launcher in favour of new valueEXPORTS
in argumentvalues
(closes #301).
New features
Improvements
Bug fixes
Fix filter optimization (breaks construction from multiple filters).
Allow comparing variables in PFilter with strings (fixes #302, as a refinement of #278).
Do not complain when about empty views when using sciexp2.launchgen.Launchgen methods (fixes #296).
Documentation
Clarify use cases of
files
sub-command in launcher.Clarify how variables are substituted in sciexp2.launchgen.Launchgen.params.
Internals
Fix representation of strings in
Filter
objects (was breaking escape sequences, used in regular expressions).
1.1
External compatibility breaks
Remove sciexp2.data.Data.dim_sort and
dim_sort
argument in sciexp2.data.Data.reshape in favour of sciexp2.data.meta.Dim.sort.Remove unused “filter override” operator.
Deprecate “add” operation in
Filter
in favour of “and”.Forbid Data indexing with intermediate results from DataIndexer.
New features
Allow sorting sciexp2.data.Data arrays using sciexp2.data.meta.Dim.sort (closes #279).
Add filter syntax to check variable existence (
exists(VAR)
; closes #262).Add
--inverse
argument in launcher to easily invert the job state selection (closes #287).Add sciexp2.data.meta.Dim.values and sciexp2.data.meta.Dim.unique_values to easily retrieve per-variable values (closes #290).
Add sciexp2.launchgen.Launchgen.translate and sciexp2.launchgen.Launchgen.expand to translate and expand expressions from instances (closes #276).
Add sciexp2.data.Data.idata attribute to allow immediate dimension-oblivious indexing (shortcut to sciexp2.data.Data.indexer; closes #282).
Improvements
Auto-optimize filter matching.
Using sciexp2.launchgen.LaunchgenView.select_inverse works properly on nested views (only inverts the last selection).
Allow sciexp2.launchgen.Launchgen.pack to dereference symbolic links (closes #280).
Allow sciexp2.data.Data.indexer to accept PFilter instances (closes #284).
Allow arbitrary values to be returned by functions used through sciexp2.data.io.lazy and similar (closes #285).
Simplify use of variables in the
files
sub-command of launcher (closes #281).Allow selecting multiple dimensions in sciexp2.data.DataDims and sciexp2.data.DataIndexer.
Bug fixes
Fix bug in string representation for
sciexp2.common.filter.Filter
.Fix indexing in Dim when using filters as a start and/or stop slice.
Fix management of
DONE
andFAIL
files in the shell template.Fix merging of PFilter with strings (closes #278).
Fix result of “or” operation in
Filter
.Fix array element-wise comparison (metadata is ignored for now).
Make indexing logic more robust (closes #283).
Documentation
Internals
Add method
sciexp2.common.utils.OrderedSet.copy
.Add methods
sciexp2.common.utils.OrderedSet.sorted
andsciexp2.common.utils.OrderedSet.sort
.Add method
sciexp2.common.instance.InstanceGroup.sorted
.Implement
sciexp2.common.instance.InstanceGroup.sort
as in-place sorting.Auto-optimize simple boolean filter expressions.
Drop method sciexp2.common.filter.Filter.constant.
Provide exception check callback for missing variable references in
select
and expand.Drop argument
allow_unknown
fromsciexp2.common.filter.Filter.match
; handle from clients instead.Never return an
ndarray
in _get_indexes (work around NumPy bug #6564).Allow variables in
find_files
to span more than one directory (closes #288).
1.0.2
External compatibility breaks
New features
Improvements
Show progress message when building new dimensions in sciexp2.data.Data.reshape.
Improve performance of sciexp2.data.Data.reshape.
Bug fixes
Fix spurious ignored AttributeError exceptions when using
progress
.
Documentation
Internals
1.0.1
External compatibility breaks
New features
Improvements
Return
None
in sciexp2.data.meta.Dim.index when dimension is not associated.
Bug fixes
Fix sciexp2.data.meta.Dim association when indexing.
Documentation
Internals
1.0
External compatibility breaks
Move dimension-related classes to sciexp2.data.meta (with shortened names).
sciexp2.data.meta.Dim constructor no longer performs copies, use sciexp2.data.meta.Dim.copy instead.
Remove argument copy_dims from Data.
Remove methods sciexp2.data.Data.dim and sciexp2.data.Data.dim_index in favour of sciexp2.data.Data.dims and sciexp2.data.meta.Dim.index, respectively.
Remove method sciexp2.data.Data.iteritems.
Remove attribute sciexp2.data.meta.Dim.contents.
Remove deprecated (since 0.18) argument promote in in data extraction routines (sciexp2.data.io).
New features
Add sciexp2.data.meta.Dim.instances attribute to access the instances of a dimension.
Add sciexp2.data.meta.Dim.translate and sciexp2.data.meta.Dim.extract.
Add sciexp2.data.DataDims to query and manipulate collections of dimension metadata objects.
Allow Dim objects with missing ticks or empty expression (closes #243).
Allow Data objects with empty dimension metadata (closes #242).
All views of a Data object have consistent metadata.
Allow element and
expression
assignments to Dim objects (closes #236).Unhandled
ndarray
methods now return a Data object without metadata.Add indexer to facilitate alignment of indexes to dimensions.
Improvements
Export lazy_wrap, lazy_wrap_realize and lazy_wrap_checkpoint through sciexp2.data.env.
Return a Data when using
newaxis
or advanced indexing.Allow
axis
numpy.ufunc
argument with multiple values (closes #274).Let
keepdims
numpy.ufunc
argument return a Data object (closes #275).Return a Data object with empty metadata when broadcasting to a
ndarray
argument.
Bug fixes
Fixed indexing results on sciexp2.data.meta.Dim.instances.
Documentation
Add a quick example of all modules in the introduction.
Document array and metadata indexing and manipulation in the user guide.
Internals
Move free functions for Data objects into sciexp2.data._funcs.
Rename sciexp2.data.meta.ExpressionError as DimExpressionError.
Refactor dimension expression logic into sciexp2.data.meta.DimExpression.
Add
progressable_simple
to wrap container iterations with a progress indicator.Sanitize sciexp2.data.meta.Dim construction.
Remove the
EXPRESSION
internal variable from dimension metadata, making it smaller at the expense of more complex expression lookups (closes #231).Remove the
INDEX
internal variable from dimension metadata, making it smaller at the expense of more costly index lookups.Allow constructing views of sciexp2.data.meta.Dim objects.
0.18.2
External compatibility breaks
New features
Lazy result realization also works without a file path (see lazy).
Add lazy_wrap, lazy_wrap_realize and lazy_wrap_checkpoint to streamline use of lazily evaluated functions.
Allow per-variable conversion rules in extract_regex (closes #270).
Improvements
Use default value if converter fails in extract_regex.
Show an error message (instead of throwing an exception) if job submission fails.
Add argument
--keep-going
to launcher to keep submitting jobs even if others fail.Provide a crude string representation of lazy results (lazy).
Bug fixes
Make wrap_reduce more resilient (closes #269).
Apply converters in extract_txt for string fields.
Fix missing value handling in first line for extract_regex.
Apply user conversions on missing values for extract_regex (closes #268).
Fix dtype detection when using
vars_to_fields
(data extraction and reshaping).Remove output file if there is an error during lazy result realization.
Documentation
Document converters and default values in extract_txt.
Internals
0.18.1
External compatibility breaks
New features
Improvements
Bug fixes
Documentation
Internals
Switch to Git repository.
0.18
External compatibility breaks
New argument fields_to_vars in data extraction routines (sciexp2.data.io; deprecates argument promote).
Rename argument columns into fields in extract_txt.
Rename argument rows into fields in extract_regex.
New features
Add generic data extraction routine extract_func (closes #233).
Add support for gzip-compressed source files in data extraction routines for io (closes #232).
Add function data_frombuffer (closes #194).
Add function data_memmap.
Add argument fields_to_vars in reshape.
Add argument vars_to_fields in data extraction routines (sciexp2.data.io) and reshape (closes #241).
Improvements
Add support for “multi-comparisons” in filters (e.g.,
1 < a < 3
).Allow pattern binding operations with any type of arguments in filters.
Add support for lists in filters (e.g.,
[1, 2, b]
).Add support for list membership checks in filters (e.g.,
a in [1, 3, 5]
).
Bug fixes
Fix handling of unary minus operator in filters.
Fix handling of override operator in filters.
Documentation
Improve documentation of routines in sciexp2.data.io.
Initial user guide for the sciexp2.data package.
Internals
Reimplement extract_txt and extract_regex on top of extract_func.
0.17.1
External compatibility breaks
Rename sciexp2.data.Data.sort as dim_sort, since
numpy.sort
already exists (closes #244).Rename argument order in reshape as dim_sort to keep naming consistent.
New features
Improvements
Lazy evaluation with sciexp2.data.io.lazy detects changes to the source code of functions passed as arguments, triggering a re-computation.
Bug fixes
Fix command line argument parsing of launcher when using a job descriptor file as a binary.
Fix reductions (wrap_reduce) when the result has no dimensions (e.g., a single number).
Fix indexing of Data objects when using the old numeric-compatible basic slicing [1].
Documentation
Internals
0.17
External compatibility breaks
Slicing a Dimension always returns a new one (or None).
New features
Added dimension slicing helpers indexes and ticks.
Functions that have an axis argument can now identify a dimension by giving the name of one of its variables (closes #200).
Improvements
Raise exception ConverterError when the user-provided conversion functions in extract_regex fail.
Slicing in Data accepts anything that can be converted into a filter (e.g., dict; closes #235).
Slicing in Data also accepts anything that can be converted into a filter (e.g., dict) as long as it selects one element (closes #230).
Fixed dimension metadata when performing binary operations with Data objects (closes #54).
Bug fixes
Raise EmptyFileError when probing a file without matches during extract_regex.
Do not fail when using extract_txt on files with one single column and row (closes #238).
Properly handle
genfromtxt
-specific arguments in extract_txt (closes #239).
Documentation
Internals
Can pretty-print Dimension instances.
Provide copy constructor of Dimension instances.
Provide public wrap_reduce and wrap_accumulate methods to wrap existing numpy functions.
0.16
External compatibility breaks
Command launcher now has a saner command and option syntax.
New syntax for template descriptor files.
Remove commands monitor and reset from launcher.
Removed variable sciexp2.common.progress.SHOW in favour of routine
level
.Changed the syntax of execute (it’s compatible with the common single-string argument).
New features
Add commands summary, variables and files to launcher.
Template descriptors can refer to some of the variables defined by their parent.
Template descriptors, launcher and launcher can define additional job submission arguments.
Program launcher can define additional job killing arguments.
Add simple begin/end progress indicator (sciexp2.common.progress.LVL_MARK).
Add file_contents to simplify inserting the contents of a file as the value of a variable.
Add support for parallel command execution in execute (closes #170).
Improvements
Can now run launcher scripts even if the execution system is not installed (assumes no job is currently running).
Improved error resilience in template scripts.
All file-generation routines in launchgen will retain the permission bits of their source file.
Be clever about interactive terminals when showing progress indicators.
User can set the desired progress reporting level when using launcher.
Program launcher now explicitly shows outdated jobs.
Bug fixes
Fix error when using
select
with a dict-based filter that contains a non-existing value.Fix path computation of
STDOUT
andSTDERR
files in the gridengine template.Properly handle operations through a LaunchgenView.
Allow creating a Launchgen from a LaunchgenView (closes #228).
Fix creation of a Launchgen from a Launchgen or LaunchgenView.
Documentation
Point out availability of examples’ input files on the source distribution.
Internals
Job submission no longer removes results; instead, job scripts do it themselves.
Do not treat template variables
DONE
andFAIL
as glob patterns.New module parallel providing simple parallelization primitives.
0.15.4
External compatibility breaks
New features
Improvements
Bug fixes
Fix data loss bug in
OrderedSet
.
Documentation
Internals
0.15.3
External compatibility breaks
New features
Improvements
Bug fixes
Fix error during executable file generation in launchgen.
Fix test number detection in find_SPEC.
Documentation
Internals
0.15.2
External compatibility breaks
New features
Improvements
Bug fixes
Add missing package declaration.
Documentation
Internals
0.15.1
External compatibility breaks
New features
Add sciexp2.data.Data.imag, sciexp2.data.imag, sciexp2.data.Data.real and sciexp2.data.real.
Improvements
Allow building new Data instances by copying metadata from others.
Any unimplemented method in Data falls back to a
numpy.ndarray
and (by default) issues a warning (see WARN_UPCAST).Add sciexp2.data.Data.copy and sciexp2.data.copy.
Add sciexp2.data.Data.ravel.
Bug fixes
Documentation
Internals
Provide sciexp2.data.Dimension.copy.
0.15
External compatibility breaks
Remove sciexp2.data.io.maybe and sciexp2.data.io.maybe_other in favour of lazy.
Removed sort in reshape in favour of order using the same semantics as sort.
New features
Simpler lazy data extraction and management infrastructure with lazy.
Allow sorting data dimensions with sort (closes #198).
Added concatenate (closes #193).
Added append (closes #50).
Added append_fields (closes #215).
Added append_fields, drop_fields, rename_fields and merge_arrays (closes #215).
Added transpose (closes #204).
Added flatten and ravel.
Added delete.
Added support for multi-line regular expressions in extract_regex (closes #206).
Improvements
Detect argument changes in results produced by lazy to force re-execution.
Allow lists of filters as arguments to find_files, and to all the extraction routines by extension (closes #209).
Allow data extraction routines to take a single input file (closes #210).
Properly handle immediate promote string in extract_txt and extract_regex.
Support both promote and count in extract_txt and extract_regex (closes #203).
Allow passing some arguments in extract_txt down to
genfromtxt
(closes #211).
Bug fixes
Make scripts generated by launcher executable.
Ensure data_array uses the appropriate dtype.
Fix handling of Ellipsis in data (closes #213).
Fix handling of data indexing with multiple filters (closes #208).
Fix data extraction when all fields have the same type (closes #205 and #225).
Fix descriptor parsing in extract_txt (closes #212 and #223).
Documentation
Internals
0.14.2
External compatibility breaks
New features
Improvements
Bug fixes
Fixed internal error in execute.
Documentation
Internals
0.14.1
External compatibility breaks
New features
Improvements
Bug fixes
Fixed internal error in find_files.
Documentation
Internals
0.14
External compatibility breaks
Removed prefixed underscore from user-visible variables LAUNCHER, DONE and FAIL generated by Launchgen (closes #216).
Removed done_expr and fail_expr arguments to launcher in favour of variables DONE and FAIL, which have a default value (closes #217).
By default, launcher generates the job descriptor in file
jobs.jd
(controlled through variable JD).If specified, argument export in launcher overrides the variables that are exported by default.
New features
Method execute can now specify stdin, stdout and stderr (closes #168).
Program launcher accepts the values (or the relative path to them, when interpreted as files) of certain variables (e.g., DONE) as a short-hand to filters (closes #182).
Method launcher accepts a list of variables that launcher will use to establish whether a job needs reexecution (closes #175).
Method params can also append new contents instead of recombining them with the existing ones (closes #202).
Function find_files accepts job descriptor files as arguments.
Improvements
The user can now easily set when the gridengine template should send a mail notification (closes #126).
Properly handle stdout and stderr redirection in the gridengine and shell templates (closes #180).
Default templates can have separate values for files holding stdout/stderr (variables STDOUT and STDERR) and the DONE and FAIL files.
Generating or copying files with Launchgen will only update these when new contents are available (closes #174).
Bug fixes
Mark jobs as failed whenever the _FAIL file exists (closes #163).
Fix handling of job descriptor files in directories other than the output base.
Fixed gridengine template to establish the base directory (closes #176).
Documentation
Internals
Method launcher only exports the appropriate variables.
Method launcher makes job launcher scripts executable.
Added
get_path
to handle path expansions (used in Launchgen).New implementation of the pretty-printing module
pp
(adds IPython as a dependency).Store some metadata in job descriptor files to ensure their formatting.
0.13
External compatibility breaks
Variable sciexp2.launchgen.Launchgen.DEFAULT_OUT has been renamed to OUTPUT_DIR.
New features
Implemented the modulus operation in filters.
Added programmatic filters to streamline the filter writing (see PFilter; relates to #185).
Instances of Launchgen can be constructed with initial contents (including copies of other instances).
Method generate now accepts filters.
Added method select to return an object that operates on a subset of the contents (closes #184 and #186).
Improvements
Bug fixes
All methods in Launchgen performing parameter recombination accept any iterable structure (closes #164).
Documentation
Rewrote the user guide for launchgen, which is now more concise and tangible, as well as describes the latest features.
Internals
0.12
External compatibility breaks
Dimension no longer handles named groups if the expression argument is a regular expression, as version 0.11.2 removed the feature from the io module.
Removed function sciexp2.data.build_dimension and method sciexp2.data.Dimension.build_instance in favour of a saner Dimension constructor and methods add and extend.
New features
Progress indicators now try to avoid updating the screen too often. Speed can be controlled through sciexp2.common.progress.SPEED.
Whether to show progress indicators on the screen can be globally controlled through sciexp2.common.progress.SHOW.
Add support for sciexp2.data.Data.reshape to sort axes according to their variables.
Improvements
Improvement of orders of magnitude on the speed of creation of new Data objects (thanks to optimized Dimension construction).
Improvement of orders of magnitude on the speed of reshape (thanks to optimized Dimension construction and improved algorithm).
Better progress indication in reshape and Data slicing.
Bug fixes
Fix sorting of results for file-finding routines.
Documentation
Reference documentation no longer shows class hierarchies.
Internals
Refactored progress indicators into the
progress
module.Use context manager protocol with Counter and Spinner.
Progress indicator type (counter or spinner) can be automatically selected through
sciexp2.common.progress.get
andsciexp2.common.progress.get_pickle
.Split
InstanceGroup
caching intocache_get
andcache_set
.Added proactive and zero-caching instance additions in
InstanceGroup
.Small performance improvements on various operations of the
instance
module.Move sciexp2.common.instance.Expander.Result into ExpanderResult.
Added
progressable
as a means to add progress indication in routines that where oblivious to it (e.g., adding it to an instance group to get progress indication when used with an expander).Add support for progress indicators in
sort
.
0.11.2
External compatibility breaks
Extraction routines in io do not retain the complete source expression as the dimension expression (now it just contains the variables).
New features
Improvements
Extraction routines in io ignore empty files by default.
Added expr_to_regexp to handle expression-to-regexp conversions in a single place.
Added expr_get_vars and regexp_get_names to handle variable/group name extraction in a single place (closes #195).
Failed translations show the offending substitution (closes #188).
Bug fixes
Documentation
0.11.1
External compatibility breaks
New features
Added “pattern binding” operators to
Filter
, so that filters can express matches with regular expressions.
Improvements
Ensure that maybe and maybe_other applied to extraction routines preserve the path used to find files as the dimension expression.
Bug fixes
Handle indexing of Data objects with boolean arrays.
Properly handle the axis argument in reductions when not explicitly named.
Properly translate named regular expression groups into variable references in dimension expressions. Integrates with complex path expressions given to extraction routines as an implicit argument to find_files.
Documentation
0.11
External compatibility breaks
Removed top-level scripts
launchgen
andplotter
(closes #119).Script
launcher
now is able to show the list and contents of templates (instead of the now removedlaunchgen
).The old functionality of injecting a default instance and its methods is now available (and documented) in the sciexp2.launchgen.env module.
Removed module
sciexp2.common.cmdline
.Was used by the top-level scripts, of which only one is present now.
Removed modules
sciexp2.common.config
andsciexp2.common.doc
(closes #118).Was used by the
--describe
argument of top-level scripts, which no longer exists.Removed
sciexp2.common.utils.check_module
.All checks are already performed by the requirements stated in the
setup.py
script.
New features
Added initial unit-testing framework. Can be run with
python ./setup test
. Seepython ./setup test --help
for additional arguments.Added module sciexp2.data.env to quickly import all relevant functions and classes.
Improvements
Depend on Python 2.7 or later (closes #43).
This implies that
sciexp2.common.utils.OrderedDict
has been removed in favour of Python’sOrderedDict
, and launcher now usesargparse
instead ofoptparse
.
Bug fixes
Documentation
Showing the contents of a template now lists the variables that must be forcefully defined by the user.
Properly document how to install and run using “virtualenv” and “pip” (closes #178).
0.10
External compatibility breaks
Moved data extraction methods in
sciexp2.data.io.RawData
into routines in sciexp2.data.io (extract_txt and extract_regex).Re-implemented data extraction routines (extract_txt and extract_regex) to provide a saner, simpler and more unified interface (closes #103).
Removed the bit-rotten “valuename” implementation from the Data object, which also helps decreasing the number of concepts (closes #192).
Removed
sciexp2.data.load
andsciexp2.data.io.extract_maybe
in favour of maybe and maybe_other.Removed bit-rotten module
sciexp2.data.save
.Remove
sciexp2.data.io.Source
in favour of find_files.All data extraction utilities in sciexp2.data.io can now accept either an
InstanceGroup
(resulting from a call to find_files), a tuple with the arguments for find_files or simply the file expression (thus without filters) as the first argument for find_files.Remove
sciexp2.data.io.RawData
andsciexp2.data.Data.from_rawdata
in favour of extraction routines that directly return a Data object (closes #122).
New features
Instances of Data can be built directly by the user (see Data and data_array; closes #51 and #65).
Added maybe and maybe_other to simplify the process of caching the initial extraction and transformation of data (closes #177).
Improvements
Data extraction routines can also work with file-like objects (aside from open existing files).
Routine extract_regex can now perform multiple extracts per file.
Routine reshape now lets the user specify per-field filling values for newly-generated entries (closes #55).
Bug fixes
Documentation
0.9.7
External compatibility breaks
New features
Added the possibility to filter which files to process with pack.
Improvements
Bug fixes
Fix a strange bug where find_files returned no results even though there were files to be found.
Documentation
0.9.6
External compatibility breaks
New features
Improvements
Return a Data object when slicing with numpy-derived arrays.
Bug fixes
Allow superclasses when specifying indexing methods with sciexp2.data.with_dim_index (closes #92).
Allow superclasses when specifying indexing methods with sciexp2.data.with_new_dim_index.
Return a Data object with the appropriate metadata when using reduction-like numpy routines; if the result has no meaning as a Data object, a
numpy.ndarray
is returned instead.Fix import path to ipython’s
IPython.core.ultratb
module.Fix a couple of typos in variable names when building Dimension objects.
Documentation
The todo is temporarily disabled due to a bug in Sphinx.
Improvements
Bug fixes
0.9.5
External compatibility breaks
New features
Improvements
Move common initialization code to abstract Job class.
Only try to kill jobs in Launcher if they’re in the running state.
Bug fixes
Do not use sciexp2.system.Job.state to query job state in generic machinery, but use the “_STATE” variable instead.
Rename abstract sciexp2.system.Job.status into sciexp2.system.Job.state (closes #125).
Fix cleanup of progress indication stack when (un)pickling contents with progress indication.
Documentation
0.9.4
External compatibility breaks
New features
Improvements
Improve memory usage during from_rawdata when using extremely large inputs (closes #66).
Improve speed of select (closes #63). This results in improved speeds during reshape.
Use a plain
dict
during reverse lookup in InstaceGroup (closes #120). This was unnecessary and adict
might yield faster lookups.Show a clearer error message when a filter contains unknown variable names during
match
(closes #123).
Bug fixes
Allow calls to sciexp2.launchgen.Launchgen.generate without any extra values.
Generate source distribution with description and template data files for launchgen.
Documentation
Rewrite the user guide for launchgen and Experiment execution — launcher for much more clarity and extensive examples.
0.9.3
External compatibility breaks
Deleted variables
QUEUE_CMD
andSTDINDIR
in launcher’s gridengine template.Job descriptor files for launcher use the
.jd
suffix instead of.dsc
.
New features
Added variable
QSUB_OPTS
in launcher’s gridengine template.Templates can provide their own default values (see
launchgen -T templatename
).
Improvements
When finding files, the same variable can appear more than once (e.g.,
@v1@-foo-@v2@-bar-@v1@
).More robust and faster file finding, including finding files using expressions without variable references.
Bug fixes
Documentation
Add some user-oriented summaries on the launcher template headers.
0.9.2
External compatibility breaks
Removed selector argument in sciexp2.launchgen.Launchgen.params.
New features
Let the user tinker with the SEARCH_PATH of launchgen templates.
Let the user tinker with the SEARCH_PATH of execution systems.
Improvements
Bug fixes
Documentation
User guide for launchgen.
User guide for launcher.
Auto-generated API documentation.