Changelog for dx-toolkit releases

This change log spiritually adheres to these guidelines.

Categories for each release: Added, Changed, Deprecated, Removed, Fixed, Security

Unreleased

[388.0] - beta

Added

dx build_asset supports new Application Execution Environment based on Ubuntu 24.04
dx build --remote supports new Application Execution Environment based on Ubuntu 24.04
Nextflow support added in new region aws:me-south-1
--target-file-relocation parameter for dx mv and dx cp

[387.0] - 2024.12.16

Added

--name-mode parameter for dx find data

Fixed

Nonce generation for python 3

[386.0] - 2024.12.2

Added

--database-results-restricted --unset-database-results-restricted for dx new project

Fixed

Remove pipes import for Python 3.13 compatibility

[385.0] - 2024.11.8

Added

nvidiaDriver field
--drive parameter for dx new project

Fixed

Throw an error when runSpec.distribution or runSpec.release is not present in dx build

[384.0] - 2024.10.21

Fixed

dx get_details provides project-id

[383.1] - 2024.9.26

Fixed

dx describe file-xxxx does not call /project-xxxx/listFolder

[382.0] - 2024.9.6

Added

Nextflow: Support for specifying the cpus directive in Docker process definitions
Nextaur 1.9.2: Improved error messages, mostly related to S3

Fixed

dx upload always provides project-id in /file-xxxx/describe call

[381.0] - 2024.8.26

No significant changes

[380.0] - 2024.7.26

Fixed

Occasional timeout issue when building Nextflow applets

[379.0] - 2024.7.8

Changed

Pin numpy < 2.0.0 in dxpy[pandas] extra requirements
Nextflow: Limit of 20 cached sessions in the project no longer applies when using S3 as workdir
Nextflow: Include NF workdir in context info block in job log

[378.0] - 2024.6.21

Changed

Nextflow Pipeline Applet script - refactoring

Fixed

Nextaur 1.9.1: Fixed remote bin dir; currently works when docker.enabled=true
Detecting current env for selecting Nextflow assets

[377.0] - 2024.6.10

Added

Released Nextaur 1.9.0: Enables option to host the Nextflow workdir on one's own S3 bucket, with access managed using job identity tokens.

Changed

websocket-client dxpy dependency to >=1.6.0,<1.8.0

Removed

dx upgrade command. python3 -m pip install -U dxpy is the recommended installation method

[376.0] - 2024.5.20

No significant changes

[375.1] - 2024.5.14

Fixed

dx run --instance-type-by-executable

[374.0] - 2024.4.29

Added

Released Nextaur 1.8.0: Enables login to AWS
AWS login for Nextflow subjobs and relogin for Nextflow headjob and subjobs
dx extract_assay germline --retrieve-genotype supports germline assay datasets with reference and no-call genotypes

[373.0] - 2024.4.18

Added

AWScliv2 asset to Nextflow applets

[372.0] - 2024.3.22

Added

dx-jobutil-get-identity-token

Fixed

Released Nextaur 1.7.4: Fixes error when multiple Docker images with the same name and digest are cached.

Removed

For Nextflow applets, removed allProjects: VIEW access. They will have only UPLOAD access to the current project context.

[371.0] - 2024.3.13

Fixed

dx run --ssh hanging if job has reused outputs

[370.2] - 2024.2.26

Added

Python 3.12 support
--nextflow-pipeline-params for dx build --nextflow --cache-docker
dx build --nextflow --cache-docker supports the --profile parameter to cache images associated with specific profile

Fixed

Released Nextaur 1.7.3
Fix for occasional head job hang during upload, fix for "No such file or directory" errors on localized dx files
Multiple fixes in resolving docker image digests, handling file path collision, and OOM when creating new input stream.

[369.1] - 2024.2.13

Added

Additional regional resources can be specified in dxworkflow.json or using --extra-args when dx build global workflows

[369.0] - 2024.2.5

Fixed

Bugfix regarding multi-assay assay selection in dx extract_assay expression

[368.1] - 2024.1.16

Changed

Require Python >= 3.8 for dxpy
Relaxed urllib3 dxpy dependency to >=1.25,<2.2

Fixed

Released Nextaur 1.7.0. It contains Nextflow update to 23.10.0 and multiple minor fixes.
Suppressing traceback when dx extract_assay expression attempts to access a dataset it does not have access to

Removed

Python 3.6 and 3.7 for dxpy

[367.0] - 2023.12.11

Fixed

Improved error messaging for dx extract_assay expression
Python docs for autodoc.dnanexus.com

[366.0] - 2023.12.1

Added

certifi dxpy dependency

Changed

urllib3 dxpy dependency to >=1.26.18,<2.2

Removed

requests, cryptography as dependencies of dxpy

[365.0] - 2023.11.21

Changed

Require Python >= 3.6 for dxpy
Update Nextflow to 23.10.0 (staging only)

Removed

Python 2.7 support

[364.0] - 2023.11.13

Added

dx extract_assay expression
Return jobLogsForwardingStatus field in dx describe --verbose job-xxxx
Nextflow Docker image can be cached on the platform

Changed

Disallow Python versions <2.7 or <3.5 in setup.py for dxpy, the next release will only support Python >=3.6

Fixed

Released Nextaur 1.6.9. It contains fixes for exception types so they are interpreted correctly by Nextflow when caching task runs.

[363.0] - 2023.11.6

Fixed

Project context added for loading dataset descriptor file when using dx extract_dataset

Added

Support dataset and CB records with Integer and Float type global primary keys as input in create_cohort
Return jobLogsForwardingStatus field in dx describe --verbose job-xxxx

[362.0] - 2023.10.30

Added

archival_state param to dxpy.bindings.search.find_data_objects()

[361.0] - 2023.10.16

Added

--fields-file argument in dx extract_dataset
ALT and alt_index columns in --additional-fields of dx extract_assay somatic

Fixed

dx extract_assay error message when no valid assay is found
In dx extract_assay germline, handled duplicate RSIDs, output sorting order, changed location filter range from 250M to 5M
Handled white spaces in dx extract command's inputs which are of the type string of comma separated values
Released Nextaur 1.6.8. It contains minor bugfixes and optimizations.
--retrieve-genotype --sql bug in UKB RAP region
Retry API call in object_exists_in_project()

[360.1] - 2023.10.16

Fixed

Released Nextaur 1.6.7. It adds DNAnexus docker image support feature and contains errorStrategy bugfixes

[359.1] - 2023.10.10

Changed

Retry ConnectionResetError in dxpy

Fixed

Reduce API calls in DXFile.read() after download URL is cached

[359.0] - 2023.9.29

Added

dx create_cohort
Nextflow pipeline readme file is used as a readme file of Nextflow applet
Default optional inputs nextflow_soft_confs and nextflow_params_file to Nextflow applets to support soft configuration override and custom parameters file in nextflow run

[358.0] - 2023.9.22

Changed

dx find org projects --property in Python 3

[357.0] - 2023.9.15

Changed

Remove optional output param nextflow_log from Nextflow pipeline applets; instead, always upload Nextflow log file to head job destination when execution completes

Fixed

Nextflow applets passes schema input params explicitly in nextflow run command, as parameters assigned in runtime config are not handled properly by nextflow-io
Unexpected splitting at whitespaces inside quotes when parsing string-type input parameters of Nextflow applets

[356.0] - 2023.9.1

Fixed

dx describe analysis-xxxx --json --verbose

[355.0] - 2023.8.16

Added

Return fields in dx describe {job/analysis}-xxxx with --verbose argument: 'runSystemRequirements', 'runSystemRequirementsByExecutable', 'mergedSystemRequirementsByExecutable', 'runStageSystemRequirements'
--monthly-compute-limit and --monthly-egress-bytes-limit for dx new project
--instance-type-by-executable for dx run and dx-jobutil-job-new
Parameters system_requirements and system_requirements_by_executable for DXExecutable.run() and DXJob.new()

[354.0] - 2023.8.1

Added

--try T for dx watch, dx tag/untag, dx set_properties/unset_properties
--include-restarted parameter for dx find executions/jobs/analyses
Restarted job fields in dx describe job-xxxx
treeTurnaroundTime fields in dx get and dx describe

Changed

dxpy User-Agent header includes Python version

[353.1] - 2023.7.24

Added

Fields from dx describe {job/analysis}-xxxx with --verbose argument: 'runSystemRequirements', 'runSystemRequirementsByExecutable', 'mergedSystemRequirementsByExecutable', 'runStageSystemRequirements'
dx watch --metrics top mode

[352.1] - 2023.7.12

Added

dx extract_assay somatic

Fixed

Log line truncation for strings > 8000 bytes

[351.0] - 2023.7.7

No significant changes

[350.1] - 2023.6.23

Added

dx watch support for detailed job metrics (cpu, memory, network, disk io, etc every 60s)
--detailed-job-metrics for dx run
--detailed-job-metrics-collect-default for dx update org

[349.1] - 2023.6.15

Added

dx extract_assay
external_upload_restricted param for DXProject

[348.0] - 2023.6.9

Added

dxpy dependencies test suite

Changed

Optimizations in Nextflow Pipeline Applet script to make fewer API calls when concluding a subjob

[347.0] - 2023.5.11

Changed

Bumped allowed colorama version to 0.4.6
Allow requests version up to 2.28.x

Removed

Unneeded python gnureadline dependency
Unused rlcompleter import which may break alternative readline implementations

[346.0] - 2023.4.20

Changed

Help message of the dx make_download_url command

Fixed

Released Nextaur 1.6.6. It includes fixes to errorStrategy handling and an update to the way AWS instance types are selected based on resource requirements in Nextflow pipelines (V2 instances are now preferred)
ImportError in test_dxpy.py
Replaced obsolete built-in file() method with open()
Printing HTTP error codes that were hidden for API requests to cloud storage

[345.0] - 2023.4.13

Changed

Bump allowed cryptography dxpy dependency version to 40.0.x
Tab completion in interactive executions now works with libedit bundled in MacOS and does not require externally installed GNU readline
Released Nextaur 1.6.5. It added a caching mechanism to DxPath file and folder resolution, which reduces number of DX API calls made during pipeline execution. It also fixes an occasional hanging of the headjob.

Fixed

Tab completion in interactive execution of dx-app-wizard
dx-app-wizard script on Windows
Tab completion in interactive executions on Windows

[344.0] - 2023.4.2

Changed

Released Nextaur 1.6.4. It includes a fix to folder download, minor fixes and default headjob instance update (mem2_ssd1_v2_x4 for AWS, mem2_ssd1_x4 for Azure)
Nextflow pipeline head job defaults to instance types mem2_ssd1_v2_x4 (AWS), azure:mem2_ssd1_x4 (Azure). No change to Nextflow task job instance types.

Fixed

Nextflow profiles runtime overriding fix

Added

Support for file (un)archival in DXJava
archivalStatus field to DXFile describe in DXJava
archivalStatus filtering support to DXSearch in DXJava
dx run support for --preserve-job-outputs and --preserve-job-outputs-folder inputs
dx describe for jobs and analyses outputs Preserve Job Outputs Folder field
Record the dxpy version used for Nextflow build in applet's metadata and job log

[343.0] - 2023.3.24

Changed

Released Nextaur 1.6.3. It includes updates to wait times for file upload and closing, and a fix to default Nextflow config path
Upgraded Nextflow to 22.10.7
Nextflow assets from aws:eu-west-2

[342.1] - 2023.3.8

Added

Pretty-printing additional fields for Granular Wait Times in dx describe for jobs and analyses

Changed

Released Nextaur 1.6.2. It includes bugfixes and default value of maxTransferAttempts used for file downloads is set to 3

Fixed

dx find jobs if stopppedRunning not in describe output

[341.0] - 2023.3.3

Added

dx ssh to connect to job's public hostname if job is httpsApp enabled
'--list-fields', '--list-entities', '--entities' arguments for dx extract_dataset

Changed

Released Nextaur 1.6.1. It includes an optimization of certain API calls and adds docker pull retry in Nextflow pipelines
Increased dxpy HTTP timeout to 15 minutes

Fixed

Helpstring of '--verbose' arg

[340.1] - 2023.2.25

Changed

Nextflow - updated default instance types based on destination region

Fixed

Use project ID for file-xxxx/describe API calls in dxjava DXFile
Nextflow errorStrategy retry ends in 'failed' state if last retry fails

[339.0] - 2023.2.10

No significant changes

[338.1] - 2023.1.27

Added

Support for Granular Spot wait times in dx run using --max-tree-spot-wait-time and --max-job-spot-wait-time
Printing of Spot wait times in dx describe for jobs and workflows
Support for private Docker images in Nextflow pipelines on subjob level

Fixed

Feature switch check for Nextflow pipeline build in an app execution environment
dx get database command reads from the API server with the API proxy interceptor
Regex global flags in path matching to support Py3.11
dx run --clone for Nextflow jobs (clear cloned job's properties)
Do not rewrite ubuntu repo mirror after failed execDepends install

Changed

Upgraded Nextflow plugin version to 1.5.0

[337.0] - 2023.1.20

Changed

Upgraded Nextflow plugin version to 1.4.0
Failed Nextflow subjobs with 'terminate' errorStrategy finish in 'failed' state
Updated Nextflow last error message in case 'ignore' errorStrategy is applied.
Exposed help messages for dx build --nextflow

[336.0] - 2023.1.7

No significant changes

[335.0] - 2022.12.12

Added

Group name for developer options in Nextflow pipeline applet

Fixed

Printing too many environment values with debug set to true
Preserving folder structure when publishing Nextflow output files
Missing required inputs passed to nextflow run

[334.0] - 2022.12.2

Added

--external-upload-restricted flag for dx update project and dx find projects
Support for --destination in nextflow build --repository
resume and preserve_cache input arguments to Nextflow applets to support Nextflow resume functionality
Support for error handling with Nextflow's errorStrategy
region argument to DXProject.new()

Fixed

retrieving session config when no parent process exists
an issue with describing global workflows by adding a resources container as a hint for describing underlying workflows

[333.0] - 2022.11.23

Added

nextflow run command in the log for easier debugging

Fixed

Overriding config arguments with an empty string for Nextflow pipelines

Changed

psutil version to 5.9.3 which includes wheelfiles for macOS arm64
Set ignore reuse in the nextflow applet template
Set restartableEntryPoints to "all" in the nextflow pipeline applet's runsSpec

[332.0] - 2022.11.04

Added

A warning for dx build when app(let)'s name is set both in --extra-args and --destination

Fixed

An error when setting app(let)s name in dx build (now the name set via --extra-args properly overrides the one set via --destination)
dx build --nextflow --repository returns json instead of a simple string

Changed

Help for building Nextflow pipelines is suppressed

[331.0] - 2022.10.14

Added

Added: dx find jobs --json and dx describe --verbose job-xxxx with --verbose argument return field internetUsageIPs if the caller is an org admin and the org has jobInternetUsageMonitoring enabled
Nextflow applets no longer have default arguments and required inputs

Fixed

dx describe user-xxxx will not try to print the name if it is not present in the API response

[330.0] - 2022.10.4

Added

Initial support for Nextflow
pyreadline3 dependency for Windows with Python >= 3.5

Fixed

Do not check python3 syntax with python2 and vice versa in dx build
dx build properly verifies the applet's name given in the extra-args parameter

[329.0] - 2022.9.23

Added

dx extract_dataset command
Optional pandas dependency for dxpy

Changed

dxpy.find_one_project, dxpy.find_one_data_object, dxpy.find_one_app raise DXError if zero_ok argument is not a bool

[328.0] - 2022.9.8

Added

--head-job-on-demand argument for dx run app(let)-xxxx
--head-job-on-demand argument for dx-jobutil-new-job
--on-behalf-of <org> argument for dx new user

Changed

dx-toolkit never included in execDepends when building app(lets) with dx build

Deprecated

--no-dx-toolkit-autodep option for dx build

Fixed

Reduce the number of API calls for dx run applet-xxxx and dx run workflow-xxxx
dx upload f1 f2 --visibility hidden now correctly marks both files as hidden
dx upload retry on all types of SSL errors

[327.1] - 2022.8.12

Fixed

Parsing ignoreReuse in dx build of workflow

Changed

DXHTTPRequest to pass ssl_context

[326.1] - 2022.7.7

Added

'--rank' argument for dx run

Fixed

Do not use job's workspace container ID in /applet-xxxx/run for detached jobs

[325.1] - 2022.5.25

Fixed

dx describe of executable with bundledDepends that is not an asset
Building globalworkflow from existing workflow with dx build --from

[324.1] - 2022.5.13

Fixed

Improvements to symlink downloading reliability by solely using aria2c and enhancing options around its use (removes wget option for downloading symlinked files, adds the ability to set max tries for aria2c, adds -c flag for continuing downloads, removes the --check-certificate=false option).
dx build comparison of workflow directory to workflow name
Set project argument for dx run --detach when executed from inside a job

Changed

Removed wget option for downloading symlinked files
Bump allowed requests dxpy dependency version to 2.27.1

Added

New argument symlink_max_tries for dxpy.download_dxfile() with default value of 15

[323.0] - 2022.4.28

Changed

Do not list folder contents to speed up dx cd

[322.1] - 2022.4.5

Added

API wrappers for dbcluster

Fixed

Pin websocket-client to 0.54.0 to fix dx watch output to include job output
Do not install pyreadline on Windows with Python 3.10

[321.0] - 2022.2.23

Fixed

KeyError in dx-app-wizard --json

Changed

dxjava dependencies log4j2, jackson-databind

[320.0] - 2022.2.1

Fixed

Python 3.10 collections imports
Recursive folder download dx download -r of folders with matching prefix

[319.2] - 2022.1.21

Fixed

Incorrect setting of the folder input option when building global workflows
Remove unused match_hostname urllib3 import

Added

Support for qualified workflow & applet IDs and paths when using dx build --from with an applet/workflow
Setting properties when building global workflows
'--allow-ssh' parameter to dx ssh
'--no-firewall-update' parameter to dx ssh

Changed

Detect client IP for SSH access to job instead of *

[318.0] - 2022.1.6

Fixed

Python 3.10 MutableMapping import

Added

--no-temp-build-project for single region app builds.
--from option to dx build for building a global workflow from a project-based workflow, including a workflow built using WDL

[317.0] - 2021.12.8

Fixed

Reduce file-xxxx/describe API load during dx upload
dx get uses a region compatible with user's billTo when downloading resources

Changed

dx run warns users if priority is specified as low/normal when using '--watch/ssh/allow-ssh'

[316.0] - 2021.11.17

Added

Support for dxpy on macOS arm64
Path input for dx list database files

Fixed

Python 3 SSH Host key output in dx describe job-xxxx

Changed

dxpy dependencies cryptography, websocket-client, colorama, requests

[315.0] - 2021.10.28

No significant changes

[314.0] - 2021.08.27

Added

Support FIPS enabled Python
dx archive and dx unarchive commands

Fixed

dx upload part retry where file would stay in an open state
dx run <globalworkflow> --project/--destination/--folder now submits analysis to given project or path

[313.0] - 2021.08.18

Added

'--cost-limit' arg for dx run
'--database-ui-view-only' flag for dx new project

Fixed

Total price for dx describe prints formatted currency based on currency metadata

[312.0] - 2021.07.06

No significant changes

[311.0] - 2021.05.21

Added

DX_WATCH_PORT env var for supporting dx watch in the job execution environment

[310.0] - 2021.05.12

No significant changes

[309.0] - 2021.04.28

Added

low option for --priority argument for dx run

Fixed

Provide job container-id when downloading bundledDepends in job execution environment

Changed

Upgrade to proot 5.2 from udocker2 fork for dx-docker
dx-app-wizard default to Ubuntu 20.04

[308.0] - 2021.04.23

Fixed

Search for reference genome project in region
Connection leak with HttpClient in DXFile

[307.0] - 2021.04.19

Added

--brief flag to dx-clone-asset so that script results can be used downstream

Changed

Bump jackson-databind from 2.9.10.5 to 2.9.10.7

Fixed

xattr import in dx-upload-all-outputs

[306.0] - 2021.01.21

Added

Added '--keep-open' flag for dx build

Fixed

Symlink download retries when error 22 is thrown

[305.0] - 2021.01.12

Added

'--detach' flag for dx run

Changed

Add xattr dependency to extras_require, only install if specified

Removed

Unused python-magic, beatifulsoup4 python dependencies

[304.1] - 2021.01.05

Fixed

Building assets for Ubuntu 20.04

[303.1] - 2020.11.13

Changed

Increase wget retries to 20 for symlink downloads

[302.1] - 2020.10.13

Changed

gnureadline macos dependency to 8.0.0 for Python versions < 3.9

[301.1] - 2020.09.16

Added

Remote builders for 16.04 v1, 20.04
Asset builder for 20.04
'--verbose' flag for dx-mount-all-inputs

Changed

Provide project-id in batch tsv file

[300.1] - 2020.08.31

Added

Archival api wrappers

Changed

Hide notebook and loupe-viewier from dx help output

[299.0] - 2020.08.26

Fixed

Macos tarball build

[298.1] - 2020.07.29

Added

Ubuntu 20.04 build targets

Changed

jackson-databind from 2.9.10.3 to 2.9.10.5

Fixed

API wrapper generation with Python 3
dx-clone-asset when no project exists
DXJava DXDataObject.Rename()

[297.1] - 2020.07.22

Changed

Python cryptography version >= 2.3

Fixed

dx-clone-asset with Python 3

Removed

Ubuntu 14.04 builds

[296.0] - 2020.07.01

Added

Examples for dx find with negative timestamp
dx build --from applet-xxx for app
--brief option to dx build for apps and applets

Fixed

Error handling during syntax check for dx build

Changed

Python 2.7 example applets to use Python 3
Commit dxpy version during release

[295.1] - 2020.05.19

Fixed

dx get for applets with python3 interpreter
dx-upload-all-outputs ---xattr-properties parsing

Changed

Python 2.7 example applets to use Python 3
Commit dxpy version during release

[294.0] - 2020.04.30

No significant changes

[293.0] - 2020.04.24

Added

dx-mount-all-inputs for dxfuse
Sci Linux compilation option for upload agent
Python 3 interpreter for dx-app-wizard

Fixed

dxR build
dx upload retry for "SSL EOF" error
Error out for dx run --clone with analysis and executable

[292.0] - 2020.04.09

Added

runSpec.version dxapp.json key for 16.04 applets
dx build_asset support for runSpecVersion key in dxasset.json

Fixed

Python documentation build
dxpy tests in Python 3

Changed

Bump jackson-databind from 2.9.10 to 2.9.10.1

[291.1] - 2020.03.10

Changed

Bump jackson-databind from 2.9.10 to 2.9.10.1
Retry symlink file downloads with aria2c

Fixed

Python3 issue in dx app builder test

Added

Remote app tarball builder for xenial
Allow disabling system exit on log client
database class in dx find data
Python3 compatibility for exec environment scripts
pip3 package manager for execDepends

Removed

Precise debian package build target

[290.1] - 2019.11.21

Changed

Run job as high priority if '--ssh' provided

Fixed

Project deletion warning if specifying file-id

Added

New instance types to dx-app-wizard

[289.0] - 2019.10.09

Changed

Upgrade jackson to 2.9.10

Fixed

Python 3 wrapper generation and tests

[288.0] - 2019.10.01

Added

dx get for database files
v2 instance types in dx-app-wizard

[287.0] - 2019.08.30

Fixed

Generating ruby wrappers in python 3
dx-app-wizard in python 3

[286.1] - 2019.07.08

Changed

Documentation links to https://documentation.dnanexus.com

[285.1] - 2019.07.08

Fixed

Remove non-ascii char from readme

[285.0] - 2019.06.19

Added

'--xattr-properties' argument for dx-upload-all-outputs to include fs metadata as k,v properties
xattr dependency for Linux builds

Changed

Only require futures package for python 2.7
Upgrade build dependencies for pip, setuptools, and wheel

[284.0] - 2019.06.13

Added

DXJava support for proxies
Approved tools CLI for dx update project

Changed

Upgrade jackson-databind and jackson-core to version 2.9.8
Provide project ID for dx make_download_url unless in job workspace

Fixed

Enabling argcomplete for dx installed with debian package in worker environment

[283.0] - 2019.05.13

Changed

dx upgrade downloads the latest version from s3

[282.0] - 2019.05.08

Changed

Reduce the number of API calls for dx download

Fixed

dx upload error via proxy in Azure

[281.0] - 2019.04.18

Added

support for passing HTTPContext in DXJava to the execute() operation

[280.0] - 2019.04.18

Added

--instance-count to dx run so that Spark cluster size can be defined at app start

Changed

dx wait behavior by adding exponential backoff and passing appropriate project
Decreased libcurl timeout in C++ bindings from infinity to default 10 min
Default Ubuntu release to 16.04 in dx-app-wizard
Link handling to better support JBORs

Fixed

Handling file arrays in batch job runner

[279.0] - 2019.04.11

no significant updates

[278.0] - 2019.03.21

Added

new findDataObjects inputs to DXJava
project name resolution to --project flag for dx run
smart reuse and SAML identity provider
dx list database <entity> for DNAnexus Apollo
--ignore-reuse and --ignore-reuse-stage working for dx run <workflow>

Changed

Upgrade proot to be compatible with kernel >= 4.8
Skip symlinks test in isolated environment

[277.0] - 2019.03.14

Fixed

Uploading binary data, such at compressed files, works in python3.
python3 parsing subcommand help output

Added

Binary mode for opening dx:files.
A --unicode flag for command line tools that need to handle unicode text files. For example: dx cat --unicode file-xxxx. This was added for cat, and download.

Removed

32-bit build target for Ubuntu 14.04
gtable code

[276.0] - 2019.02.08

Added

--phi flag for dx new project

Fixed

Bug in downloading symlinks when using aria2c
Max number of aria2c connections <= 16

[275.0] - 2019.02.01

Fixed

argcomplete eval in the worker when sourcing environment

[274.0] - 2019.02.01

Fixed

Preserve httpsApp field in dxapp.json when calling dx get
The --except [array:file variable] option for dx-download-all-inputs

[273.0] - 2019.01.24

Fixed

upload issue using api proxy in Python 3
--no-project option in dx-jobutil-parse-link

[272.0] - 2019.01.17

Added

A script for reconnecting to a jupyter notebook session

[271.0] - 2019.01.10

Added

support for dx building a global workflow with apps in multiple regions

Changed

Mark Ubuntu 12.04 as deprecated in Readme

Fixed

setting instance types on global workflow stages
fix test code for Spark 2.4.0 upgrade

[271.2] - 2019.01.03

Fixed

symlink download with aria2

[270.1] - 2018.12.07

No significant changes

[269.2] - 2018.12.07

Fixed

Update MANIFEST.in to include python Readme

[269.1] - 2018.12.07

Fixed

Writing to stdout for py2 in dx cat
Python virtualenv path in traceability runner

[269.0] - 2018.12.07

Fixed

Failing build app when suggestion is not a dxlink
Handle keyboard interrupt and system exit gracefully
Use absolute path to set argcomplete
If a bash variable is long, print a warning, do not omit it
Issue with backports deps on Windows

Changed

Make dx-toolkit python 2 and 3 compatible
Update macOS gnureadline version
Allow Windows to use certifi CA bundle
Update bindings for Apollo API routes
Update urllib import in dx-docker
Update requests in make doc target

Added

Test that attempts to upload to a closed file
First draft of an environment file for fish shell
If available, use aria2 rather than wget as it's much faster

Removed

Use of ubuntu 12.04 in test_dxclient.py
Old asset building script
Rare subcommands (compile, compile_dxni, sh)
The dx-run-app-locally script

[268.1] - 2018.11.29

Changed

jackson-databind and jackson-core upgrade

[267.1] - 2018.10.31

Changes

Update cran repository

[267.0] - 2018.10.18

Added

Add release/distro to example app dxapp.json files

[266.1] - 2018.10.18

Fixed

Download links for docker2aci
The test error: No root certificates specified for verification of other-side certificates

[266.0] - 2018.10.18

Added

A convenience login script for a cloud workstation
Add parents param to clone()
Allow batch IDs as output folders for batch run

Fixed

Setting a handler for a global workflow
Redirecting proxy printout to stderr
cat argument too long in a bash script
Ensure we can pop items from the main dictionary (python3)

Changed

Warn user before dx rm -r on root of a project
Let urllib3 set default CA bundle on Windows
Updgrade pyopenssl for test suite to 17.5.0
Replace ws4py with websocket-client library in job_log_client

[265.0] - 2018.10.15

Added

Pass stages input from assetDepends to bundledDepends
Traceability ID for locked workflow test (#406)

Fixed

Python 3 incompatibilites
Python 3 dx upload
import dxpy when sys.stdin is None

[264.0] - 2018.09.06

Fixed

dxpy.describe() used with a list of object IDs and an additional fields argument

[263.0] - 2018.08.30

Fixed

Sort inputs in dx generate_batch_inputs

[262.0] - 2018.08.30

Removed

12.04 builds

[261.1] - 2018.08.24

Changed

Windows install Python version upgrade to 2.7.15

Fixed

Windows installation

[261.0]

Added

dx run --ignore-reuse to disable job reuse for job
ignoreReuse key in dxapp.json

[260.0] - 2018.08.17

Changed

dxWDL version 0.74

[259.0] - 2018.08.09

Added

Ubuntu release and asset version as parameters for dx-docker create-asset
Builds for Ubuntu 16.04

Fixed

dx wait where a file with object IDs is provided in path
dx compile for debian install dxWDL.jar filepath

[258.1] - 2018.08.06

Added

Database wrappers
Support dxni compilation

Changed

requests >= 2.8.0
psutil >= 3.3.0

Fixed

Python 3 incompatibilities

[257.3] - 2018.07.26

Fixed

Revert of finding project for data object

[257.0]

Added

support for setting and updating details on global workflows
decorators for traceability tests
dx watch support for smart reuse
test for dx watch job log relay
dx find jobs/executions support for smart reuse
ability to provide a file which contains a list of jobs or data objects upon which to dx wait
dxWDL integration (dx compile command)

Changed

python-dateutil version upgrade to 2.5

Fixed

unclear behavior when --clone, --folder, and --project are simultaneously provided to dx run
DXObject.set_ids() with project set to None (it is now set to an arbitrary project associated with the file)
bash helpers tests

[256.0]

Changed

Cosmetic update to dx publish
dx publish now sets the default alias on the published version by default

[255.0] - 2018.05.24

Added

Support for updating a global workflow
Wiki links to API documentation for API wrappers
addTags/removeTags API wrappers for global workflow

Fixed

Better checking for inputs and/or inputSpec before batch running workflows

[254.0] - 2018.05.10

Changed

A workflow must be closed before initializing a global workflow from it (test updates for API changes)

[253.0] - 2018.05.02

Changed

Remove the "temporary" field from dx describe workflow-xxxx

Added

"Deleted" field to dx describe globalworkflow-xxxx
a note to dx describe that the workflow is locked

[252.0] - 2018.04.26

Added

Print proper dx describe analysis of a global workflow
dx publish command
inline cluster bootstrap script
dx run for global workflows
dx find apps tests
resolve paths for global workflows
add, remove, list users for global workflows
add, remove, list developers for global workflows
public project test
API tests

[251.2] - 2018.04.19

Added

Support for dx find global workflows
Initial support for dx build global workflow
Publish method to global workflow bindings
Support for dx get global workflow

[250.3] - 2018.04.12

Fixed

dx download for symlinks

[250.0]

Added

Support for dx describe for global workflows

[249.0] -2018.04.05

Added

zsh support
API wrappers for global workflow routes
Basic Python bindings for global workflow
set_properties() method to DXProject

Fixed

dx get applet without execDepends

[248.0] - 2018.03.29

Fixed

dx-clone-asset SSL error

[247.0] - 2018.03.14

Added

Clarify documentation of stage key for dx run
Asset builder support for Ubuntu 16.04
--singlethread parameter for dx upload

Changed

dx-docker pull retries more often

Fixed

c-ares package version for upload agent build
Bug with Azure instance type selection in dx-app-wizard
Do not retry code 422 in dxpy

[246.0] - 2018.03.14

Added

socketTimeout and connectionTimeout parameters to JAVA DXEnvironment (thanks, @pkokoshnikov)
Generate batch inputs CLI

Changed

Accept 302 as a success for ua --test

[245.0] - 2018.02.15

Added

Batch runner CLI

Changed

Updated c-ares and file packages (related to upload agent builds)

[244.0] - 2018.02.08

Added

API wrappers for [applet,app,workflow]-xxxx/validateBatch

Fixed

Issue where dx-clone-asset doesn't create projects in different regions if they don't already exist

[243.0] - 2018.02.01

Added

App version and published/unpublished note to dx run -h <app>

Changed

Recursive search for asset in a path is disabled, and we strictly enforce only one matching asset name

[242.0] - 2018.01.25

Changed

Use twine for pypy uploads

Fixed

Error that blocks org from being added as developers

[241.0]

Changed

dx-docker: cleanup of the quote code (regression fix)
dx-docker: use shutil.move() instead of os.rename() with the aci image
Accept 'http_proxy', 'HTTP_PROXY', 'https_proxy', 'HTTPS_PROXY' for proxy environmental variables
Error out instead of warning when old pricing scheme is used in dxapp.json
Fix certain tests flakiness

[240.1] - 2017.11.15

Added

Warning when runSpec.release is not specified in dxapp.json with a note it will be required in the future
Numerous enhancements to Jupyter notebook support (see https://github.com/dnanexus/dx-toolkit/commit/7ecbcb6b75118c0acd27f7a7cfe37d0a19e6d6c3 for more information)

Changed

dx-app-wizard and dx get specify systemRequirements in regionalOptions and not in runSpec of dxapp.json
multiple updates to jupyter notebook support

Deprecated

setting systemRequirements in runSpec of dxapp.json

Removed

dx-configset-to-fasta script

Fixed

dx-clone-asset sets the default regions to be all regions that user has access to and skips redundant cloning to current region
dx build now works seamlessly across apps and applets

[239.0] - 2017.11.02

Changed

Update run input help and describe messages for locked workflows
Warn when old, top-level pricing policy scheme is used in dxapp.json

Removed

dx-gtable-to-csv and dx-gtable-to-tsv scripts
dx-workflow-to-applet script
include_hidden_links parameter from clone()

[238.1] - 2017.10.27

Fixed

dx-toolkit and apt conflict with argcomplete

Added

dx-clone-asset script
dx-docker: ignore user argument if given
TLS documentation

Changed

app building tests now include runSpec.release and runSpec.distribution in app specs
dx-docker: better handling of quotes

[237.0] - 2017.10.19

Added

New InvalidTLSProtocol Exception raised when connection fails due to wrong TLS protocol.

Changed

Remove rstudio option for dx notebook

[236.2] - 2017.10.4

Fixed

dx-toolkit and apt conflict with jq

Added

Azure instance types to the list of available types in dx-app-wizard
ua -- test now displays system messages coming from the apiserver.

Changed

Update references to workflow inputs and outputs to keep them in sync with API changes

[235.1] - 2017.09.28

No significant changes

[233.0] - 2017.09.07

Added

Priority arg to build_asset
Pass region-specific pricing policy in dxapp.json

[232.1] - 2017.08.31

Fixed

Execution of old workflows built without explicit IO

[231.0] - 2017.08.24

Added

CLI support for workflow lockdown

Removed

Deprecated dx-mount

[230.0] - 2017.08.17

Added

Initial 'dx notebook' commit
Python bindings for workflow-level input and output
Support for the 'downloadRestricted' flag

[229.0] - 2017.08.10

Changed

Default to 14.04 release for all instance types in dx-app-wizard

[228.0] - 2017-07-27

No significant changes

[227.1] - 2017-07-20

Point release to fix release version issues

[227.0] - 2017-07-13

Fixed

dx-jobutil-new-job now properly supports instance type

Changed

Installation instructions now centralized on Github page
Incrementally upgraded dependencies for Java bindings

Added

Helper script to check TLS 1.2 support
A region parameter can now be passed to dx find projects, dx find data, dx find org projects

[226.0] - 2017-06-29

No significant changes

[225.0] - 2017-06-22

Added

A region parameter can now be passed to dxpy.find_data_objects() and dxpy.find_projects()

Fixed

dx-docker now no longer bind mounts $HOME and /tmp into the guest as this is consistent with Docker

[224.0] - 2017-06-08

Fixed

Python 3 compatibility with dx-app-wizard
dx get does not redundantly inline description and developerNotes in dxapp.json any more

Added

Client support for partial folder deletion

[223.0] - 2017-06-02

Added

Add methods __next__() and next() to DXFile to complete iteration interface (thanks to Eric Talevich!)

[222.0] - 2017-05-25

Fixed

--bill-to option is utilized when building multi-region apps with dx build

[221.0] - 2017-05-18

Changed

Mac install no longer uses virtualenv. Instead, we ask users to install desired version of Python

Fixed

dx-docker bug where environment variables passed in through the CLI do not get set within container

Added

dx build creates a workflow on the platform based on the workflow's local source directory

Removed

the deprecated dx-build-app and dx-build-applet scripts (that were replaced with dx build) were removed

[220.0] - 2017-05-11

Fixed

Bug introduced in release 204.0: including app resources fails

Added

Ability to specify additional resources for multi-region apps

Changed

dx ls -l and friends now request only needed describe fields

[219.0] - 2017-04-28

Added

Ability to specify bundledDepends and assetDepends for multi-region apps

[218.0]

Changed

Use DNAnexus DockerHub repo for dx-docker tests

Fixed

Issue where selecting from exactly 10 projects resulted in a stacktrace error

Added

dx get workflow-xxxx creates a local representation of a workflow

[217.0] - 2017-04-13

No significant changes.

[216.1] - 2017-04-06

Fixed

Python 3 compatibility with repr import

[216.0]

No significant changes.

[215.0]

Added

dx-docker test suite to toolkit
Retry download of Docker image if there is transient network failure
Allow image ID as a parameter for dx-docker; see wiki documentation for more details

[214.1] - 2017-03-23

Fixed

missing pyasn1 module for OSX list of install dependencies; gets rid of import warning messages

[214.0]

Added

Alternative export for dx-docker when docker image is improperly exported by docker engine

Fixed

dx run -i=filename now prompts user to select input file if duplicate filenames exist in project
dx-docker create-asset now supports output path
dx download failure when run within project that user has lost access to
dx build -f now removes all applets with matching name/directory

[213.1] - 2017-03-16

Fixed

dx-docker run KeyError when docker image is built from container

[213.0]

Fixed

Recursive file upload on Windows for the Upload Agent
Show download progress for calls to dx download -r
Issue where calls to dxpy.download_all_inputs(parallel=True) hang

[212.0]

Fixed

Upload agent now does not gzip compress .gz files on Ubuntu 14.04
Minor log message fix if file is already uploaded

Added

Mark routes as retryable for those that support idempotent calls (e.g. creating a new file)

Removed

High-level GTable bindings

[211.0] - 2017-02-23

No significant changes.

[210.0] - 2017-02-16

Fixed

Fix dx get untar issue with leading /
Missing dx-verify-file and jq dependencies on Windows

[209.0] - 2017-02-09

Added

Support to build of upload agent on RHEL7
Ability to build and get multi-region apps with custom system requirements

Fixed

Environment file so that source environment works properly on RHEL7
Modified dx-docker so that --rm passes through gracefully
Modified dx-docker so that the HOME environment variable defaults to /root

[208.1] - 2017-02-02

Fixed

bug where dx download incorrectly interpreted the overwrite option during argument parsing

[208.0]

No significant changes.

[207.0] - 2017-01-26

Added

Workflow handler to the Java bindings
Checksum verification for file downloads in Java

Fixed

Bug resulting in transient errors on large downloads
dx-app-wizard now correctly specifies timeoutPolicy
dx-docker now handles default working directory and override properly

[206.3] - 2017-01-19

Added

On Mac OS dx-toolkit now supports TLS 1.2 by activating a virtualenv

Changed

dx-app-wizard now defaults to Ubuntu 14.04 as opposed to 12.04
Cosmetic improvements to CLI describe and ls subcommands

Deprecated

Perl and Ruby bindings are no longer supported and associated code has moved to the dx-toolkit/contrib directory

Files

CHANGELOG.md

Latest commit

History

CHANGELOG.md

File metadata and controls

Changelog for dx-toolkit releases

Unreleased

[388.0] - beta

Added

[387.0] - 2024.12.16

Added

Fixed

[386.0] - 2024.12.2

Added

Fixed

[385.0] - 2024.11.8

Added

Fixed

[384.0] - 2024.10.21

Fixed

[383.1] - 2024.9.26

Fixed

[382.0] - 2024.9.6

Added

Fixed

[381.0] - 2024.8.26

[380.0] - 2024.7.26

Fixed

[379.0] - 2024.7.8

Changed

[378.0] - 2024.6.21

Changed

Fixed

[377.0] - 2024.6.10

Added

Changed

Removed

[376.0] - 2024.5.20

[375.1] - 2024.5.14

Fixed

[374.0] - 2024.4.29

Added

[373.0] - 2024.4.18

Added

[372.0] - 2024.3.22

Added

Fixed

Removed

[371.0] - 2024.3.13

Fixed

[370.2] - 2024.2.26

Added

Fixed

[369.1] - 2024.2.13

Added

[369.0] - 2024.2.5

Fixed

[368.1] - 2024.1.16

Changed

Fixed

Removed

[367.0] - 2023.12.11

Fixed

[366.0] - 2023.12.1

Added

Changed

Removed

[365.0] - 2023.11.21

Changed

Removed

[364.0] - 2023.11.13

Added

Changed

Fixed

[363.0] - 2023.11.6

Fixed

Added

[362.0] - 2023.10.30

Added