Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package Ray #11160

Merged
merged 79 commits into from
Dec 10, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
6a10ca4
Add ray/meta.yaml.
dHannasch Jan 19, 2020
5577e07
Cannot use noarch even if pure Python if restricting Python version.
dHannasch Jan 19, 2020
6e0d141
ray/ci/travis/install-bazel.sh
dHannasch Jan 19, 2020
1e22955
bazel can be conda dependency
dHannasch Jan 19, 2020
3318f9e
skip install-bazel.sh
dHannasch Jan 19, 2020
1f37f0f
FileNotFoundError: [Errno 2] No such file or directory: ../README.rst
dHannasch Jan 19, 2020
49a7587
ls ../README.rst
dHannasch Jan 19, 2020
eeec768
--editable
dHannasch Jan 19, 2020
d517fab
../build.sh: line 113: unzip: command not found
dHannasch Jan 19, 2020
a6573c7
/home/conda/staged-recipes/build_artifacts/ray_1579455767395/_h_env_p…
dHannasch Jan 19, 2020
e417871
ray: cleanup
mbargull Jan 19, 2020
b536977
ray: Add compiler(cxx); move build/run deps
mbargull Jan 19, 2020
32fc2c6
see what versions we do have
dHannasch Jan 19, 2020
ffa7872
CXXABI
dHannasch Jan 22, 2020
139fdd6
fresh start
h-vetinari Mar 29, 2020
694315d
add upstream LICENCE
h-vetinari Mar 29, 2020
db08bc5
update dependencies
h-vetinari Mar 29, 2020
f6502b6
remove setup requires
h-vetinari Mar 29, 2020
75e03ae
unvendor pyarrow/pickle5
h-vetinari Mar 29, 2020
e89179f
add bazel; skip windows
h-vetinari Mar 29, 2020
958d90a
split outputs (following example of cvxpy-recipe)
h-vetinari Mar 29, 2020
b222844
add more patches
h-vetinari Mar 29, 2020
bbe7a4e
remove deprecated bazel option
h-vetinari Mar 29, 2020
5c3b243
add comment about build script
h-vetinari Mar 29, 2020
ae1391c
start adding some bazel-deps to recipe reqs
h-vetinari Mar 29, 2020
5f1b5ad
Merge remote-tracking branch 'upstream/master' into ray
h-vetinari Mar 29, 2020
2c10acb
add skips for outputs that do not have all dependencies packaged yet
h-vetinari Mar 29, 2020
28ab5bc
Revert "start adding some bazel-deps to recipe reqs"
h-vetinari Mar 29, 2020
18a9b43
fix overzealous indent
h-vetinari Mar 29, 2020
fdb485d
add dummy test for ray-debug
h-vetinari Mar 29, 2020
e949741
add recipe for ray-all; order subpackages alphabetically
h-vetinari Mar 29, 2020
5a71453
rename feedstock to avoid collision with bioconda (review beckermr)
h-vetinari Mar 29, 2020
21e8077
move folder
h-vetinari Mar 29, 2020
36b9eaf
Start migrating to Ray 1.0
vnlitvinov Nov 5, 2020
4349ea2
Align dependencies with updated setup.py
vnlitvinov Nov 5, 2020
3e698eb
ray-core now builds but fails tests
vnlitvinov Nov 5, 2020
7184a1e
Small cleaning of meta.yaml
vnlitvinov Nov 5, 2020
7290ee6
ray-core finally builds
vnlitvinov Nov 6, 2020
1ff4ea7
Sync subpackages requirements with setup.py; might not work yet
vnlitvinov Nov 9, 2020
085cb03
ray-all finally builds but needs colorful and dm-tree built first
vnlitvinov Nov 10, 2020
6858420
Make rllib and tune scripts be part of respective subpackages
vnlitvinov Nov 10, 2020
dfeee03
Update to ray-1.0.1
vnlitvinov Nov 10, 2020
b83b968
Ignore warnings in one file
vnlitvinov Nov 10, 2020
aa4216e
Disable missing deps for now
vnlitvinov Nov 16, 2020
af5c5bb
Revert je_defrag patch, change installation to "setup.py install"
vnlitvinov Nov 18, 2020
a867cef
Enable ray-rllib as dm-tree is now present
vnlitvinov Nov 18, 2020
76d554d
Remove install reqs so "python setup.py install" works
vnlitvinov Nov 18, 2020
0fcf3e0
Move to "python setup.py install" in separte script
vnlitvinov Nov 19, 2020
40255db
Linux build should finally be fixed
vnlitvinov Nov 20, 2020
e0a75e3
Start enabling Windows build
vnlitvinov Nov 20, 2020
ec342ba
Continue enabling Windows build
vnlitvinov Nov 20, 2020
2bb0a58
Enable subpackages back
vnlitvinov Nov 20, 2020
5ce80dd
Merge pull request #1 from vnlitvinov/ray-1.0.1
vnlitvinov Nov 20, 2020
5b6c756
Skip on MacOS; clean much more aggressive on other platforms
vnlitvinov Nov 22, 2020
6eac914
Fix forge linting
vnlitvinov Nov 22, 2020
683f0d3
Use renamed patch
vnlitvinov Nov 22, 2020
8ace4fe
Convert symlinks to junctions earlier
vnlitvinov Nov 22, 2020
7811029
Wipe bazel-root along with bazel-out, do not workaround correct symli…
vnlitvinov Nov 23, 2020
0d5b9d8
Add debug prints
vnlitvinov Nov 23, 2020
de69724
Disable ray-debug for now - weird compat issues
vnlitvinov Nov 23, 2020
fd9ce2d
Add workaround for os.path.isdir quirk on Windows
vnlitvinov Nov 23, 2020
3e7429e
More workarounds and debug prints
vnlitvinov Nov 23, 2020
d3d3daa
Remove debug prints, enable Linux back
vnlitvinov Nov 23, 2020
f253f56
Require mesa-libgl for rllib on Linux, patch raylet_client for Window…
vnlitvinov Nov 23, 2020
4e64c3c
Patch util.h for wider coverage
vnlitvinov Nov 23, 2020
9add46c
Add extra packages via yum for testing, remove mesa from reqs
vnlitvinov Nov 24, 2020
25c174e
Omit gpustat optional dependency on Windows
vnlitvinov Nov 24, 2020
6aff083
Disable atari deps for ray-rllib on Windows
vnlitvinov Nov 24, 2020
430bb36
Do not build ray-rllib on Windows as some deps are missing now
vnlitvinov Nov 25, 2020
a6faab2
Add licenses for all potentially included in the resulting bits libra…
vnlitvinov Dec 4, 2020
1c1067f
Change the way setup.py reqs are satisfied
vnlitvinov Dec 4, 2020
d67c78a
Enable ray.rllib on Windows
vnlitvinov Dec 5, 2020
4b32d88
Add comments about ray autoscaler on Windows
vnlitvinov Dec 5, 2020
724bc92
Remove unnecessary host requirements
vnlitvinov Dec 5, 2020
1a81806
Add pickle5 back
vnlitvinov Dec 6, 2020
a98b922
Add even more resilience to removed deps
vnlitvinov Dec 7, 2020
43a05e8
Package all licenses properly
vnlitvinov Dec 7, 2020
e3fc07f
Workaround symlinks on Windows better (hopefully)
vnlitvinov Dec 7, 2020
624613c
Adjust selectors styling
vnlitvinov Dec 8, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
272 changes: 272 additions & 0 deletions recipes/ray-packages/LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,272 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/

TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

1. Definitions.

"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.

"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.

"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.

"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.

"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.

"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.

"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).

"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.

"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."

"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.

2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.

3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.

4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:

(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and

(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and

(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and

(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.

You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.

5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.

6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.

7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.

8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.

9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.

END OF TERMS AND CONDITIONS

APPENDIX: How to apply the Apache License to your work.

To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "{}"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.

Copyright {yyyy} {name of copyright owner}

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

--------------------------------------------------------------------------------

Code in python/ray/rllib/{evolution_strategies, dqn} adapted from
https://github.com/openai (MIT License)

Copyright (c) 2016 OpenAI (http://openai.com)

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

--------------------------------------------------------------------------------

Code in python/ray/rllib/impala/vtrace.py from
https://github.com/deepmind/scalable_agent

Copyright 2018 Google LLC

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

--------------------------------------------------------------------------------
Code in python/ray/rllib/ars is adapted from https://github.com/modestyachts/ARS

Copyright (c) 2018, ARS contributors (Horia Mania, Aurelia Guy, Benjamin Recht)
All rights reserved.

Redistribution and use of ARS in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation and/or
other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
16 changes: 16 additions & 0 deletions recipes/ray-packages/build-core.bat
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
cd python
set SKIP_THIRDPARTY_INSTALL=1
"%PYTHON%" setup.py install
rem remember the return code
set RETCODE=%ERRORLEVEL%

rem Now clean everything up so subsequent builds (for potentially
rem different Python version) do not stumble on some after-effects.
"%PYTHON%" setup.py clean --all

rem Now shut down Bazel server, otherwise Windows would not allow moving a directory with it
bazel "--output_user_root=%SRC_DIR%\..\bazel-root" "--output_base=%SRC_DIR%\..\b-o" clean
bazel "--output_user_root=%SRC_DIR%\..\bazel-root" "--output_base=%SRC_DIR%\..\b-o" shutdown
rd /s /q "%SRC_DIR%\..\b-o" "%SRC_DIR%\..\bazel-root"
rem Ignore "bazel shutdown" errors
exit /b %RETCODE%
20 changes: 20 additions & 0 deletions recipes/ray-packages/build-core.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
#!/bin/bash
set -xe

cd python/
export SKIP_THIRDPARTY_INSTALL=1
"${PYTHON}" setup.py build
# bazel by default makes everything read-only,
# which leads to patchelf failing to fix rpath in binaries.
# find all ray binaries and make them writable
grep -lR ELF build/ | xargs chmod +w

# now install the thing so conda could pick it up
"${PYTHON}" setup.py install

# now clean everything up so subsequent builds (for potentially
# different Python version) do not stumble on some after-effects
"${PYTHON}" setup.py clean --all
bazel "--output_user_root=$SRC_DIR/../bazel-root" "--output_base=$SRC_DIR/../b-o" clean
bazel "--output_user_root=$SRC_DIR/../bazel-root" "--output_base=$SRC_DIR/../b-o" shutdown
rm -rf "$SRC_DIR/../b-o" "$SRC_DIR/../bazel-root"
Loading