-
-
Notifications
You must be signed in to change notification settings - Fork 124
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs(split): Cloud documentation was moved to a separate tab. (#856)
* Cloud moved to a separate tab. Added Enterprise button Co-authored-by: Anton Malinskiy <anton@malinskiy.com>
- Loading branch information
Showing
133 changed files
with
2,315 additions
and
1,650 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,28 @@ | ||
--- | ||
title: "CI/CD" | ||
--- | ||
|
||
### | ||
Marathon Cloud offers several ways to run tests within your CI/CD platform: | ||
1. Platform-Specific Steps | ||
2. Docker Image | ||
3. Marathon CLI | ||
|
||
### Platform steps | ||
We provide shared steps for the following platforms: | ||
- [Github Action](https://github.com/MarathonLabs/action-test) | ||
- [CircleCI Orb](https://circleci.com/developer/orbs/orb/marathonlabs/marathon-cloud-orb) | ||
- [Bitrise](https://bitrise.io/integrations/steps/run-tests-using-marathon-cloud) | ||
|
||
### Docker image | ||
Some platforms allow usage of [Marathon Docker Image](https://hub.docker.com/r/marathonlabs/marathon-cloud) | ||
for each step. We recommend using it for Gitlab CI and Jenkins environments. | ||
|
||
### Marathon CLI | ||
If the previously mentioned solutions are not applicable, | ||
you have the alternative of installing the [Marathon CLI](./installation) and executing tests using it. | ||
|
||
### Other | ||
If you have problems using all of the previous solutions feel free to [contact us](email:sy@marathonlabs.io). | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
--- | ||
title: "Installation" | ||
--- | ||
|
||
### | ||
This section of the documentation explains how to install the Marathon CLI on your local machine. | ||
For running tests within your Continuous Integration and Delivery (CI/CD) system, please refer to the [CI/CD section](./cicd). | ||
The installation can be performed using [Homebrew](https://brew.sh/). Here’s how to add the MarathonLabs repository: | ||
```shell | ||
brew tap malinskiy/tap | ||
``` | ||
Next, install the Marathon Cloud CLI: | ||
|
||
```shell | ||
brew install malinskiy/tap/marathon-cloud | ||
``` | ||
Alternatively, you can download prebuilt binaries for Windows, Linux, | ||
or MacOS from [the Release page](https://github.com/MarathonLabs/marathon-cloud-cli/releases). | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
--- | ||
title: "Parameters" | ||
--- | ||
|
||
### | ||
You can find all the available marathon-cli parameters by running the command `marathon-cloud --help.` | ||
Below, you'll find a list of the parameters you can set. | ||
|
||
```bash | ||
marathon-cloud --help | ||
-app string | ||
application filepath. Required | ||
android example: /home/user/workspace/sample.apk | ||
ios example: /home/user/workspace/sample.zip | ||
-testapp string | ||
test apk file path. Required | ||
android example: /home/user/workspace/testSample.apk | ||
ios example: /home/user/workspace/sampleUITests-Runner.zip | ||
-platform string | ||
testing platform. Required | ||
possible values: "Android" or "iOS" | ||
-api-key string | ||
api-key for client. Required | ||
-os-version string | ||
Android or iOS OS version | ||
-link string | ||
link to commit | ||
-name string | ||
name for run, for example it could be description of commit | ||
-o string | ||
allure raw results output folder | ||
-system-image string | ||
OS-specific system image. For Android one of [default,google_apis]. For iOS only [default] | ||
-isolated bool | ||
Run each test using isolated execution. Default is false. | ||
``` |
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,79 @@ | ||
import Tabs from '@theme/Tabs'; | ||
import TabItem from '@theme/TabItem'; | ||
|
||
# Quick start | ||
|
||
### Install | ||
The installation can be performed using [Homebrew](https://brew.sh/). Here’s how to add the MarathonLabs repository: | ||
```shell | ||
brew tap malinskiy/tap | ||
``` | ||
Next, install the Marathon Cloud CLI: | ||
|
||
```shell | ||
brew install malinskiy/tap/marathon-cloud | ||
``` | ||
Alternatively, you can download prebuild binaries for Windows, Linux, or MacOS from [the Release page](https://github.com/MarathonLabs/marathon-cloud-cli/releases). | ||
|
||
|
||
### API Key | ||
|
||
Token creation and management are available at the [Tokens page](https://cloud.marathonlabs.io/tokens). Generate a token and save it somewhere safe for the next step. | ||
|
||
### Samples (optional) | ||
|
||
To showcase the advantages of Marathon Cloud compared to other solutions, we’ve prepared a sample app with 300 tests, out of which 15% are flaky. During the initial run, our platform will gather information about the tests. During the second run, it will optimize it to fit within 15 minutes. | ||
|
||
<Tabs groupId="operating-systems" > | ||
<TabItem value="iOS" label="iOS" className="tab-content-with-text"> | ||
Download prebuild iOS Application | ||
|
||
```shell | ||
curl https://cloud.marathonlabs.io/samples/ios/sample.zip -o sample.zip | ||
``` | ||
Download prebuild iOS Test Application | ||
|
||
```shell | ||
curl https://cloud.marathonlabs.io/samples/ios/sampleUITests-Runner.zip -o sampleUITests-Runner.zip | ||
``` | ||
|
||
</TabItem> | ||
<TabItem value="Android" label="Android" className="tab-content-with-text" > | ||
Download prebuild Android Application | ||
|
||
```shell | ||
curl https://cloud.marathonlabs.io/samples/android/app.apk -o app.apk | ||
``` | ||
|
||
Download prebuild Android Test Application | ||
```shell | ||
curl https://cloud.marathonlabs.io/samples/android/appTest.apk -o appTest.apk | ||
``` | ||
|
||
</TabItem> | ||
</Tabs> | ||
|
||
To use your own applications please read Documentation. | ||
|
||
|
||
### Execution | ||
|
||
Now you can start running your tests. Use the following command to execute the CLI with the necessary parameters: | ||
|
||
<Tabs groupId="operating-systems"> | ||
<TabItem value="iOS" label="iOS"> | ||
|
||
```shell | ||
marathon-cloud -api_key generated_api_key -apk sample.zip -testapk sampleUITests-Runner.zip -platform iOS | ||
``` | ||
</TabItem> | ||
<TabItem value="Android" label="Android"> | ||
|
||
```shell | ||
marathon-cloud -api_key api_key -apk app.apk -testapk appTest.apk -platform Android | ||
``` | ||
</TabItem> | ||
</Tabs> | ||
|
||
For additional parameters, refer to the [marathon-cloud-cli README](https://github.com/MarathonLabs/marathon-cloud-cli/#installation) | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
--- | ||
title: "Android" | ||
--- | ||
|
||
### Supported frameworks | ||
Marathon Cloud supports tests written with: | ||
- UIAutomator | ||
- Cucumber | ||
- Espresso | ||
- [Kaspresso](https://github.com/KasperskyLab/Kaspresso) | ||
- Flutter (with [Patrol](https://patrol.leancode.co/)) | ||
|
||
### Application and Test Application | ||
|
||
Before initiating the testing process for your application, you’ll require two APK files: | ||
one for the application that’s being tested, and another for the tests themselves. | ||
Typically, debug variants are utilized for this purpose. | ||
|
||
If the primary application resides under the app/ subproject, | ||
you can execute the following command to build both the app and test APK: | ||
|
||
```shell | ||
# file structure | ||
# | | ||
# |--home | ||
# |--john | ||
# |--project <== you are here | ||
# |--app <== it's your primary application | ||
# ... | ||
# |--build.gragle | ||
# |--settings.gradle | ||
./gradlew :app:assembleDebug :app:assembleDebugAndroidTest | ||
``` | ||
Be sure to note the relative paths of the test APK and the app APK, as they will be required for running the tests. | ||
In the context of our example, involving the `app` project and the `debug` build, these files can be located at the following paths: | ||
|
||
- Application APK: `/home/john/project/app/build/outputs/apk/debug/app-debug.apk` | ||
- Test Application APK: `/home/john/project/app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,58 @@ | ||
--- | ||
title: "iOS" | ||
--- | ||
|
||
### Supported frameworks | ||
Marathon Cloud supports tests written with **XCTest and XCUITest frameworks**. | ||
Both the application and the tests must be built for the **ARM architecture**. | ||
|
||
### Application and Test Application | ||
|
||
Before initiating the testing process for your iOS application, you’ll need to create two `.app` bundles: one for the application that's being tested, and another for the tests themselves. Typically, `debug` variants are utilized for this purpose. | ||
Let's say our project is called "Sample". The code snippet below shows how to build the .app bundle: | ||
|
||
```shell | ||
# file structure | ||
# | | ||
# |--home | ||
# |--john | ||
# |--sample <== you are here | ||
# |--sample <== it's your application | ||
# ... | ||
# |--sample.xcodeproj | ||
|
||
xcodebuild build-for-testing \ | ||
-project sample.xcodeproj \ | ||
-scheme sample \ | ||
-destination 'platform=iOS Simulator,name=iPhone 14,OS=16.1' \ | ||
-derivedDataPath ./build | ||
``` | ||
|
||
Note the relative paths of applications, as they will be required for running the tests. In the context of our example and `debug` build, these files can be located at the following paths: | ||
|
||
- Application: `/home/john/sample/build/Build/Products/Debug-iphonesimulator/sample.app` | ||
- Test APK: `/home/john/sample/build/Build/Products/Debug-iphonesimulator/sampleUITests-Runner.app` | ||
|
||
One important thing to note is that `*.app` files are actually folders in disguise. To transfer them, it's necessary to convert these bundles into `.ipa` format or standard `zip` archives: | ||
|
||
```shell | ||
# file structure | ||
# | | ||
# |--home | ||
# |--john | ||
# |--sample <== you are here | ||
# |--build <== derivedData folder | ||
# |--sample <== it's your application | ||
# ... | ||
# |--sample.xcodeproj | ||
cd build/Build/Products/Debug-iphonesimulator | ||
# convert to zip archive in this example | ||
zip -r sample.zip sample.app | ||
zip -r sampleUITests-Runner.zip sampleUITests-runner.app | ||
``` | ||
|
||
Further, we will use these files: | ||
|
||
- Application: `/home/john/sample/build/Build/Products/Debug-iphonesimulator/sample.zip` | ||
- Test APK: `/home/john/sample/build/Build/Products/Debug-iphonesimulator/sampleUITests-Runner.zip` | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
--- | ||
title: "Overview" | ||
--- | ||
|
||
### | ||
[Marathon Cloud](https://cloud.marathonlabs.io) is a cloud testing infrastructure built on top of the Marathon test runner. | ||
It automatically provisions virtual devices to accommodate your tests within 15 minutes. | ||
The test execution is then delegated to Marathon test runner, which handles tasks such as batching, sorting, preventive retries, and post-factum retries. | ||
This ensures an even distribution of tests across the provisioned devices. | ||
|
||
### How it works | ||
Whenever you submit an application and test application to Marathon Cloud, the following steps are taken: | ||
- Calculation of the necessary number of devices | ||
- Provisioning of the devices | ||
- Distribution and execution of the tests | ||
- Control of the tests and real-time load balancing | ||
- Generation of reports | ||
- Uploading of artifacts and the generated report | ||
|
||
|
||
### Device provisioning | ||
|
||
When you run tests with Marathon Cloud for the first time, we begin storing the test history in our database. | ||
The next time you run these tests, we already have information on the average time and the probability of a successful execution for each test. | ||
Using this data, we calculate the necessary number of devices to ensure that your tests will complete within 15 minutes. | ||
We also monitor the progress of test executions and adjust the distribution of tests across devices as needed. | ||
While the tests are running, our service can dynamically increase the number of devices to expedite the execution process. | ||
|
||
### Batching | ||
|
||
Balancing speed and stability is one of the primary challenges for Marathon Cloud. | ||
In order to maintain fast test execution, we employ a batching strategy where we group 5 tests together in a single batch. | ||
This approach involves executing 5 tests consecutively, and afterward, we reset the device to a clean state. | ||
If you prefer, you can enable the "isolated" parameter to manage device cleaning yourself, | ||
but please note that this may lead to an increase in the number of devices required and the overall time of devices taken for testing. | ||
However, the total execution time will still be 15 minutes. | ||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
||
|
6 changes: 3 additions & 3 deletions
6
...ioned_docs/version-0.8.4/cloud/pricing.md → docs/cloud/misc/pricing.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,12 +1,12 @@ | ||
--- | ||
title: "Pricing" | ||
--- | ||
|
||
### | ||
We operate on a Pay-As-You-Go model, meaning you only pay for the actual time spent running tests on our virtual devices. The pricing is set at a straightforward rate of $2 per hour for each virtual device. | ||
|
||
## Billing | ||
### Billing | ||
To better understand this, picture having a test suite that requires 2 hours and 15 minutes (or 135 minutes) to run on one device. With Marathon Cloud, we automatically deploy an infrastructure of 9 virtual devices for you (135 minutes / 9 devices = 15 minutes). We then evenly distribute the tests among these devices, run them, and deliver the results to you within just 15 minutes. However, your tests will have used up 2 hours and 15 minutes (or 2.25 hours) of our device time in total. This means the cost of this test run for you would be $4.5. | ||
|
||
# Calculating Time per Device | ||
### Calculating Time per Device | ||
|
||
The computation of time for each device during a test run is based on the time span from the beginning of the first test on that device to the end of the last test on the same device. This means we don't factor in the device booting process or the application installation stage. We do, however, take into account the brief pauses between tests and the periodic device clean-ups that occur when using a batching strategy. Typically, pauses between tests take only fractions of a second, but clean-ups can require up to 6-8 seconds. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,51 @@ | ||
/** | ||
* Creating a sidebar enables you to: | ||
- create an ordered group of docs | ||
- render a sidebar for each doc of that group | ||
- provide next/previous navigation | ||
The sidebars can be generated from the filesystem, or explicitly defined here. | ||
Create as many sidebars as you want. | ||
*/ | ||
|
||
// @ts-check | ||
|
||
/** @type {import('@docusaurus/plugin-content-docs').SidebarsConfig} */ | ||
const sidebars = { | ||
docs: [ | ||
{ | ||
type: 'category', | ||
label: 'Getting started', | ||
collapsed: false, | ||
items: [ | ||
'index', | ||
'intro/overview', | ||
'intro/android', | ||
'intro/ios', | ||
] | ||
}, | ||
{ | ||
type: 'category', | ||
label: 'CLI', | ||
collapsed: false, | ||
items: [ | ||
'cli/installation', | ||
'cli/parameters', | ||
'cli/cicd' | ||
] | ||
|
||
}, | ||
{ | ||
type: 'category', | ||
label: 'Misc', | ||
collapsed: false, | ||
items: [ | ||
'misc/pricing' | ||
] | ||
}, | ||
|
||
], | ||
}; | ||
|
||
module.exports = sidebars; |
Oops, something went wrong.