Skip to content

Commit

Permalink
Execute smaller subset of benchmarks on PR (#233)
Browse files Browse the repository at this point in the history 
Resolves #227
  • Loading branch information
@OptimumCode
OptimumCode authored Jan 8, 2025
1 parent 51c63c2 commit 2f756ab
Show file tree
Hide file tree
Showing 2 changed files with 16 additions and 7 deletions.
17 changes: 10 additions & 7 deletions .github/workflows/benchmark.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@ on:

env:
REPORT_FORMAT: ${{ (github.event_name == 'push' || github.event_name == 'pull_request' ) && 'json' || 'csv' }}
MAIN_BENCH_TASK: ${{ github.event.pull_request.number && 'prValidationBenchmark' || 'benchmark' }}
MAIN_BENCH_RESULTS: ${{ github.event.pull_request.number && 'prValidation' || 'main' }}
COMPARISON_BENCH_RESULTS: ${{ github.event.pull_request.number && 'prValidationComparison' || 'comparison' }}

concurrency:
cancel-in-progress: true
Expand All @@ -50,13 +53,13 @@ jobs:
matrix:
include:
- os: ubuntu-latest
additional-task: ':benchmark:jvmComparisonBenchmark'
additional-task: ":benchmark:${{ github.event.pull_request.number && 'jvmPrValidationComparisonBenchmark' || 'jvmComparisonBenchmark' }}"
- os: macos-latest
additional-task: '-x :benchmark:jvmBenchmark'
additional-task: "-x :benchmark:${{ github.event.pull_request.number && 'jvmPrValidationBenchmark' || 'jvmBenchmark' }}"
- os: macos-13 # for macosX64
additional-task: '-x :benchmark:jvmBenchmark'
additional-task: "-x :benchmark:${{ github.event.pull_request.number && 'jvmPrValidationBenchmark' || 'jvmBenchmark' }}"
- os: windows-latest
additional-task: '-x :benchmark:jvmBenchmark'
additional-task: "-x :benchmark:${{ github.event.pull_request.number && 'jvmPrValidationBenchmark' || 'jvmBenchmark' }}"
runs-on: ${{ matrix.os }}
name: Run benchmarks on ${{ matrix.os }}
env:
Expand Down Expand Up @@ -89,7 +92,7 @@ jobs:
gradle-version: wrapper
- name: Run benchmarks
run: >
./gradlew --no-daemon :benchmark:benchmark ${{ matrix.additional-task }}
./gradlew --no-daemon :benchmark:${{ env.MAIN_BENCH_TASK }} ${{ matrix.additional-task }}
-Pbenchmark_warmups=${{ inputs.warmups }}
-Pbenchmark_iterations=${{ inputs.iterations }}
-Pbenchmark_iteration_time=${{ inputs.iteration-time }}
Expand Down Expand Up @@ -124,13 +127,13 @@ jobs:
uses: actions/upload-artifact@v4
with:
name: bench-result-${{ matrix.os }}
path: ${{ env.BENCHMARK_RESULTS }}/main/**/*.json
path: ${{ env.BENCHMARK_RESULTS }}/${{ env.MAIN_BENCH_RESULTS }}/**/*.json
- name: Store comparison results as artifact
if: env.REPORT_FORMAT == 'json' && matrix.os == 'ubuntu-latest'
uses: actions/upload-artifact@v4
with:
name: bench-comparison-result-${{ matrix.os }}
path: ${{ env.BENCHMARK_RESULTS }}/comparison/**/*.json
path: ${{ env.BENCHMARK_RESULTS }}/${{ env.COMPARISON_BENCH_RESULTS }}/**/*.json

upload-benchmark-results:
if: (github.event_name == 'push' || github.event_name == 'pull_request') && github.repository == 'OptimumCode/json-schema-validator'
Expand Down
6 changes: 6 additions & 0 deletions benchmark/build.gradle.kts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -62,9 +62,15 @@ benchmark {
getByName("main") {
include(".*Common.*Bench.*")
}
create("prValidation") {
include(".*CommonAvgTimeBench.*")
}
create("comparison") {
include(".*Comparison.*Benchmark.*")
}
create("prValidationComparison") {
include(".*ComparisonAvgTimeBenchmark.*")
}
}
targets {
register("jvm")
Expand Down

1 comment on commit 2f756ab

@github-actions
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'KMP JSON schema validator'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.50.

Benchmark suite Current: 2f756ab Previous: 9481407 Ratio
macosX64.CommonAvgTimeBench.validateBasic ( {"objectPath":"openapi.json","schemaPath":"openapi_schema.json"} ) 5569.831924324324 us/op 2863.359403458214 us/op 1.95
macosX64.CommonAvgTimeBench.validateBasic ( {"objectPath":"openapi-invalid.json","schemaPath":"openapi_schema.json"} ) 6111.161118324608 us/op 2945.2695735384614 us/op 2.07
macosX64.CommonAvgTimeBench.validateDetailed ( {"objectPath":"openapi-invalid.json","schemaPath":"openapi_schema.json"} ) 17672.665524 us/op 6564.317594701986 us/op 2.69
macosX64.CommonAvgTimeBench.validateFlag ( {"objectPath":"openapi.json","schemaPath":"openapi_schema.json"} ) 5670.318724324325 us/op 2096.1968796680494 us/op 2.71
macosX64.CommonAvgTimeBench.validateFlag ( {"objectPath":"openapi-invalid.json","schemaPath":"openapi_schema.json"} ) 3.796817491324954 us/op 1.5657862056729583 us/op 2.42
macosX64.CommonAvgTimeBench.validateVerbose ( {"objectPath":"openapi.json","schemaPath":"openapi_schema.json"} ) 21949.299624 us/op 11234.260193258424 us/op 1.95
macosX64.CommonAvgTimeBench.validateVerbose ( {"objectPath":"openapi-invalid.json","schemaPath":"openapi_schema.json"} ) 25025.111837499997 us/op 11014.56541777778 us/op 2.27
macosX64.CommonAvgTimeBench.validate ( {"objectPath":"openapi.json","schemaPath":"openapi_schema.json"} ) 5347.561913131312 us/op 2757.369600546448 us/op 1.94
macosX64.CommonAvgTimeBench.validate ( {"objectPath":"openapi-invalid.json","schemaPath":"openapi_schema.json"} ) 5954.428256481481 us/op 3165.890876203966 us/op 1.88

This comment was automatically generated by workflow using github-action-benchmark.

CC: @OptimumCode

Please sign in to comment.