Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support post_exec on Raftstore Proxy #140

Merged
merged 60 commits into from
Aug 23, 2022
Merged
Show file tree
Hide file tree
Changes from 56 commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
0b7bf7f
f
CalvinNeo Jul 4, 2022
fea5a4a
fix
CalvinNeo Jul 4, 2022
67c47da
add engine_tiflash with this version of engine_rocks
CalvinNeo Jul 5, 2022
c96c0c3
fmt
CalvinNeo Jul 5, 2022
a3d6722
Merge remote-tracking branch 'upstream/raftstore-proxy-6.2' into try-…
CalvinNeo Jul 5, 2022
e3a716b
add engine_tiflash actually
CalvinNeo Jul 5, 2022
9bfec36
fix
CalvinNeo Jul 6, 2022
5afb804
fix some tests
CalvinNeo Jul 6, 2022
0f38d9d
f
CalvinNeo Jul 6, 2022
ed1708b
it runs if we do_write everytime
CalvinNeo Jul 6, 2022
8ab1acb
add ensure_no_common_unrecognized_keys to config checker
CalvinNeo Jul 7, 2022
5237c17
reorg tests
CalvinNeo Jul 8, 2022
4d4c583
raftstore: Implement coprocessor observer on_empty_cmd (#12851)
CalvinNeo Jun 22, 2022
b147018
fix tests
CalvinNeo Jul 8, 2022
3d50bcc
move proxy-related code in component/server into proxy_server
CalvinNeo Jul 11, 2022
8833251
Merge remote-tracking branch 'upstream/raftstore-proxy-6.2' into try-…
CalvinNeo Jul 11, 2022
f0ad93d
move codes in component/server/src/setup.rs into proxy_server
CalvinNeo Jul 11, 2022
453eb87
support new ffis in mock-engine-store and new-mock-engine-store
CalvinNeo Jul 11, 2022
573efc6
add disclaimer
CalvinNeo Jul 11, 2022
af8b448
enlength some test time
CalvinNeo Jul 12, 2022
cb5bc33
Remove pub fields of PdCluster, add feature to origin import
CalvinNeo Jul 14, 2022
e8e1851
raftstore: Implement coprocessor observer pre_exec_admin(query) (#12868)
CalvinNeo Jul 13, 2022
3dae81c
add
CalvinNeo Jul 14, 2022
731c57d
pf
CalvinNeo Jul 14, 2022
39c1db6
Merge remote-tracking branch 'upstream/raftstore-proxy-6.2' into migr…
CalvinNeo Jul 15, 2022
c301334
remove logs
CalvinNeo Jul 15, 2022
f1bba4e
remove hack: computehash/verifyhash/transferhash
CalvinNeo Jul 18, 2022
b688e40
raftstore: Implement coprocessor observer post_exec_admin(query) (#12…
CalvinNeo Jul 18, 2022
e83a37d
remove rollback of CompactLog
CalvinNeo Jul 18, 2022
a41861e
Merge branch 'migrate-pre-exec' into migrate-post-exec
CalvinNeo Jul 18, 2022
3f046f6
support post_exec
CalvinNeo Jul 18, 2022
b727d5e
raftstore: pub `check_sst_for_ingestion` (#13040)
CalvinNeo Jul 18, 2022
661767b
remove self-defined check_sst_for_ingestion
CalvinNeo Jul 18, 2022
7a78bd4
fix tests
CalvinNeo Jul 19, 2022
4eb4ef4
fix some tests and mock
CalvinNeo Jul 19, 2022
f93498e
Merge branch 'migrate-pre-exec' into migrate-post-exec
CalvinNeo Jul 19, 2022
f43c37d
add ingest tests
CalvinNeo Jul 19, 2022
951bcd1
Merge remote-tracking branch 'upstream/raftstore-proxy-6.2' into migr…
CalvinNeo Aug 9, 2022
3faedb0
remove hack, add back test
CalvinNeo Aug 9, 2022
b7fe8b7
f
CalvinNeo Aug 9, 2022
de8aa46
f
CalvinNeo Aug 9, 2022
b694fc7
fix old tests by default enable new proxy
CalvinNeo Aug 10, 2022
af80e62
raise log level
CalvinNeo Aug 11, 2022
34b3fe7
disable test_early_apply_yield_followed_with_many_entries
CalvinNeo Aug 11, 2022
e44a3d1
fix tests by enabling persist on old
CalvinNeo Aug 11, 2022
1f6bc02
fmt
CalvinNeo Aug 11, 2022
9adabd8
add back ingest sst with compat_old_proxy, to fix some bugs
CalvinNeo Aug 12, 2022
b71dcc9
fix
CalvinNeo Aug 12, 2022
8b72794
add comment
CalvinNeo Aug 15, 2022
e529489
f
CalvinNeo Aug 15, 2022
608dea5
Merge branch 'raftstore-proxy-6.2' into migrate-post-exec
CalvinNeo Aug 15, 2022
9d86bd1
raftstore: allow exec observers delay deletion of applied ssts (#13061)
CalvinNeo Aug 16, 2022
3b0569e
support pending delete ssts
CalvinNeo Aug 16, 2022
2e65d9b
Merge branch 'migrate-post-exec' of github.com:CalvinNeo/tidb-engine-…
CalvinNeo Aug 16, 2022
f34b8f8
f
CalvinNeo Aug 16, 2022
d56fa5d
add back test_empty_cmd
CalvinNeo Aug 17, 2022
5afbb35
Merge branch 'raftstore-proxy-6.2' into migrate-post-exec
CalvinNeo Aug 17, 2022
b2958e8
Merge branch 'migrate-post-exec' of github.com:CalvinNeo/tidb-engine-…
CalvinNeo Aug 18, 2022
3bae7d3
f
CalvinNeo Aug 18, 2022
c6bcbca
f
CalvinNeo Aug 19, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 17 additions & 15 deletions .github/workflows/pr-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,19 +58,21 @@ jobs:
export ENGINE_LABEL_VALUE=tiflash
export RUST_BACKTRACE=full
cargo check
cargo test --features compat_new_proxy --package tests --test proxy normal
cargo test --features compat_old_proxy --package tests --test failpoints cases::test_normal
cargo test --features compat_old_proxy --package tests --test failpoints cases::test_bootstrap
cargo test --features compat_old_proxy --package tests --test failpoints cases::test_compact_log
cargo test --features compat_old_proxy --package tests --test failpoints cases::test_early_apply
cargo test --features compat_old_proxy --package tests --test failpoints cases::test_encryption
cargo test --features compat_old_proxy --package tests --test failpoints cases::test_pd_client
cargo test --features compat_old_proxy --package tests --test failpoints cases::test_pending_peers
cargo test --features compat_old_proxy --package tests --test failpoints cases::test_transaction
cargo test --features compat_old_proxy --package tests --test failpoints cases::test_cmd_epoch_checker
cargo test --features compat_old_proxy --package tests --test failpoints cases::test_disk_full
cargo test --features compat_old_proxy --package tests --test failpoints cases::test_snap
cargo test --features compat_old_proxy --package tests --test failpoints cases::test_merge
cargo test --features compat_old_proxy --package tests --test failpoints cases::test_import_service
cargo test --features compat_old_proxy --package tests --test failpoints cases::test_proxy_replica_read
# tests based on new-mock-engine-store, with compat for new proxy
cargo test --package tests --test proxy normal
# tests based on new-mock-engine-store, for some tests not available for new proxy
cargo test --package tests --test proxy proxy
cargo test --package tests --test failpoints cases::test_normal
cargo test --package tests --test failpoints cases::test_bootstrap
cargo test --package tests --test failpoints cases::test_compact_log
cargo test --package tests --test failpoints cases::test_early_apply
cargo test --package tests --test failpoints cases::test_encryption
cargo test --package tests --test failpoints cases::test_pd_client
cargo test --package tests --test failpoints cases::test_pending_peers
cargo test --package tests --test failpoints cases::test_transaction
cargo test --package tests --test failpoints cases::test_cmd_epoch_checker
cargo test --package tests --test failpoints cases::test_disk_full
cargo test --package tests --test failpoints cases::test_snap
cargo test --package tests --test failpoints cases::test_merge
cargo test --package tests --test failpoints cases::test_import_service
cargo test --package tests --test failpoints cases::test_proxy_replica_read
5 changes: 3 additions & 2 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ cloud-azure = [
"encryption_export/cloud-azure",
"sst_importer/cloud-azure",
]
testexport = ["raftstore/testexport", "api_version/testexport"]
testexport = ["raftstore/testexport", "api_version/testexport", "engine_tiflash/testexport"]
test-engine-kv-rocksdb = [
"engine_test/test-engine-kv-rocksdb"
]
Expand All @@ -58,7 +58,8 @@ cloud-storage-grpc = ["sst_importer/cloud-storage-grpc"]
cloud-storage-dylib = ["sst_importer/cloud-storage-dylib"]
pprof-dwarf = ["pprof/backtrace-rs"]
pprof-fp = ["pprof/frame-pointer"]
compat_new_proxy = ["engine_tiflash/compat_new_proxy"]
compat_new_proxy = ["engine_tiflash/compat_new_proxy", "raftstore/compat_new_proxy"]
compat_old_proxy = ["engine_tiflash/compat_old_proxy", "raftstore/compat_old_proxy"]

# for testing configure propegate to other crates
# https://stackoverflow.com/questions/41700543/can-we-share-test-utilites-between-crates
Expand Down
1 change: 1 addition & 0 deletions components/cdc/src/delegate.rs
Original file line number Diff line number Diff line change
Expand Up @@ -446,6 +446,7 @@ impl Delegate {
for cmd in batch.into_iter(self.region_id) {
let Cmd {
index,
term: _,
mut request,
mut response,
} = cmd;
Expand Down
3 changes: 2 additions & 1 deletion components/proxy_server/src/proxy.rs
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ pub unsafe fn run_proxy(

let matches = App::new("RaftStore Proxy")
.about("RaftStore proxy to connect TiKV cluster")
.author("tongzhigao@pingcap.com")
.author("tongzhigao@pingcap.com;luorongzhen@pingcap.com")
.version(crate::proxy_version_info().as_ref())
.long_version(crate::proxy_version_info().as_ref())
.arg(
Expand Down Expand Up @@ -308,6 +308,7 @@ pub unsafe fn run_proxy(
fatal!("unknown configuration options: {}", e);
}
}
crate::config::address_proxy_config(&mut config);
println!("config check successful");
process::exit(0)
}
Expand Down
2 changes: 2 additions & 0 deletions components/raftstore/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ test-engines-panic = [
cloud-aws = ["sst_importer/cloud-aws"]
cloud-gcp = ["sst_importer/cloud-gcp"]
cloud-azure = ["sst_importer/cloud-azure"]
compat_new_proxy = ["engine_tiflash/compat_new_proxy"]
compat_old_proxy = ["engine_tiflash/compat_old_proxy"]

[dependencies]
batch-system = { path = "../batch-system", default-features = false }
Expand Down
Loading