Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add qbg-handler #2807

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

add qbg-handler #2807

wants to merge 2 commits into from

Conversation

datelier
Copy link
Contributor

@datelier datelier commented Jan 22, 2025

Description

  • Add rust-QBG-Handler
    • Not implemented Multi/Stream API yet

Related Issue

Versions

  • Vald Version: v1.7.15
  • Go Version: v1.23.4
  • Rust Version: v1.83.0
  • Docker Version: v27.4.0
  • Kubernetes Version: v1.32.0
  • Helm Version: v3.16.3
  • NGT Version: v2.3.5
  • Faiss Version: v1.9.0

Checklist

Special notes for your reviewer

Summary by CodeRabbit

Release Notes

  • New Features

    • Added support for a new QBG (Quantum Bit Graph) algorithm service.
    • Enhanced object management with new methods for searching, inserting, updating, and removing objects.
    • Introduced new error types for improved error handling.
  • Improvements

    • Upgraded concurrency control with thread-safe implementations.
    • Refined method signatures for better readability and safety.
    • Added detailed logging for request processing.
    • Enhanced error reporting with more context.
  • Performance

    • Optimized index management and search operations.
    • Improved thread safety for shared data structures.
  • Bug Fixes

    • Resolved issues with dimension size validation.
    • Improved UUID handling and validation.

These changes represent a significant upgrade to the agent's functionality and reliability.

Copy link
Contributor

coderabbitai bot commented Jan 22, 2025

📝 Walkthrough

Walkthrough

This pull request introduces significant changes to the Rust agent implementation, primarily focusing on integrating a new qbg library and enhancing the agent's functionality. The changes span multiple files across the agent and algorithm libraries, introducing new error handling, concurrency improvements, and expanded method implementations for object and index management. The modifications include adding new dependencies, updating method signatures, implementing more robust error handling, and introducing thread-safe data structures.

Changes

File Change Summary
rust/bin/agent/Cargo.toml Added qbg dependency with version 0.1.0
rust/bin/agent/src/handler.rs Updated Agent struct to use Arc<RwLock<dyn algorithm::ANN>> for improved concurrency
rust/bin/agent/src/handler/index.rs Enhanced error handling and logging for index-related methods
rust/bin/agent/src/handler/insert.rs Improved error handling and validation for vector insertion
rust/bin/agent/src/handler/object.rs New implementation of object_server::Object trait with methods for object management
rust/bin/agent/src/handler/remove.rs Comprehensive implementation of remove method with detailed error handling
rust/bin/agent/src/handler/search.rs Refactored error handling logic for search method
rust/bin/agent/src/handler/update.rs Enhanced update method with comprehensive error handling
rust/bin/agent/src/handler/upsert.rs Added new methods for upsert operations with improved error handling
rust/bin/agent/src/main.rs Replaced MockService with QBGService and updated implementation
rust/libs/algorithm/src/lib.rs Added new error variants and expanded ANN trait methods
rust/libs/algorithms/qbg/src/* Added const qualifiers to methods and improved thread safety

Sequence Diagram

sequenceDiagram
    participant Client
    participant Agent
    participant QBGService
    participant Index

    Client->>Agent: Request (insert/search/update)
    Agent->>QBGService: Process request
    QBGService->>Index: Perform operation
    alt Operation Successful
        Index-->>QBGService: Return result
        QBGService-->>Agent: Return response
        Agent-->>Client: Send response
    else Operation Failed
        Index-->>QBGService: Return error
        QBGService-->>Agent: Generate error details
        Agent-->>Client: Send error status
    end
Loading

Possibly related PRs

Suggested labels

size/XXXL, area/agent/core/ngt, area/agent/core/faiss, actions/backport/release/v1.7

Suggested reviewers

  • kmrmt
  • hlts2
  • vankichi

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@vdaas-ci
Copy link
Collaborator

[CHATOPS:HELP] ChatOps commands.

  • 🙆‍♀️ /approve - approve
  • 🍱 /format - format codes and add licenses
  • /gen-test - generate test codes
  • 🏷️ /label - add labels
  • 🔚 2️⃣ 🔚 /label actions/e2e-deploy - run E2E deploy & integration test

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 23

🧹 Nitpick comments (17)
rust/bin/agent/src/handler/upsert.rs (1)

64-64: Use is_empty() instead of len() == 0 for string emptiness check

For better readability and idiomatic Rust code, use the is_empty() method to check if a string is empty.

Apply this diff:

-            if uuid.len() == 0 {
+            if uuid.is_empty() {
rust/bin/agent/src/handler/index.rs (5)

32-32: Correct the typo in the log message.

There's a spelling error in the log message: "Recieved" should be corrected to "Received".

Apply this diff to correct the typo:

-println!("Recieved a request from {:?}", request.remote_addr());
+println!("Received a request from {:?}", request.remote_addr());

36-36: Handle potential Unicode errors when converting hostname to string.

Using to_str().unwrap() might cause a panic if the hostname contains invalid UTF-8. Consider handling this case to prevent potential runtime panics.

Apply this diff to handle the error gracefully:

-let domain = hostname.to_str().unwrap();
+let domain = match hostname.to_str() {
+    Some(s) => s.to_string(),
+    None => {
+        let status = Status::internal("Invalid hostname encoding");
+        return Err(status);
+    }
+};

94-94: Correct the typo in the log message.

There's a spelling error in the log message: "Recieved" should be corrected to "Received".

Apply this diff to correct the typo:

-println!("Recieved a request from {:?}", request.remote_addr());
+println!("Received a request from {:?}", request.remote_addr());

96-96: Handle potential Unicode errors when converting hostname to string.

Using to_str().unwrap() may cause a panic if the hostname contains invalid UTF-8. It's advisable to handle this potential error.

Apply this diff:

-let domain = hostname.to_str().unwrap();
+let domain = match hostname.to_str() {
+    Some(s) => s.to_string(),
+    None => {
+        let status = Status::internal("Invalid hostname encoding");
+        return Err(status);
+    }
+};

137-137: Correct the typo in the log message.

Again, "Recieved" should be corrected to "Received" to fix the spelling error.

Apply this diff:

-println!("Recieved a request from {:?}", request.remote_addr());
+println!("Received a request from {:?}", request.remote_addr());
rust/bin/agent/src/handler/insert.rs (3)

32-32: Correct the typo in the log message.

The word "Recieved" is misspelled; it should be "Received".

Apply this diff:

-println!("Recieved a request from {:?}", request.remote_addr());
+println!("Received a request from {:?}", request.remote_addr());

59-59: Fix typos in the error message.

The error message contains typos: "Incombatible" should be "Incompatible", and "detedted" should be "detected".

Apply this diff:

-"Insert API Incombatible Dimension Size detedted",
+"Insert API Incompatible Dimension Size detected",

51-51: Handle potential UTF-8 conversion errors when encoding the request.

Using String::from_utf8(req.encode_to_vec()).unwrap() can panic if the encoded bytes are not valid UTF-8. Handle the Result to prevent possible panics.

Apply this diff:

-err_details.set_request_info(vec.id, String::from_utf8(req.encode_to_vec()).unwrap());
+let req_str = match String::from_utf8(req.encode_to_vec()) {
+    Ok(s) => s,
+    Err(_) => "<invalid UTF-8>".to_string(),
+};
+err_details.set_request_info(vec.id, req_str);
rust/bin/agent/src/handler/update.rs (3)

32-32: Correct the typo in the log message.

"Recieved" should be corrected to "Received" to fix the spelling error.

Apply this diff:

-println!("Recieved a request from {:?}", request.remote_addr());
+println!("Received a request from {:?}", request.remote_addr());

64-64: Use is_empty() method for string emptiness check.

Instead of comparing uuid.len() == 0, it's more idiomatic to use uuid.is_empty().

Apply this diff:

-if uuid.len() == 0 {
+if uuid.is_empty() {

71-74: Handle potential UTF-8 conversion errors when encoding the request.

As with previous instances, unwrapping the result of String::from_utf8 may panic if the data is not valid UTF-8.

Apply this diff:

-err_details.set_request_info(
-    uuid.clone(),
-    String::from_utf8(req.encode_to_vec()).unwrap(),
-);
+let req_str = match String::from_utf8(req.encode_to_vec()) {
+    Ok(s) => s,
+    Err(_) => "<invalid UTF-8>".to_string(),
+};
+err_details.set_request_info(uuid.clone(), req_str);
rust/bin/agent/src/handler/search.rs (2)

57-57: Fix typos in error message

The error message at line 57 contains typos: "Incombatible" should be "Incompatible", and "detedted" should be "detected".

Apply this diff to correct the typos:

-                        "Search API Incombatible Dimension Size detedted",
+                        "Search API Incompatible Dimension Size detected",

73-146: Refactor repetitive error handling code

The error handling within the match err block repeats similar patterns for constructing err_details and returning Status. Refactoring this code can reduce duplication and improve maintainability.

Consider creating a helper function to construct the Status with error details:

fn build_error_status(
    code: Code,
    message: &str,
    err: &Error,
    domain: &str,
    metadata: HashMap<String, String>,
    request_id: &str,
    request_info: &str,
    resource_type: &str,
    resource_name: &str,
    field_violations: Option<Vec<FieldViolation>>,
) -> Status {
    let mut err_details = ErrorDetails::new();
    err_details.set_error_info(err.to_string(), domain, metadata);
    err_details.set_request_info(request_id, request_info);
    err_details.set_resource_info(resource_type, resource_name, "", "");
    if let Some(violations) = field_violations {
        err_details.set_bad_request(violations);
    }
    Status::with_error_details(code, message, err_details)
}

Then refactor the match arms to use this helper function:

-                            Error::CreateIndexingIsInProgress {} => {
-                                let mut err_details = ErrorDetails::new();
-                                // ... (repeated code)
-                                Status::with_error_details(Code::Aborted, "Search API aborted to process search request due to creating indices is in progress", err_details)
-                            }
+                            Error::CreateIndexingIsInProgress {} => build_error_status(
+                                Code::Aborted,
+                                "Search API aborted to process search request due to creating indices is in progress",
+                                &err,
+                                domain,
+                                metadata,
+                                &config.request_id,
+                                debug_info,
+                                &resource_type,
+                                &resource_name,
+                                None,
+                            ),

Apply similar changes to the other match arms to streamline the error handling.

rust/bin/agent/src/handler/object.rs (1)

35-35: Fix typo in debug print statement.

"Recieved" should be "Received".

-        println!("Recieved a request from {:?}", request.remote_addr());
+        println!("Received a request from {:?}", request.remote_addr());
rust/bin/agent/src/handler/remove.rs (2)

32-32: Fix typo in log message.

"Recieved" should be "Received".

-        println!("Recieved a request from {:?}", request.remote_addr());
+        println!("Received a request from {:?}", request.remote_addr());

63-129: Refactor duplicated error handling code.

The error handling blocks contain significant code duplication in creating error details. Consider extracting the common pattern into a helper function.

impl super::Agent {
    fn create_error_details(
        &self,
        err: Error,
        domain: &str,
        uuid: String,
        req: &remove::Request,
        add_field_violation: bool,
    ) -> ErrorDetails {
        let metadata = HashMap::new();
        let resource_type = self.resource_type.clone() + "/qbg.Remove";
        let resource_name = format!("{}: {}({})", self.api_name, self.name, self.ip);
        
        let mut err_details = ErrorDetails::new();
        err_details.set_error_info(err.to_string(), domain, metadata);
        err_details.set_request_info(
            uuid.clone(),
            String::from_utf8(req.encode_to_vec()).unwrap(),
        );
        if add_field_violation {
            err_details.set_bad_request(vec![FieldViolation::new("uuid", err.to_string())]);
        }
        err_details.set_resource_info(resource_type, resource_name, "", "");
        
        err_details
    }
}

This would simplify the error handling blocks significantly.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 8179e7d and d6f9cbc.

⛔ Files ignored due to path filters (1)
  • rust/Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (14)
  • rust/bin/agent/Cargo.toml (1 hunks)
  • rust/bin/agent/src/handler.rs (1 hunks)
  • rust/bin/agent/src/handler/index.rs (2 hunks)
  • rust/bin/agent/src/handler/insert.rs (1 hunks)
  • rust/bin/agent/src/handler/object.rs (1 hunks)
  • rust/bin/agent/src/handler/remove.rs (1 hunks)
  • rust/bin/agent/src/handler/search.rs (2 hunks)
  • rust/bin/agent/src/handler/update.rs (1 hunks)
  • rust/bin/agent/src/handler/upsert.rs (1 hunks)
  • rust/bin/agent/src/main.rs (1 hunks)
  • rust/libs/algorithm/src/lib.rs (1 hunks)
  • rust/libs/algorithms/qbg/src/input.cpp (3 hunks)
  • rust/libs/algorithms/qbg/src/input.h (1 hunks)
  • rust/libs/algorithms/qbg/src/lib.rs (2 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (4)
  • GitHub Check: check-format-diff
  • GitHub Check: build / build
  • GitHub Check: Analyze (go)
  • GitHub Check: Cloudflare Pages
🔇 Additional comments (17)
rust/libs/algorithm/src/lib.rs (1)

70-76: Verify the return type of the search method

The search method now returns Result<search::Response, Error> instead of Result<tonic::Response<search::Response>, Error>. Ensure this change is compatible with the rest of the codebase and that all callers handle the updated return type appropriately.

rust/bin/agent/src/handler/index.rs (3)

35-35: Verify the use of cargo::util::hostname() function.

The function cargo::util::hostname() may not be the standard way to retrieve the hostname. Please ensure that this function is correctly implemented or consider using hostname::get() or another appropriate method.


34-34: Unused variable pool_size.

The variable pool_size is extracted from the request but not utilized in the create_index method. If pool_size is intended to influence index creation, ensure it's passed to s.create_index() or the relevant function.


95-95: Verify the use of cargo::util::hostname() function.

Similar to the previous occurrence, ensure that cargo::util::hostname() correctly retrieves the hostname. Consider using hostname::get() if appropriate.

rust/bin/agent/src/handler.rs (2)

19-19: Module addition is appropriate

Adding pub mod object; correctly includes the new object module into the handler, aligning with the project structure.


24-25: Proper use of Arc<RwLock> for shared access

Updating the Agent struct to use Arc<RwLock<dyn algorithm::ANN>> for the s field enhances concurrency control, allowing safe shared access across asynchronous contexts. The initialization in the new method is correctly implemented.

Also applies to: 28-44

rust/libs/algorithms/qbg/src/input.h (1)

111-113: Good practice: Adding const qualifiers to methods

Adding const qualifiers to the search, get_object, and get_dimension methods improves const-correctness, allowing these methods to be called on const instances and ensuring they do not modify the object's state.

rust/libs/algorithms/qbg/src/input.cpp (3)

336-336: LGTM! Good improvement in const-correctness.

Making the search method const-correct is a good practice as it doesn't modify the index state and enables better thread safety.


390-390: LGTM! Good improvement in const-correctness.

Making the get_object method const-correct is a good practice as it doesn't modify the index state and enables better thread safety.


406-406: LGTM! Good improvement in const-correctness.

Making the get_dimension method const-correct is a good practice as it doesn't modify the index state and enables better thread safety.

rust/libs/algorithms/qbg/src/lib.rs (3)

98-105: LGTM! Good alignment with C++ const-correctness.

The FFI function signatures have been correctly updated to use immutable references, matching the C++ const methods. This ensures consistent behavior across the language boundary.


373-384: LGTM! Good error handling and type conversion.

The search method implementation correctly handles the FFI boundary and provides appropriate error handling. The return type change to Result<Vec<(u32, f32)>> is more idiomatic Rust.


109-112: Verify thread safety of Send and Sync implementations.

While marking Property and Index as Send and Sync is necessary for thread safety, we need to ensure that the underlying C++ implementations are actually thread-safe.

✅ Verification successful

Thread safety verification successful for Property and Index

The Send and Sync implementations are safe because:

  • Both types use UniquePtr for exclusive ownership
  • All mutations require exclusive access through &mut self
  • No shared mutable state exists that could cause data races
🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Search for mutex/lock usage in C++ code
rg -A 5 "std::(mutex|shared_mutex|recursive_mutex|lock_guard|unique_lock|shared_lock)" rust/libs/algorithms/qbg/src/

Length of output: 118


Script:

#!/bin/bash
# Check FFI type definitions and their usage
echo "=== FFI Type Definitions ==="
rg -A 10 "struct.*Property|struct.*Index" rust/libs/algorithms/qbg/src/

echo -e "\n=== Property/Index Usage Patterns ==="
ast-grep --pattern 'impl $_ for ffi::Property { $$$ }'
ast-grep --pattern 'impl $_ for ffi::Index { $$$ }'

echo -e "\n=== Shared State Access ==="
rg -g '*.rs' -A 5 "static|lazy_static|thread_local" rust/libs/algorithms/qbg/src/

Length of output: 3783

rust/bin/agent/Cargo.toml (1)

25-25: Verify qbg package version compatibility.

The qbg package is pinned to version 0.1.0. Since this is an internal dependency, ensure that this version is compatible with the rest of the project and follows the project's versioning strategy.

✅ Verification successful

Version specification is consistent across workspace

The qbg package version 0.1.0 is consistently specified across all workspace dependencies, and all references use appropriate path-based workspace dependencies.

🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Check version consistency across workspace
rg -A 1 'version = "0.1.0"' rust/libs/algorithms/qbg/Cargo.toml

Length of output: 100


Script:

#!/bin/bash
# Find all Cargo.toml files that depend on qbg
rg -l "qbg.*=.*" --type toml

# Check workspace configuration
cat rust/Cargo.toml

Length of output: 906


Script:

#!/bin/bash
# Check qbg dependency in algorithm crate
rg -A 1 'qbg.*=.*' rust/libs/algorithm/Cargo.toml

Length of output: 156

rust/bin/agent/src/handler/remove.rs (3)

16-24: LGTM! Imports are well-organized and appropriate.

The imports cover all necessary components for implementing a gRPC service with proper error handling and protobuf support.


130-135: LGTM! Success response is well-structured.

The response includes all necessary information (name, UUID, and IPs) in the object::Location structure.


Line range hint 138-168: Track unimplemented methods.

The following methods need to be implemented:

  • remove_by_timestamp
  • stream_remove
  • multi_remove

These are part of the required trait implementation and should be tracked for future implementation.

Would you like me to create issues to track the implementation of these methods?

rust/libs/algorithm/src/lib.rs Outdated Show resolved Hide resolved
rust/libs/algorithm/src/lib.rs Outdated Show resolved Hide resolved
rust/bin/agent/src/main.rs Outdated Show resolved Hide resolved
rust/bin/agent/src/main.rs Show resolved Hide resolved
rust/bin/agent/src/main.rs Show resolved Hide resolved
rust/bin/agent/src/handler/object.rs Show resolved Hide resolved
rust/bin/agent/src/handler/object.rs Show resolved Hide resolved
rust/bin/agent/src/handler/object.rs Outdated Show resolved Hide resolved
rust/bin/agent/src/handler/remove.rs Show resolved Hide resolved
rust/bin/agent/src/handler/remove.rs Outdated Show resolved Hide resolved
Copy link

cloudflare-workers-and-pages bot commented Jan 22, 2025

Deploying vald with  Cloudflare Pages  Cloudflare Pages

Latest commit: 259bae6
Status: ✅  Deploy successful!
Preview URL: https://e5f3c15b.vald.pages.dev
Branch Preview URL: https://feature-agent-rust-agent-qbg.vald.pages.dev

View logs

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

♻️ Duplicate comments (5)
rust/libs/algorithm/src/lib.rs (2)

26-27: 🛠️ Refactor suggestion

Ensure consistent use of UUID data types in Error variants

The Error variants UUIDAlreadyExists and UUIDNotFound use different data types for the UUID field. UUIDAlreadyExists uses uuid: String, while UUIDNotFound uses id: usize. For consistency and clarity, consider changing id: usize to uuid: String in UUIDNotFound.

Apply this diff to maintain consistency:

 pub enum Error {
     CreateIndexingIsInProgress {},
     FlushingIsInProgress {},
     EmptySearchResult {},
     IncompatibleDimensionSize { got: usize, want: usize },
     UUIDAlreadyExists { uuid: String },
-    UUIDNotFound { id: usize },
+    UUIDNotFound { uuid: String },
     UncommittedIndexNotFound {},
     InvalidUUID { uuid: String },
     ObjectIDNotFound { uuid: String },
     Unknown {},
 }

47-53: 🛠️ Refactor suggestion

Update fmt::Display implementations to match Error variant changes

After changing the Error variant UUIDNotFound to use uuid: String, update the fmt::Display implementation accordingly to reflect the change.

Apply this diff:

             Error::UUIDAlreadyExists { uuid } => write!(f, "uuid {} index already exists", uuid),
-            Error::UUIDNotFound { id } => {
-                if *id == (0 as usize) {
+            Error::UUIDNotFound { uuid } => {
+                if uuid.is_empty() {
                     write!(f, "object uuid not found")
                 } else {
-                    write!(f, "object uuid {}'s metadata not found", id)
+                    write!(f, "object uuid {}'s metadata not found", uuid)
                 }
             }
rust/bin/agent/src/main.rs (2)

106-114: ⚠️ Potential issue

Implement proper UUID to ID conversion in exists method

Currently, the exists method ignores the provided uuid and uses a hardcoded id = 1. This will not correctly check for the existence of the specified object. Implement a proper conversion from UUID to the corresponding ID.

Example implementation:

 fn exists(&self, _uuid: String) -> bool {
     // convert uuid to id
-    let id = 1;
+    let id = self.convert_uuid_to_id(_uuid);
     let result = self.index.get_object(id);
     match result {
         Ok(_vec) => true,
         Err(_err) => false,
     }
 }

171-178: ⚠️ Potential issue

Implement proper UUID to ID conversion in get_object method

Currently, the method ignores the provided uuid and uses a hardcoded id = 1. This will not retrieve the intended object. Implement correct conversion to retrieve the specified object.

Example adjustment:

 fn get_object(&self, _uuid: String) -> Result<(Vec<f32>, i64), Error> {
     // convert uuid to id
-    let id = 1;
+    let id = self.convert_uuid_to_id(_uuid);
     let vec = self.index.get_object(id).map_err(Error::from)?;
     // get timestamp
     let ts: i64 = 0;
     Ok((vec.to_vec(), ts))
 }
rust/bin/agent/src/handler/remove.rs (1)

43-44: ⚠️ Potential issue

Add proper error handling for hostname conversion.

The current implementation could panic if the hostname contains invalid UTF-8 characters.

🧹 Nitpick comments (9)
rust/bin/agent/src/handler/upsert.rs (4)

74-74: Use is_empty() instead of len() == 0 for checking empty strings

It's more idiomatic and clear to use uuid.is_empty() when checking if a string is empty.

Apply this diff:

-            if uuid.len() == 0 {
+            if uuid.is_empty() {

97-98: Avoid holding the read lock for prolonged operations

Within the scope where the read lock s is held (lines 41-159), there are asynchronous operations (self.update and self.insert) that could potentially block other readers or writers. Consider restructuring the code to minimize the duration for which the read lock is held.

You can limit the scope of the read lock to only the necessary operations, like accessing s.exists(uuid.clone()), and release it before performing asynchronous operations.


96-123: Consider initializing rt_name after determining the operation

The variable rt_name is declared before the match on exists, but its value depends on whether the UUID exists or not. To improve code clarity, declare and initialize rt_name within each branch where it's used.

Apply this diff:

             let result;
-            let rt_name;
             let exists = s.exists(uuid.clone());
             if exists {
                 result = self
                     .update(tonic::Request::new(update::Request {
                         vector: req.vector.clone(),
                         config: Some(update::Config {
                             skip_strict_exist_check: true,
                             filters: config.filters,
                             timestamp: config.timestamp,
                             disable_balanced_update: config.disable_balanced_update,
                         }),
                     }))
                     .await;
+                let rt_name = format!("{}{}", "/qbg.Upsert", "/qbg.Update");
             } else {
                 result = self
                     .insert(tonic::Request::new(insert::Request {
                         vector: req.vector.clone(),
                         config: Some(insert::Config {
                             skip_strict_exist_check: true,
                             filters: config.filters,
                             timestamp: config.timestamp,
                         }),
                     }))
                     .await;
+                let rt_name = format!("{}{}", "/qbg.Upsert", "/qbg.Insert");
             }

125-156: Simplify error handling in match statement

The error handling logic in the match result block is complex and could be simplified. Consider restructuring the error handling to make it more readable and maintainable.

For example, you could extract the error handling code into a separate function or simplify the match arms to reduce nested code.

rust/bin/agent/src/handler/remove.rs (2)

32-32: Fix typo in logging message.

There's a typo in the logging message: "Recieved" should be "Received".

-        println!("Recieved a request from {:?}", request.remote_addr());
+        println!("Received a request from {:?}", request.remote_addr());

16-24: Consider creating a shared error handling module.

The error handling code is duplicated across insert.rs, update.rs, and remove.rs. Consider extracting common error handling logic into a shared module.

Create a new module (e.g., error_utils.rs) with shared error handling functions:

pub fn create_error_details(
    err: Error,
    domain: &str,
    uuid: String,
    request: &impl Message,
    resource_type: String,
    resource_name: String,
) -> ErrorDetails {
    let mut err_details = ErrorDetails::new();
    let metadata = HashMap::new();
    err_details.set_error_info(err.to_string(), domain, metadata);
    err_details.set_request_info(
        uuid,
        String::from_utf8(request.encode_to_vec())
            .unwrap_or_else(|_| "<invalid UTF-8>".to_string()),
    );
    err_details.set_resource_info(resource_type, resource_name, "", "");
    err_details
}
rust/bin/agent/src/handler/insert.rs (1)

68-68: Fix typo in error message.

There's a typo in the error message: "Incombatible" should be "Incompatible" and "detedted" should be "detected".

-                    "Insert API Incombatible Dimension Size detedted",
+                    "Insert API Incompatible Dimension Size detected",
rust/bin/agent/src/handler/update.rs (2)

Line range hint 190-215: Track unimplemented streaming methods.

The streaming methods are marked with todo!(). These should be tracked for future implementation.

Would you like me to create issues to track the implementation of these streaming methods? The issues would cover:

  • StreamUpdate implementation
  • MultiUpdate implementation
  • UpdateTimestamp implementation

41-152: Optimize lock duration for better concurrency.

The write lock is held for the entire duration of the operation, including error handling and response generation. Consider reducing the lock scope to improve concurrency.

Example optimization pattern:

let dimension_size = {
    let s = self.s.read().await;
    s.get_dimension_size()
};
// Validate dimensions without holding the lock
if vec.vector.len() != dimension_size {
    // Handle error...
}
// Only acquire write lock for the actual update
{
    let mut s = self.s.write().await;
    s.update(uuid.clone(), vec.vector.clone(), config.timestamp)?;
}
// Generate response without holding the lock

Also applies to: 41-189, 41-147

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d6f9cbc and 259bae6.

📒 Files selected for processing (8)
  • rust/bin/agent/src/handler/insert.rs (1 hunks)
  • rust/bin/agent/src/handler/object.rs (1 hunks)
  • rust/bin/agent/src/handler/remove.rs (1 hunks)
  • rust/bin/agent/src/handler/search.rs (2 hunks)
  • rust/bin/agent/src/handler/update.rs (1 hunks)
  • rust/bin/agent/src/handler/upsert.rs (1 hunks)
  • rust/bin/agent/src/main.rs (1 hunks)
  • rust/libs/algorithm/src/lib.rs (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
  • rust/bin/agent/src/handler/search.rs
  • rust/bin/agent/src/handler/object.rs
🧰 Additional context used
📓 Learnings (1)
rust/bin/agent/src/handler/upsert.rs (1)
Learnt from: datelier
PR: vdaas/vald#2807
File: rust/bin/agent/src/handler/search.rs:48-48
Timestamp: 2025-01-22T06:18:47.326Z
Learning: When converting Protobuf-encoded request data to String in Vald, use `String::from_utf8(...).unwrap_or_else(|_| "<invalid UTF-8>".to_string())` instead of `unwrap()` to handle invalid UTF-8 gracefully.
⏰ Context from checks skipped due to timeout of 90000ms (4)
  • GitHub Check: check-format-diff
  • GitHub Check: build / build
  • GitHub Check: Analyze (go)
  • GitHub Check: Cloudflare Pages
🔇 Additional comments (4)
rust/bin/agent/src/main.rs (3)

155-156: ⚠️ Potential issue

Handle errors properly in search method

Using unwrap() on self.index.search(...) can cause a panic if an error occurs. Propagate the error instead.

Apply this diff:

 let vec = self
     .index
     .search(vector.as_slice(), k as usize, radius, epsilon)
-    .unwrap();
+    .map_err(Error::from)?;

Likely invalid or redundant comment.


128-131: ⚠️ Potential issue

Handle errors properly in insert method

The append method may return an error. Using unwrap() can cause a panic. Propagate the error instead.

Apply this diff:

 fn insert(&mut self, _uuid: String, vector: Vec<f32>, _ts: i64) -> Result<(), Error> {
-    let _i = self.index.append(vector.as_slice()).unwrap();
+    self.index.append(vector.as_slice()).map_err(Error::from)?;
     Ok(())
 }

Likely invalid or redundant comment.


142-143: ⚠️ Potential issue

Handle errors properly in remove method

Using unwrap() on self.index.remove(id) may cause a panic if an error occurs. Propagate the error instead.

Apply this diff:

 fn remove(&mut self, _uuid: String, _ts: i64) -> Result<(), Error> {
     // convert uuid to id
     let id = 1;
-    self.index.remove(id).unwrap();
+    self.index.remove(id).map_err(Error::from)?;
     Ok(())
 }

Likely invalid or redundant comment.

rust/bin/agent/src/handler/update.rs (1)

56-61: Review error message content for sensitive information.

The error details include raw request data which might contain sensitive information. Consider sanitizing or limiting the information included in error messages.

Also applies to: 83-85, 106-109

rust/bin/agent/src/main.rs Show resolved Hide resolved
rust/bin/agent/src/main.rs Show resolved Hide resolved
rust/bin/agent/src/main.rs Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants