Testing

Testing and Quality Assurance

In our platform, we employ a multi-faceted approach to ensure the highest standards of code quality and functionality through rigorous testing and static analysis. Below is an outline of our practices:

Static Analysis with Pylint

We use Pylint to enforce coding standards and ensure consistent code quality across the platform. Our strict policy requires that no Pylint warnings are accepted. If a particular rule needs to be bypassed, developers must explicitly disable it with a comment, such as # pylint: disable=R0903, and provide a justification. This approach encourages adherence to best practices while allowing flexibility when necessary.

Example:

class DataStorage:  # pylint: disable=R0903

Automated Testing with Pytest

Automated testing is a cornerstone of our quality assurance strategy. We utilize pytest to conduct unit tests, focusing on both the factory creation of components and the specific implementations of instances. These tests ensure that the components function as expected in isolation and within the system.

Example Pytest Script for ChatModel:

import pytest
from self_serve_platform.chat.model import ChatModel
from self_serve_platform.chat.models.langchain_chat_openai import LangChainChatOpenAIModel
from self_serve_platform.chat.models.langchain_azure_chat_openai import LangChainAzureChatOpenAIModel

@pytest.mark.parametrize("config, expected_class", [
    (
        {
            "type": "LangChainChatOpenAI",
            "model_name": "gpt-3",
            "api_key": "your_api_key"
        },
        LangChainChatOpenAIModel
    ),
    (
        {
            "type": "LangChainAzureChatOpenAI",
            "api_key": "your_api_key",
            "azure_deployment": "your_deployment",
            "api_version": "your_version",
            "endpoint": "api_endpoint"
        },
        LangChainAzureChatOpenAIModel
    )
])
def test_create(config, expected_class):
    """
    Test the create factory method to ensure it returns instances of the correct classes
    based on the configuration provided.
    """
    model_instance = ChatModel.create(config)
    assert isinstance(model_instance, expected_class)

@pytest.fixture
def langchain_chatopenai_model_config():
    """
    Mockup LangChain_ChatOpenAI model
    """
    return {
        "type": "LangChainChatOpenAI",
        "model_name": "gpt-3",
        "api_key": "your_api_key"
    }
@patch.object(ChatOpenAI, 'invoke')
def test_langchain_chatopenaimodel_invoke(mock_invoke, langchain_chatopenai_model_config):  # pylint: disable=W0621
    """
    Test the invoke method of LangChainChatOpenAIModel to verify it returns a result
    with the correct status, content, and metadata.
    """
    # Create a mock response object
    mock_response = MagicMock()
    mock_response.content = "Mocked response"
    mock_response.response_metadata = {"key": "value"}
    # Set the mock to return the mock response object
    mock_invoke.return_value = mock_response
    factory = ChatModel.create(langchain_chatopenai_model_config)
    result = factory.invoke("Hello, world!")
    assert result.status == "success"
    assert result.content == "Mocked response"
    assert result.metadata == {"key": "value"}  # Update this to match expected metadata
    mock_invoke.assert_called_once_with("Hello, world!")

This script verifies that the factory method correctly initializes and returns instances of the appropriate model types based on the given configuration. It also tests the specific behavior of each model's methods to ensure correct setup and functionality.

Manual Testing with Gherkin

For end-to-end (E2E) testing, we employ manual testing using Gherkin syntax. This method allows us to thoroughly validate the overall functionality of our tools and web applications in real-world scenarios. Manual testing is particularly valuable for verifying complex interactions and user interfaces that might not be fully covered by automated tests.

Example Gherkin Scenarios:

Feature: Manual Testing of Web Applications and Tools

Scenario: Verify the manifest endpoint
    Given the Temperature API tool is running
    When I navigate to its manifest endpoint
    Then I should manually verify that the manifest output matches the expected structure and content

Scenario: Verify the temperature retrieval endpoint
    Given the Temperature API tool is running
    When I submit a valid longitude and latitude to the temperature retrieval endpoint
    Then I should manually verify that the output correctly reflects the current temperature for the specified location