CVE Consumer webapp

This repository contains the code for the CVE Consumer webapp. The application consumes CVE data from provided kafka topic and inserts it into the database.

Prerequisites

To build and deploy the application locally, you will need the following:

• Go (version 1.22 or later)
• Docker
• PostgreSQL (version 15 or later)
• Kafka (version 3.0.0 or later)

Build and Deploy Instructions

Set Environment Variables

Before building and running the application, you need to set the following environment variables:

# Kafka Configuration
export KAFKA_BROKERS="localhost:9092,localhost:9093,localhost:9094" # Defaults to localhost:9092,localhost:9093,localhost:9094
export KAFKA_TOPIC="cve"                                            # Defaults to cve

# Kafka Consumer Configuration
export KAFKA_CONSUMER_GROUP_ID="cve" # Optional - Defaults to cve
export KAFKA_MAX_SIZE="10e6"         # Optional - Defaults to 10e6

# Kafka Scram SHA-512 Credentials (Skips auth if not provided)
export KAFKA_USERNAME=""                           # Optional - Defaults to ""
export KAFKA_PASSWORD=""                           # Optional - Defaults to ""

Build and Run the Application

To build the application, execute the following command:

go build -o cve-consumer cmd/main.go

To run the application, execute the following command:

./cve-consumer

Application Details

The application will:

• Consumes the messages from the provided KAFKA_TOPIC and parses the data to JSON.
• Store the CVE JSONs in a PostgreSQL database using the JSON data type along with some metadata.
• Support versioning of CVEs to handle updates after initial publication.
• Handle duplicate records.
• Retries Kafka & Database connections on failure.