SPIKE - Wazuh Puppet

[teddytpc1]

Objective
https://github.com/wazuh/internal-devel-requests/issues/1319

Description

As part of the DevOps overhaul objective, we need to research, analyze alternatives, and design how to implement the following changes for the Wazuh Puppet module:

1. Package Usage:

   • Transition to using Wazuh packages instead of repositories.

2. Configuration Standardization:

   • Ensure the Wazuh Puppet module uses out-of-the-box Wazuh configurations for installations.

3. Testing Improvements:

   • Enhance DevOps-owned testing for the Puppet module.
   • GitHub workflows must support testing with production, pre-release, and development packages.

4. Documentation Updates:

   • Define and document prerequisites for deploying Wazuh using Puppet.
   • Use the Wazuh Kubernetes documentation style as a reference for clarity and completeness.

Implementation restrictions

• Testing Environment:
  • Tests must be implemented using GitHub Actions (GHA).
• Compatibility:
  • Ensure compatibility with supported operating systems and Puppet versions.
• Minimal Maintenance:
  • Focus on reducing complexity and ensuring the module is easy to maintain.

Plan

1. Research & Analysis:

   • Review the current Wazuh Puppet module and identify required changes for transitioning to package usage.
   • Analyze and incorporate the best practices for using out-of-the-box configurations.

2. Configuration Updates:

   • Modify the Puppet module to use Wazuh packages instead of repositories.
   • Implement default configurations to align with Wazuh out-of-the-box settings.

3. Testing Enhancements:

   • Develop GHA workflows to test the module with production, pre-release, and development packages.
   • Include validations for successful installation, configuration application, and deployment.

4. Documentation Updates:

   • Clearly define the prerequisites for using the Wazuh Puppet module.
   • Update the documentation to follow the Wazuh Kubernetes documentation style, emphasizing simplicity and user-friendly instructions.

[vcerenu]

I've been researching the tools we have for creating a function to install Wazuh from packages downloaded from a URL

Example of a function to detect the OS family where we are trying to install

case $facts['os']['family'] {
'RedHat', 'Debian': {
$pkg_manager = $facts['os']['family'] ? {
'RedHat' => 'rpm',
'Debian' => 'dpkg',
}

Example of checking installed package, so as not to repeat the same process

$is_installed = $version ? {
undef => "${pkg_manager} -q ${package_name}",
default => "${pkg_manager} -q ${package_name}-${version}",
}

Package download example:

exec { "download_${package_name}":
command => "/usr/bin/curl -o ${download_path} ${url}",
creates => $download_path,
path => ['/usr/bin', '/bin'],
unless => $is_installed,
}

Package installation example and removal of downloaded package

exec { "install_${package_name}":
command => $pkg_manager ? {
'rpm' => "rpm -Uvh ${download_path}",
'dpkg' => "dpkg -i ${download_path}",
},
path => ['/usr/bin', '/bin'],
require => Exec["download_${package_name}"],
unless => $is_installed,
}

file { $download_path:
ensure => absent,
require => Exec["install_${package_name}"],
}

These examples help determine the possibility of creating a function that does not have problems with the Puppet job, which is always maintaining a configuration and not simply running a workflow, so it needs to effectively detect that the necessary packages have already been installed and not repeat the process due to problems detecting the installed product.

These examples correspond to a Linux installation, but these same cases can be extended to Windows.

[vcerenu]

Conclusions

1. Research & Analysis:
   • Review the current Wazuh Puppet module and identify required changes for transitioning to package usage.
   • Analyze and incorporate the best practices for using out-of-the-box configurations.

I have been reviewing the installation possibilities of Wazuh components. This process could be done with a class created by us for this purpose, in which we can perform all the necessary steps and be able to analyze the data from the Puppet agent in order to use the correct package for each OS distribution and corresponding architecture.
Within this class we will have a file with the productive URLs from where to obtain each of the packages, with which through the analysis of Pupper facts we can know which package corresponds to the OS and architecture where we are going to install. We should generate the necessary code logic to correctly verify that the product is already installed and with the configuration that we want to perform.

2. Configuration Updates:

• Modify the Puppet module to use Wazuh packages instead of repositories.
• Implement default configurations to align with Wazuh out-of-the-box settings.

Regarding the topic corresponding to the configuration of Wazuh, we currently use templates to be able to adapt the necessary parameters for the configuration. We need to change this, but we need to run several tests to see if Puppet doesn't interfere if we modify other parameters within the configuration file itself, and to know that once the necessary configuration is applied without a template, it doesn't repeat this process every time we notice a change made to a parameter in the same file that we are not configuring with the module, since we will only perform out-of-the-box configurations and the other parameters must be modified by the user.

3. Testing Enhancements:

• Develop GHA workflows to test the module with production, pre-release, and development packages.
• Include validations for successful installation, configuration application, and deployment.

Regarding the tests, we currently have a workflow that tests the installation of an AIO environment within a github runner only, we don't have tests of multi-node environments or agent installation, and the tests are performed on production or pre-release environments, which would require adding staging environments. The workflow logic will need to change according to the changes made in the module, since with the new classes that need to be created the deployment method will be modified, but they serve as a good basis to start with these tests.
In the tests we will have to adapt the use of development packages, checking if the input parameters need the commits of the branches to be tested or if we should send the URLs of the packages to be tested.

4. Documentation Updates:

• Clearly define the prerequisites for using the Wazuh Puppet module.
• Update the documentation to follow the Wazuh Kubernetes documentation style, emphasizing simplicity and user-friendly instructions.

Regarding the documentation, as mentioned in the previous spikes, we should remove all the Puppet installation information from our documentation, which we normally do not maintain, and add links to the official documentation, which removes the need to update the changes that the Puppet developers make to their products and the compatibility of the same with the different systems.
In addition, we should catalog the list of class parameters so that the user has a better understanding of the configurations that can be made for the installation and configuration of the Wazuh components.
We can keep the deployment examples that we currently have to provide examples of how to use them, but clarifying that they are base installations and for testing purposes.

[vcerenu]

Update

I was doing an analysis regarding the change from the installation by a package manager to the installation by downloading the package that corresponds to the version of Wazuh, operating system used and architecture.

Using Puppet facts we can delimit which package we will use, which allows us to have variables to verify among the available files of all the components of Wazuh, which is the correct one to install. Later we have to verify that the product is installed correctly, to know if it is necessary or not to repeat the installation.
After the previous reviews, we can proceed to install the corresponding package and then verify if the package is correctly installed, if the subsequent configurations were made and if the product is running correctly, in order to determine if it is necessary to perform the configuration again.

To determine the correct URL to download, you can use the json file parsing that Puppet has, which would save us the use of a dependency within the VM of the Puppet agent where we are installing and with this determine which URL is the correct one.

  # Determine the package type (rpm or deb) based on the operating system family
  $package_type = $facts['os']['family'] ? {
    /(RedHat|Suse|Amazon|/ => 'rpm',
    /(Debian|Ubuntu)/           => 'deb',
    default                             => fail("Unsupported operating system"),
  }

  $package_arch = $facts['os']['architecture']

  # Download the JSON file
  file { $json_file:
    ensure => file,
    source => '<JSON_FILE_URL>',
    mode   => '0644',
  }

  # Read and parse the JSON
  $packages = parsejson(file($json_file))

  # Filter the correct package
  $selected_package = $packages.filter |$pkg| {
    ($pkg['package_type'] == $package_type) and
    ($pkg['package_arch'] == $package_arch) and
    ($pkg['version'] == $desired_version) and
    ($pkg['name'] == $package_name)
  }

I'm still working on the code for the installation and configuration of the components

[vcerenu]

Update

I've been working on customizing the configuration files, which will change their operation because we will remove all the templates we are currently using for this purpose.

I've been working on the best way to configure these files and I've been checking that the files that we will configure with the Wazuh module classes are Yaml or XML files. With this information I've been checking how to generate these changes, taking into account that replacing configurations in Yaml and XML files is quite different, so they require different logic.

Using the same class I worked on the logic to modify both files, but it is necessary to pass as a parameter what type of file it is, the path where it is located, an array with the different strings to configure and an additional parameter for the XML files where taking the configuration sent, it replaces the current one in the file or adds it directly.

class modify_config_file (
  String $config_file,                     # Path to the configuration file
  Array[String] $config_lines,             # Array of configurations to modify or add
  Enum['yaml', 'xml'] $file_type,          # File type: yaml or xml
  Boolean $replace_all = true              # Replace entire content (default true)
) {

  # Convert all configuration keys to lowercase
  $normalized_config_lines = $config_lines.map |$line| {
    regsubst($line, '^([^:]+):', '\l\1:', 'G') # Convert parameter names to lowercase
  }

  if $replace_all {
    # Replace the entire file content
    replace { "replace_all_${config_file}":
      path    => $config_file,
      pattern => '.*', # Match the entire file content
      replace => $normalized_config_lines.join("\n"), # Replace with normalized lines
    }
  } else {
    # Add configurations at the end of the file
    $normalized_config_lines.each |$line| {
      replace { "add_line_${line}":
        path             => $config_file,
        pattern          => "^${regsubst($line, '^([^:]+):.*$', '\\1', 'G')}.*$", # Match the key
        replace          => $line, # Replace the line if it exists
        append_on_no_match => true, # Add the line if it does not exist
      }
    }
  }

  # Specific handling for XML files
  if $file_type == 'xml' {
    $normalized_config_lines.each |$line| {
      $key = regsubst($line, '^<([^>]+)>.*$', '\\1', 'G') # Extract the XML tag
      replace { "modify_xml_${key}":
        path    => $config_file,
        pattern => "<${key}>.*?</${key}>", # Match the complete XML block
        replace => $line, # Replace the entire block if it exists
        append_on_no_match => true, # Add the XML block if it does not exist
      }
    }
  }
}

[vcerenu]

Update

I am currently working on a way to install packages by URL so that Puppet maintains the idempotence of the classes to be executed and does not generate problems.
I have also worked on the possibility of using both a file of URLs of production or pre-release packages, as well as custom versions to be able to deploy development packages.

[vcerenu]

Update

I've been working on creating a class that allows installing packages, getting the URLs from a file, and allows installing from downloaded packages.

I've had a number of problems creating this class, since Puppet has several limitations and requires maintaining idempotency in its execution.

One of the limitations that has caused me the most problems is that the variables that receive values ​​before execution cannot be modified at runtime, so modifying variables so that routes are modified according to variables that can be loaded or not to generate different actions within the class (for example using the URL file from a local medium or downloading it from the Internet) becomes quite complicated, since checking that a file exists in a remote path (for example receiving the URL file from a path of a Puppet module) cannot be done simply and requires generating separate processes that review each of the steps.
We also have some problems when executing certain commands in the OS, which we need to do because by not having an apt or yum repository, the download of the file, installation and subsequent configuration of the components without any template, break the idempotence of Puppet, since checking that the command has already been executed and does not need to be executed again, requires many checks that we have to program manually.

I am currently doing a kind of test that checks the path within the Archive module of Puppet where we can leave the URLs file, if it does not find it, it downloads the file from a URL that is loaded by default, then it verifies according to variables that are loaded when running and Puppet facts regarding the agent where it is run, the correct URL of the package to install on that agent. The correct agent is downloaded and then it tries to install, which at the moment generates an error when using the dpkg binary in Ubuntu:

#
# Class to install Wazuh product
#
# @param package_name The name of the package to be installed.
# @param wazuh_version The version of the Wazuh package to be installed.
# @param prod_url The URL to download the package list from.
# @param source_url The Puppet URL to download the package list from.
# @param destination Destination path for the downloaded file
# @param rpm_based Regex for RPM-based OS families
# @param deb_based Regex for DEB-based OS families
# @param download_dir parameter for download directory
class wazuh::install_product (
  String $package_name = 'wazuh-manager',
  String $wazuh_version = '4.9.2',
  String $prod_url = 'https://devops-wazuh-artifacts-pub.s3.us-west-1.amazonaws.com/devops-overhaul/packages_url.txt',
  String $source_url = 'puppet:///modules/archive/packages_url.txt',
  String $destination = '/tmp/packages_url.txt',
  String $rpm_based = 'RedHat|Suse|Amazon|OracleLinux|AlmaLinux|Rocky',
  String $deb_based = 'Debian|Ubuntu|Mint|Kali|Raspbian',
  String $download_dir = '/tmp',
) {
  # Determine the package type (rpm or deb) based on the OS family.
  if $facts['os']['family'] =~ Regexp($rpm_based) {
    $package_type = 'rpm'
    $check_command = "/bin/rpm -q ${package_name}" # Command to check if the package is installed (RPM)
  } elsif $facts['os']['family'] =~ Regexp($deb_based) {
    $package_type = 'deb'
    $check_command = "/usr/bin/dpkg-query -l ${package_name} | grep '^ii'" # Command to check if the package is installed (DEB)
  } else {
    fail("Unsupported OS family: ${facts['os']['family']}") # Fail if the OS family is not supported
  }

  # Determine the package architecture.
  $package_arch = $facts['os']['architecture'] ? {
    'x86_64' => 'amd64',
    default  => $facts['os']['architecture'],
  }

  # Construct the package filename.
  $package_pattern = "${package_name}-${wazuh_version}-${package_arch}.${package_type}"

  # Download the file using the archive resource.
  archive { $destination:
    ensure => present,
    source => $source_url,
    path   => $destination,
  }

  exec { 'download_packages_url_from_url':
    command   => "/usr/bin/curl --fail --location -o ${destination} ${prod_url}",
    path      => ['/usr/bin', '/bin'],
    creates   => $destination, # is created when the file does not exist
    unless    => "test -f ${destination}", # not executed if file exists.
    logoutput => true,
  }

  # Find the package URL in the downloaded file.
  exec { 'filter_and_extract_url':
    command   => "/usr/bin/sed -n '/^${package_pattern}:/p' ${destination} | /usr/bin/awk -F': ' '{print \$2}' > ${destination}.bak && mv ${destination}.bak ${destination}",
    path      => ['/usr/bin', '/bin'],
    logoutput => true,
  }

  notify { "Extracted package URL: ${destination}": }

  if $destination {
    exec { 'download_file_from_url':
      command   => "tr -d '\r' < ${destination} | xargs /usr/bin/curl -o '${download_dir}/${package_pattern}'",
      path      => ['/usr/bin', '/bin'],
      logoutput => true,
    }

    # Determine the install command based on the package type.
    $install_command = $package_type ? {
      'rpm' => "/bin/rpm -ivh ${download_dir}/${package_pattern}",
      'deb' => "dpkg -i --debug=72200 ${download_dir}/${package_pattern} || apt-get install -f -y",
    }

    notify { "Command to install: ${install_command}": }

    # Install the package.
    exec { "install_${package_pattern}":
      command   => $install_command,
      path      => ['/bin', '/usr/bin'],
      onlyif    => "dpkg-deb --info ${download_dir}/${package_pattern}",
      unless    => $check_command, # Only install if the package is not already installed
      logoutput => true,
    }

    # Remove the downloaded package file.
    file { "${download_dir}/${package_pattern}":
      ensure => absent,
      force  => true,
    }
  } else {
    warning("URL for ${package_pattern} not found in ${destination}")
  }
}

The error generated does not have much data regarding why it is generated, and executing the command manually does not generate errors, so I continue investigating what problem it may have to execute the installation.

I still need more checks so that the class maintains idempotence, not needing to execute any process of the same if it has already been executed correctly previously, which with the installation of a package can be simple, but with the configuration of the components through commands and without using templates can become quite complicated.

[vcerenu]

Update

I've been checking for errors in the use of the new classes created in other more general classes.

I currently have the wazuh::install_product class created, which installs any component and requires the component name parameter to be added, which allows it to be reused in the classes created for each component, but I'm not able to use it to deploy because it doesn't find this new class. I'm investigating within the documentation what may be happening with this problem.

[vcerenu]

Update

I continued analyzing the wazuh::modify_config_file class, which contained errors when configuring XML files mainly.

The main problem with the wazuh::modify_config_file class is that we do not have a resource or module in Puppet that allows the modification of XML files while maintaining idempotence. Investigating on this topic I found the Argeus module, which should allow the modification of these XML files, but they require a configuration of the same module before being able to correctly analyze and modify the files, which requires much more analysis and development.
I tried to perform procedures with OS commands for the modification of XML files while maintaining idempotence, but it does not allow it, it is necessary to use some process that maintains it by itself.

With yaml style files it is much easier, since the file_line resource allows these files to be modified while maintaining idempotence, but I cannot do this with XML files.

I continue with the investigation

[vcerenu]

Update

Performing a partial analysis of the installation and configuration process of Wazuh components using Puppet, we have found some problems to achieve a correct installation process:

1. Since there is no package repository, the installation is done manually and the packages do not have fixed URLs, so the installation process of downloading packages and installing them have problems with idempotence, since they also have to be able to be configured manually in each installation separately, causing the possibility of being able to determine if the process that is configured to run needs to be executed or not, determining if the steps have already been previously executed or if they will generate changes in order to determine if they need to be executed again or not.

2. The restriction on the use of templates causes the possibility of maintaining the idempotence of the configuration customization process to generate problems in order to determine if the changes that are requested to be made need to be executed or not, either because they have already been executed previously or because they do not generate changes. Puppet tasks need to determine before their execution if they need to be executed, so that when the server wants to apply the changes again, I can skip that task if it is not necessary and the use of scripts on the OS with the execution of commands in bash, do not have the necessary traceability to be able to determine this.
   For the changes in the configuration files I was trying to make a short PoC with Augeas, a tool that allows the manipulation of configuration files in a safer and more reliable way, and that according to its capabilities, allows to maintain the idempotence of the process during the execution of the Puppet class, but I could not finish it in these days, so we will analyze it better in the planning phase.

[vcerenu]

Update
According to the spike performed, I have determined a series of general tasks that we must perform for the correct update of the wazuh-puppet repository and its respective documentation:

Steps:

• Create a class or function that allows you to install RPM or DEB packages that will be obtained by downloading them from a public bucket, since this process is currently being carried out by a Puppet module that we have as a dependency and that automates the entire installation process.
• Create a class or function that allows you to modify configuration files. It is important to verify the feasibility of creating this class since Puppet does not allow you to configure changes in files dynamically and for these cases templates are used.
• Modify the current classes of the module to adopt the new classes created. The resources used will no longer be useful because we will not have repositories, so it is necessary to modify these resources with the class that installs from downloaded packages. In addition, we must eliminate the templates that we are using, so we must implement the class that customizes configuration files.

[vcerenu]

An issue has been created for task control for the Wazuh Puppet MVP v5.0.0:

• MVP - Puppet deployment #1220