Merge branch 'main' into fix/fields_lookup_stored_spec

build-tools-internal/src/test/groovy/org/elasticsearch/gradle/internal/doc/RestTestsFromDocSnippetTaskSpec.groovy

@@ -454,11 +454,11 @@ Requires the `manage_ml` cluster privilege. This privilege is included in the
 
 `<job_id>`::
 (Required, string)
-include::{es-repo-dir}/ml/ml-shared.asciidoc[tag=job-id-anomaly-detection]
+include::{es-ref-dir}/ml/ml-shared.asciidoc[tag=job-id-anomaly-detection]
 
 `<snapshot_id>`::
 (Required, string)
-include::{es-repo-dir}/ml/ml-shared.asciidoc[tag=snapshot-id]
+include::{es-ref-dir}/ml/ml-shared.asciidoc[tag=snapshot-id]
 
 [[ml-update-snapshot-request-body]]
 == {api-request-body-title}
@@ -470,7 +470,7 @@ The following properties can be updated after the model snapshot is created:
 
 `retain`::
 (Optional, Boolean)
-include::{es-repo-dir}/ml/ml-shared.asciidoc[tag=retain]
+include::{es-ref-dir}/ml/ml-shared.asciidoc[tag=retain]
 
 
 [[ml-update-snapshot-example]]

docs/changelog/106133.yaml

@@ -1,5 +1,19 @@
 pr: 106133
-summary: Add an optimised vector distance function for aarch64
+summary: Add a SIMD (Neon) optimised vector distance function for int8
 area: Search
 type: enhancement
 issues: []
+highlight:
+  title: A SIMD (Neon) optimised vector distance function for merging int8 Scalar Quantized vectors has been added
+  body: |-
+    An optimised int8 vector distance implementation for aarch64 has been added.
+    This implementation is currently only used during merging.
+    The vector distance implementation outperforms Lucene's Pamana Vector
+    implementation for binary comparisons by approx 5x (depending on the number
+    of dimensions). It does so by means of SIMD (Neon) intrinsics compiled into a
+    separate native library and link by Panama's FFI. Comparisons are performed on
+    off-heap mmap'ed vector data.
+    Macro benchmarks, SO_Dense_Vector with scalar quantization enabled, shows
+    significant improvements in merge times, approximately 3 times faster. 
+  notable: true
+

docs/changelog/107178.yaml

@@ -0,0 +1,5 @@
+pr: 107178
+summary: "Add support for Azure OpenAI embeddings to inference service"
+area: Machine Learning
+type: feature
+issues: [ ]

docs/changelog/107272.yaml

@@ -0,0 +1,5 @@
+pr: 107272
+summary: "ESQL: extend BUCKET with spans"
+area: ES|QL
+type: enhancement
+issues: []

docs/changelog/107328.yaml

@@ -0,0 +1,7 @@
+pr: 107328
+summary: "ESQL: Fix missing refs due to pruning renamed grouping columns"
+area: ES|QL
+type: bug
+issues:
+ - 107083
+ - 107166

docs/changelog/107334.yaml

@@ -0,0 +1,5 @@
+pr: 107334
+summary: Adding `cache_stats` to geoip stats API
+area: Ingest Node
+type: enhancement
+issues: []

docs/changelog/107449.yaml

@@ -0,0 +1,5 @@
+pr: 107449
+summary: Leverage ordinals in enrich lookup
+area: ES|QL
+type: enhancement
+issues: []

docs/changelog/107517.yaml

@@ -0,0 +1,5 @@
+pr: 107517
+summary: Add GET `_inference` for all inference endpoints
+area: Machine Learning
+type: enhancement
+issues: []

docs/changelog/107533.yaml

@@ -0,0 +1,5 @@
+pr: 107533
+summary: Add setting for max connections to S3
+area: Snapshot/Restore
+type: enhancement
+issues: []

docs/reference/aggregations/bucket/categorize-text-aggregation.asciidoc

@@ -40,15 +40,15 @@ is an object it has the following properties:
 =====
 `char_filter`::::
 (array of strings or objects)
-include::{es-repo-dir}/ml/ml-shared.asciidoc[tag=char-filter]
+include::{es-ref-dir}/ml/ml-shared.asciidoc[tag=char-filter]
 
 `tokenizer`::::
 (string or object)
-include::{es-repo-dir}/ml/ml-shared.asciidoc[tag=tokenizer]
+include::{es-ref-dir}/ml/ml-shared.asciidoc[tag=tokenizer]
 
 `filter`::::
 (array of strings or objects)
-include::{es-repo-dir}/ml/ml-shared.asciidoc[tag=filter]
+include::{es-ref-dir}/ml/ml-shared.asciidoc[tag=filter]
 =====
 
 `categorization_filters`::

docs/reference/aggregations/bucket/random-sampler-aggregation.asciidoc

@@ -94,6 +94,17 @@ higher sampling rates, the relative error is still low.
 
 NOTE: This represents the result of aggregations against a typical positively skewed APM data set which also has outliers in the upper tail. The linear dependence of the relative error on the sample size is found to hold widely, but the slope depends on the variation in the quantity being aggregated. As such, the variance in your own data may
       cause relative error rates to increase or decrease at a different rate.
+[[random-sampler-consistency]]
+==== Random sampler consistency
+
+For a given `probability` and `seed`, the random sampler aggregation is consistent when sampling unchanged data from the same shard.
+However, this is background random sampling if a particular document is included in the sampled set or not is dependent on current number of segments.
+
+Meaning, replica vs. primary shards could return different values as different particular documents are sampled.
+
+If the shard changes in via doc addition, update, deletion, or segment merging, the particular documents sampled could change, and thus the resulting statistics could change.
+
+The resulting statistics used from the random sampler aggregation are approximate and should be treated as such.
 
 [[random-sampler-special-cases]]
 ==== Random sampling special cases
@@ -105,6 +116,6 @@ for a bucket is `10,000` with `probability: 0.1`, the actual number of documents
 
 An exception to this is <<search-aggregations-metrics-cardinality-aggregation, cardinality aggregation>>. Unique item
 counts are not suitable for automatic scaling. When interpreting the cardinality count, compare it
-to the number of sampled docs provided in the top level `doc_count` within the random_sampler aggregation. It gives 
+to the number of sampled docs provided in the top level `doc_count` within the random_sampler aggregation. It gives
 you an idea of unique values as a percentage of total values. It may not reflect, however, the exact number of unique values
 for the given field.

docs/reference/aggregations/pipeline/inference-bucket-aggregation.asciidoc

@@ -65,23 +65,23 @@ aggregations some options can be overridden for each of the two types of model.
 
 `num_top_feature_importance_values`::
 (Optional, integer)
-include::{es-repo-dir}/ml/ml-shared.asciidoc[tag=inference-config-regression-num-top-feature-importance-values]
+include::{es-ref-dir}/ml/ml-shared.asciidoc[tag=inference-config-regression-num-top-feature-importance-values]
 
 [discrete]
 [[inference-agg-classification-opt]]
 ===== Configuration options for {classification} models
 
 `num_top_classes`::
 (Optional, integer)
-include::{es-repo-dir}/ml/ml-shared.asciidoc[tag=inference-config-classification-num-top-classes]
+include::{es-ref-dir}/ml/ml-shared.asciidoc[tag=inference-config-classification-num-top-classes]
 
 `num_top_feature_importance_values`::
 (Optional, integer)
-include::{es-repo-dir}/ml/ml-shared.asciidoc[tag=inference-config-classification-num-top-feature-importance-values]
+include::{es-ref-dir}/ml/ml-shared.asciidoc[tag=inference-config-classification-num-top-feature-importance-values]
 
 `prediction_field_type`::
 (Optional, string)
-include::{es-repo-dir}/ml/ml-shared.asciidoc[tag=inference-config-classification-prediction-field-type]
+include::{es-ref-dir}/ml/ml-shared.asciidoc[tag=inference-config-classification-prediction-field-type]
 
 
 [[inference-bucket-agg-example]]

docs/reference/alias.asciidoc

@@ -308,7 +308,7 @@ POST _aliases
 ----
 // TEST[s/^/PUT _data_stream\/logs-nginx.access-prod\nPUT _data_stream\/logs-my_app-default\n/]
 
-include::{es-repo-dir}/indices/aliases.asciidoc[tag=write-index-defaults]
+include::{es-ref-dir}/indices/aliases.asciidoc[tag=write-index-defaults]
 
 TIP: We recommend using data streams to store append-only time series data. If
 you need to update or delete existing time series data, you can perform update or delete operations

docs/reference/analysis/tokenfilters/stemmer-override-tokenfilter.asciidoc

@@ -48,7 +48,7 @@ Where the file looks like:
 
 [source,stemmer_override]
 --------------------------------------------------
-include::{es-test-dir}/cluster/config/analysis/stemmer_override.txt[]
+include::{elasticsearch-root}/docs/src/test/cluster/config/analysis/stemmer_override.txt[]
 --------------------------------------------------
 
 You can also define the overrides rules inline:

docs/reference/api-conventions.asciidoc

@@ -205,18 +205,18 @@ on a remote cluster is currently not supported. For example, this will throw an
 Multi-target APIs that can target indices support the following query
 string parameters:
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=index-ignore-unavailable]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=index-ignore-unavailable]
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=allow-no-indices]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=allow-no-indices]
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=expand-wildcards]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=expand-wildcards]
 
 The defaults settings for the above parameters depend on the API being used.
 
 Some multi-target APIs that can target indices also support the following query
 string parameter:
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=ignore_throttled]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=ignore_throttled]
 
 NOTE: APIs with a single target, such as the <<docs-get,get document API>>, do
 not support multi-target syntax.

docs/reference/cat/alias.asciidoc

@@ -39,19 +39,19 @@ wildcards (`*`). To retrieve all aliases, omit this parameter or use `*` or
 [[cat-alias-api-query-params]]
 ==== {api-query-parms-title}
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=http-format]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=http-format]
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=cat-h]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=cat-h]
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=help]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=help]
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=local]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=local]
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=cat-s]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=cat-s]
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=cat-v]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=cat-v]
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=expand-wildcards]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=expand-wildcards]
 
 [[cat-alias-api-example]]
 ==== {api-examples-title}

docs/reference/cat/allocation.asciidoc

@@ -30,26 +30,26 @@ and their disk space.
 [[cat-allocation-api-path-params]]
 ==== {api-path-parms-title}
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=node-id]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=node-id]
 
 [[cat-allocation-api-query-params]]
 ==== {api-query-parms-title}
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=bytes]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=bytes]
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=http-format]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=http-format]
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=local]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=local]
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=master-timeout]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=master-timeout]
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=cat-h]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=cat-h]
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=help]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=help]
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=cat-s]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=cat-s]
 
-include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=cat-v]
+include::{es-ref-dir}/rest-api/common-parms.asciidoc[tag=cat-v]
 
 [[cat-allocation-api-response-body]]
 ==== {api-response-body-title}