Skip to content

Commit

Permalink
OpenTelemetry for Eclipse Dirigible (#4451)
Browse files Browse the repository at this point in the history
* Add Spring Boot Admin (SBA)

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* add more comments

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* enable liveness and readiness

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* increase waiting time in CsvimIT.java

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* code formatting

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* OpenTelemetry spring boot setup

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* update test controller

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* update service name

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* enable actuator metrics report

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* otel profile and otel meter example

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* do not call GlobalOpenTelemetry

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* add micrometer counter

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* add custom span with remote call

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* add manual span via code not annotations

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* add query param as span attribute

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* add span with failure

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* add call with httpclient + create otel module

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* add repository calls to the example

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* add trace for scripts execution

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* add OpenTelmetry stack setup

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* adapt tests

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* loki cleanup, opensearch using docker profile only

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* add Dirigible sample project

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* export traces to debug as well

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* update README.md

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* less verbose debug exporter

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* update README.md, update sample project

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* add job details to the job execution span

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* enable flowable actuator

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* export flowable metrics based on data in flowable actuator endpoint

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* add traces for BPM tasks execution

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* more details in js and ts endpoints

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* remove span names for the endpoints

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* add traces for the synchronizers

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* rebase with master

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* fix flowable close

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* update headers

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* fix local tests execution

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* add custom quartz metrics

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* wip - camel metrics and traces

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* code formatting

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* delete CamelTelemetryConfigurator.java

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* add span for synchronizers span

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* properties cleanup

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* fix SecurityIT.java

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* cleanup and change attribute names - align with the others

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* cleanup

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* remove starter vs agent files

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* remove test REST

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* remove comment

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* update README

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>

* Update README.md

---------

Signed-off-by: Iliyan Velichkov <velichkov.iliyan@gmail.com>
  • Loading branch information
iliyan-velichkov authored Nov 29, 2024
1 parent a00ec55 commit 68e9168
Show file tree
Hide file tree
Showing 77 changed files with 3,261 additions and 502 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
package org.eclipse.dirigible;

import de.codecentric.boot.admin.server.config.EnableAdminServer;
import org.apache.camel.opentelemetry.starter.CamelOpenTelemetry;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration;
Expand All @@ -21,6 +22,7 @@
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.web.client.RestTemplate;

@CamelOpenTelemetry
@EnableAdminServer
@SpringBootApplication(exclude = {DataSourceAutoConfiguration.class, DataSourceTransactionManagerAutoConfiguration.class,
HibernateJpaAutoConfiguration.class, JdbcTemplateAutoConfiguration.class})
Expand Down
33 changes: 33 additions & 0 deletions build/application/src/main/resources/logback.xml
Original file line number Diff line number Diff line change
Expand Up @@ -77,4 +77,37 @@

<logger name="net.snowflake.client" level="ERROR"/>

<!-- Configuration for the 'open-telemetry' Spring profile -->
<springProfile name="open-telemetry">
<appender name="OpenTelemetry"
class="io.opentelemetry.instrumentation.logback.appender.v1_0.OpenTelemetryAppender">
<captureExperimentalAttributes>false</captureExperimentalAttributes>
<captureCodeAttributes>true</captureCodeAttributes>
<captureMarkerAttribute>true</captureMarkerAttribute>
<captureKeyValuePairAttributes>true</captureKeyValuePairAttributes>
<captureLoggerContext>true</captureLoggerContext>
<captureMdcAttributes>*</captureMdcAttributes>
</appender>

<logger name="org.eclipse.dirigible" level="INFO" additivity="false">
<appender-ref ref="STDOUT" />
<appender-ref ref="FILE_CORE" />
<appender-ref ref="ConsoleLoggingAppender" />
<appender-ref ref="OpenTelemetry"/>
</logger>

<logger name="app" level="INFO" additivity="false">
<appender-ref ref="STDOUT" />
<appender-ref ref="FILE_APPS" />
<appender-ref ref="ConsoleLoggingAppender" />
<appender-ref ref="OpenTelemetry"/>
</logger>

<root level="INFO">
<appender-ref ref="STDOUT" />
<appender-ref ref="FILE_BASE" />
<appender-ref ref="ConsoleLoggingAppender" />
<appender-ref ref="OpenTelemetry"/>
</root>
</springProfile>
</configuration>
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
#!/bin/sh
# fail the whole script if any command bellow fails
set -e

Expand All @@ -11,4 +12,4 @@ esbuild $(find . -iname '*.ts' -not -iname '*.d.ts') '--out-extension:.js=.mjs'
esbuild $(find . -iname '*.ts' -not -iname '*.d.ts') --sourcemap=inline --outdir=dist/cjs --format=cjs --target=es2022

# build dts
tsc --emitDeclarationOnly --outDir dist/dts
tsc --emitDeclarationOnly --outDir dist/dts
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,6 @@
*/
package org.eclipse.dirigible.components.api.test;

import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.result.MockMvcResultHandlers.print;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;

import org.eclipse.dirigible.components.engine.javascript.service.JavascriptService;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
Expand All @@ -27,28 +22,36 @@
import org.springframework.test.web.servlet.MockMvc;
import org.springframework.web.context.WebApplicationContext;

import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.get;
import static org.springframework.test.web.servlet.result.MockMvcResultHandlers.print;
import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.status;

@WithMockUser
@ExtendWith(SpringExtension.class)
@SpringBootTest
@AutoConfigureMockMvc
@ComponentScan(basePackages = {"org.eclipse.dirigible.components.*"})
public class APIAssertTest {

@Autowired
protected WebApplicationContext wac;
@Autowired
private JavascriptService javascriptService;

@Autowired
private MockMvc mockMvc;

@Autowired
protected WebApplicationContext wac;

// @Autowired
// private FilterChainProxy springSecurityFilterChain;
//
// @Autowired
// private IRepository repository;


@SpringBootApplication
static class TestConfiguration {
}

@Test
public void successful() throws Exception {
// javascriptService.handleRequest("test", "successful.js", null, null, false);
Expand All @@ -72,8 +75,4 @@ public void failed() throws Exception {
.endsWith("Assertion 'assertTrue' failed"));
}
}

@SpringBootApplication
static class TestConfiguration {
}
}
8 changes: 6 additions & 2 deletions components/core/core-base/pom.xml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

Expand All @@ -20,11 +20,15 @@
<groupId>org.eclipse.dirigible</groupId>
<artifactId>dirigible-commons-helpers</artifactId>
</dependency>
<dependency>
<groupId>org.eclipse.dirigible</groupId>
<artifactId>dirigible-components-engine-open-telemetry</artifactId>
</dependency>
</dependencies>

<properties>
<license.header.location>../../../licensing-header.txt</license.header.location>
<parent.pom.folder>../../../</parent.pom.folder>
</properties>

</project>
</project>
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@
*/
package org.eclipse.dirigible.components.base.artefact.topology;

import io.opentelemetry.api.trace.Span;
import io.opentelemetry.instrumentation.annotations.WithSpan;
import org.eclipse.dirigible.components.base.artefact.ArtefactPhase;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
Expand All @@ -34,7 +36,12 @@ public class TopologicalDepleter<T extends TopologicallyDepletable> {
* @param flow the flow
* @return the list
*/
@WithSpan
public Set<T> deplete(Set<T> list, ArtefactPhase flow) {
Span span = Span.current();
span.setAttribute("flow", flow.getValue());
span.setAttribute("artifacts.count", list.size());

Set<T> depletables = new HashSet<>();
depletables.addAll(list);
int count = depletables.size();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,29 +9,64 @@
*/
package org.eclipse.dirigible.components.base.synchronizer;

import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;
import org.eclipse.dirigible.components.base.artefact.Artefact;
import org.eclipse.dirigible.components.base.artefact.ArtefactLifecycle;
import org.eclipse.dirigible.components.base.artefact.ArtefactPhase;
import org.eclipse.dirigible.components.base.artefact.topology.TopologyWrapper;
import org.eclipse.dirigible.components.base.spring.BeanProvider;
import org.eclipse.dirigible.components.base.tenant.TenantContext;
import org.eclipse.dirigible.components.base.tenant.TenantResult;
import org.eclipse.dirigible.components.open.telemetry.OpenTelemetryProvider;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.nio.file.Path;
import java.nio.file.attribute.BasicFileAttributes;
import java.text.ParseException;
import java.util.List;

/**
* The Class BaseSynchronizer.
*
*/
public abstract class BaseSynchronizer<A extends Artefact, ID> implements Synchronizer<A, ID> {

/** The Constant logger. */
private static final Logger logger = LoggerFactory.getLogger(BaseSynchronizer.class);

@Override
public final List<A> parse(String location, byte[] content) throws ParseException {
Tracer tracer = OpenTelemetryProvider.get()
.getTracer("eclipse-dirigible");

Span span = tracer.spanBuilder(getSynchronizerSpanPrefix() + "parse_execution")
.startSpan();

try (Scope scope = span.makeCurrent()) {
span.setAttribute("location", location);

return parseImpl(location, content);

} catch (RuntimeException e) {
span.recordException(e);
span.setStatus(io.opentelemetry.api.trace.StatusCode.ERROR, "Exception occurred during synchronization");

throw e;
} finally {
span.end();
}
}

private String getSynchronizerSpanPrefix() {
String synchronizerClassName = this.getClass()
.getSimpleName();
return "synchronizer_" + synchronizerClassName + "_";
}

protected abstract List<A> parseImpl(String location, byte[] content) throws ParseException;

/**
* Complete.
*
Expand All @@ -41,6 +76,27 @@ public abstract class BaseSynchronizer<A extends Artefact, ID> implements Synchr
*/
@Override
public final boolean complete(TopologyWrapper<A> wrapper, ArtefactPhase flow) {
Tracer tracer = OpenTelemetryProvider.get()
.getTracer("eclipse-dirigible");

Span span = tracer.spanBuilder(getSynchronizerSpanPrefix() + "complete_execution")
.startSpan();

try (Scope scope = span.makeCurrent()) {
addSpanAttributes(span, wrapper, flow);
return completeInternal(wrapper, flow);

} catch (RuntimeException e) {
span.recordException(e);
span.setStatus(io.opentelemetry.api.trace.StatusCode.ERROR, "Exception occurred during synchronization");

throw e;
} finally {
span.end();
}
}

private boolean completeInternal(TopologyWrapper<A> wrapper, ArtefactPhase flow) {
A artefact = wrapper.getArtefact();
ArtefactLifecycle lifecycle = artefact.getLifecycle();

Expand All @@ -60,7 +116,7 @@ public final boolean complete(TopologyWrapper<A> wrapper, ArtefactPhase flow) {

return results.stream()
.map(TenantResult::getResult)
.allMatch(r -> Boolean.TRUE.equals(r));
.allMatch(Boolean.TRUE::equals);
}

/**
Expand Down Expand Up @@ -92,12 +148,46 @@ protected boolean isMultitenantArtefact(A artefact) {
*/
protected abstract boolean completeImpl(TopologyWrapper<A> wrapper, ArtefactPhase flow);

private void addSpanAttributes(Span span, TopologyWrapper<A> wrapper, ArtefactPhase phase) {
addSpanAttributes(wrapper.getArtefact(), span);

span.setAttribute("phase", phase.getValue());
}

private void addSpanAttributes(A artefact, Span span) {
span.setAttribute("synchronizer", this.getClass()
.getName());
span.setAttribute("artefact.key", artefact.getKey());
}

/**
* Cleanup.
*
* @param artefact the artefact
*/
public final void cleanup(A artefact) {
Tracer tracer = OpenTelemetryProvider.get()
.getTracer("eclipse-dirigible");

Span span = tracer.spanBuilder(getSynchronizerSpanPrefix() + "cleanup_execution")
.startSpan();

try (Scope scope = span.makeCurrent()) {
addSpanAttributes(artefact, span);

cleanupInternal(artefact);

} catch (RuntimeException e) {
span.recordException(e);
span.setStatus(io.opentelemetry.api.trace.StatusCode.ERROR, "Exception occurred during synchronization");

throw e;
} finally {
span.end();
}
}

private void cleanupInternal(A artefact) {
if (!multitenantExecution() || !isMultitenantArtefact(artefact)) {
logger.debug("[{} will cleanup artefact [{}]", this, artefact);
cleanupImpl(artefact);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ public boolean isAccepted(String type) {
* @throws ParseException the parse exception
*/
@Override
public List<ExtensionPoint> parse(String location, byte[] content) throws ParseException {
protected List<ExtensionPoint> parseImpl(String location, byte[] content) throws ParseException {
ExtensionPoint extensionPoint = JsonHelper.fromJson(new String(content, StandardCharsets.UTF_8), ExtensionPoint.class);
Configuration.configureObject(extensionPoint);
extensionPoint.setLocation(location);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ public boolean isAccepted(String type) {
* @throws ParseException the parse exception
*/
@Override
public List<Extension> parse(String location, byte[] content) throws ParseException {
protected List<Extension> parseImpl(String location, byte[] content) throws ParseException {
Extension extension = JsonHelper.fromJson(new String(content, StandardCharsets.UTF_8), Extension.class);
Configuration.configureObject(extension);
extension.setLocation(location);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,8 @@ protected SimpleScheduleBuilder getSchedule() {
*/
@Override
protected String getJobKey() {
return "SynchronizationJobDetail";
return this.getClass()
.getSimpleName();
}

/**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ public boolean isAccepted(String type) {
* @throws ParseException the parse exception
*/
@Override
public List<Csvim> parse(String location, byte[] content) throws ParseException {
protected List<Csvim> parseImpl(String location, byte[] content) throws ParseException {
Csvim csvim = JsonHelper.fromJson(new String(content, StandardCharsets.UTF_8), Csvim.class);
Configuration.configureObject(csvim);
csvim.setLocation(location);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ public boolean isAccepted(String type) {
* @throws ParseException the parse exception
*/
@Override
public List<DataSource> parse(String location, byte[] content) throws ParseException {
protected List<DataSource> parseImpl(String location, byte[] content) throws ParseException {
DataSource datasource = JsonHelper.fromJson(new String(content, StandardCharsets.UTF_8), DataSource.class);
Configuration.configureObject(datasource);
datasource.setLocation(location);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ public boolean isAccepted(String type) {
* @throws ParseException the parse exception
*/
@Override
public List<Entity> parse(String location, byte[] content) throws ParseException {
protected List<Entity> parseImpl(String location, byte[] content) throws ParseException {
Entity entity = new Entity();
entity.setLocation(location);
entity.setName(Paths.get(location)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ public boolean isAccepted(String type) {
* @throws ParseException the parse exception
*/
@Override
public List<Schema> parse(String location, byte[] content) throws ParseException {
protected List<Schema> parseImpl(String location, byte[] content) throws ParseException {
final Schema schema = parseSchema(location, new String(content, StandardCharsets.UTF_8));
Configuration.configureObject(schema);
schema.setLocation(location);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ public boolean isAccepted(String type) {
* @throws ParseException the parse exception
*/
@Override
public List<Table> parse(String location, byte[] content) throws ParseException {
protected List<Table> parseImpl(String location, byte[] content) throws ParseException {
Table table = JsonHelper.fromJson(new String(content, StandardCharsets.UTF_8), Table.class);
Configuration.configureObject(table);
table.setLocation(location);
Expand Down
Loading

0 comments on commit 68e9168

Please sign in to comment.