{% image "./src/assets/images/team/" + member.image, "" %}
diff --git a/src/pages/index.njk b/src/pages/index.njk
index 1b1ac80..b236cf4 100644
--- a/src/pages/index.njk
+++ b/src/pages/index.njk
@@ -1,101 +1,86 @@
---
-layout: base
-permalink: /index.html
-title: 'The Scalable Way'
-description: 'Eleventy starter using modern CSS, fluid type, fluid spacing, flexible layout and progressive enhancement.'
-bodyClass: home
-
hero:
- title: 'Empowering Data Platforms with Precision and Scale'
- subtitle: |
- **The Scalable Way** is a professional services agency focusing on "as a code" data platform infrastructure, analytics and data science workflows.
+ title: Empowering Analytics Teams with Scalable & Automated Data Platforms
+ subtitle: "**The Scalable Way** is a professional services agency bringing software engineering practices to data platforms in order to achieve higher scalability, cost effectiveness, usability and reliability."
button_text: Learn more
- button_target: #
-
+ button_target: ""
intro:
- title: 'Is your data tech stack working with you or against you?'
- content: |
+ title: Is your data tech stack working with you or against you?
+ content: |-
Most data professionals are stuck navigating complex analytics workflows. They deal with fragmented access, manual maintenance, “ad hoc” pipelines, and weak security and governance—largely because they rely on a patchwork of poorly integrated tools instead of a cohesive data platform.
In contrast, high-performing analytics teams prioritize scalable solutions, borrowing best practices from software development. These solutions are automated, provide streamlined workflows, ensure a strong developer experience, are secure by default, and consistently deliver reliable production insights. A well-designed data platform is essential to achieve these outcomes.
-
highlight_one:
- title: 'We help data analytics teams & departments achieve greater efficiency and more robust solutions in production'
- subtitle: 'by helping with developing infrastructure, ingestion, transformation, and documentation as code; training your teams on the enhanced development workflow; servicing recurring data ops and data platform needs.'
-
+ title: We help data analytics teams & departments achieve greater efficiency and more robust solutions in production
+ subtitle: by helping with developing infrastructure, ingestion, transformation, and documentation as code; training your teams on the enhanced development workflow; servicing recurring data ops and data platform needs.
highlight_two:
title1: The ultimate data platform goal?
title2: Stuff just works!
- content: 'Unfortunately, data analytics departments I meet often don’t end up where they had hoped. They struggle to build reliable data pipelines. They don’t manage to automate, and because of that they have problems onboarding users and excessive manual work.'
-
-foundations:
+ content: Unfortunately, data analytics departments I meet often don’t end up where they had hoped. They struggle to build reliable data pipelines. They don’t manage to automate, and because of that they have problems onboarding users and excessive manual work.
+foundations_titles:
title1: The 5 data platform
title2: foundations
- keys:
- - title: Automation
- content: 'with solid CI/CD and documentation'
- - title: Developer Experience
- content: 'Intuitive tools, clear APIs, and robust frameworks'
- - title: Scalability
- content: 'Growing usage without exponentially growing tech debt and costs'
- - title: Governance & Security
- content: 'Data governance, networking, and access'
- - title: Data Readiness
- content: 'Clean, consistent, and enriched data for analysis'
-
+foundations_elements:
+ - title: Automation
+ content: with solid CI/CD and documentation
+ - title: Developer Experience
+ content: Intuitive tools, clear APIs, and robust frameworks
+ - title: Scalability
+ content: Growing usage without exponentially growing tech debt and costs
+ - title: Governance & Security
+ content: Data governance, networking, and access
+ - title: Data Readiness
+ content: Clean, consistent, and enriched data for analysis
about:
title: About us
- content: |
+ content: |-
We believe in automation and good developer experience!
With a combined 30+ years of experience in infrastructure, data platforms, and data engineering, we are on a mission to help data analytics teams build more robust and scalable solutions.
-
-why:
- title: 'Why do clients come to us?'
- intro: >
- Analytics Departments and Teams typically suffer from one or more of these **11 challenges**
- keys:
- - title: Unscalable.
- content: 'Lacking strong tech foundations to build a scalable and long term solution'
- - title: Unclear reference architecture.
- content: 'Struggling to define a target architecture and to address specific analytics pain points'
- - title: Unclear security model.
- content: 'Not being sure what security and access management is needed to fulfil company policies'
- - title: Shaky production.
- content: 'Lacking a proper split of dev and prod environments and thereby making production less stable'
- - title: Increasing tech debt.
- content: 'Lacking a robust code review process to avoid accumulation of tech debt'
- - title: Inadequate networking configuration.
- content: 'Networking designed with applications in mind, making it difficult to stay secure and fast when developing analytical capabilities'
- - title: Manual operations.
- content: 'Time intensive, error prone, tedious daily manual work'
- - title: Slow data ingestions.
- content: 'Experiencing difficulties ingesting data from many applications and APIs'
- - title: Struggling with best practices.
- content: 'Lacking guidance in what tools to choose and avoiding making expensive mistakes'
- - title: Difficulty in retaining key talent.
- content: 'Experiencing challenges in keeping team members motivated'
- - title: Difficulty in onboarding talent.
- content: 'Experiencing challenges when onboarding new analysts to the complex web of tools in use'
-
-services:
- title: 'Our Services'
- items:
- - title: Development
- prefix: Data Platform
- description: >
- Build scalable, reliable, and efficient data infrastructure with modern "as code" practices, enabling seamless data integration, transformation, and analysis.
-
- - title: Operations
- prefix: Data Platform
- description: >
- Ensure continuous, reliable data platform performance through proactive monitoring, maintenance, and automated operational workflows.
-
+whyus_intro:
+ title: Why do clients come to us?
+ content: Analytics Departments and Teams typically suffer from one or more of these **11 challenges**
+whyus_elements:
+ - title: Unscalable.
+ content: Lacking strong tech foundations to build a scalable and long term solution
+ - title: Unclear reference architecture.
+ content: Struggling to define a target architecture and to address specific analytics pain points
+ - title: Unclear security model.
+ content: Not being sure what security and access management is needed to fulfil company policies
+ - title: Shaky production.
+ content: Lacking a proper split of dev and prod environments and thereby making production less stable
+ - title: Increasing tech debt.
+ content: Lacking a robust code review process to avoid accumulation of tech debt
+ - title: Inadequate networking configuration.
+ content: Networking designed with applications in mind, making it difficult to stay secure and fast when developing analytical capabilities
+ - title: Manual operations.
+ content: Time intensive, error prone, tedious daily manual work
+ - title: Slow data ingestions.
+ content: Experiencing difficulties ingesting data from many applications and APIs
+ - title: Struggling with best practices.
+ content: Lacking guidance in what tools to choose and avoiding making expensive mistakes
+ - title: Difficulty in retaining key talent.
+ content: Experiencing challenges in keeping team members motivated
+ - title: Difficulty in onboarding talent.
+ content: Experiencing challenges when onboarding new analysts to the complex web of tools in use
+our_services:
+ title: Our Services
+our_services_items:
+ - title: Development
+ prefix: Data Platform
+ description: Build scalable, reliable, and efficient data infrastructure with modern "as code" practices, enabling seamless data integration, transformation, and analysis.
+ - title: Operations
+ prefix: Data Platform
+ description: Ensure continuous, reliable data platform performance through proactive monitoring, maintenance, and automated operational workflows.
+description: Eleventy starter using modern CSS, fluid type, fluid spacing, flexible layout and progressive enhancement.
+bodyClass: home
blog:
- title: 'Our Insights'
- intro: 'Selected Articles. Check our blog for more.'
+ title: Our Insights
+ intro: Selected Articles. Check our blog for more.
+layout: base
+permalink: /index.html
+title: The Scalable Way
---
-
@@ -142,13 +127,13 @@ blog:
- {{ foundations.title1 }} - {{ foundations.title2 }} + {{ foundations_titles.title1 }} + {{ foundations_titles.title2 }}
{% svg "brand/brick" %}
-
- {% for key in foundations.keys %}
+ {% for key in foundations_elements %}
-
{{ loop.index }}
@@ -169,7 +154,7 @@ blog:{{ about.content | markdownFormat | safe }}- {% for member in team %} + {% for member in admin.team %}{% image "./src/assets/images/team/" + member.image, "" %} @@ -184,12 +169,12 @@ blog:
- {{ why.title }}
- {{ why.intro | markdownFormat | safe }} +{{ whyus_intro.title }}
+ {{ whyus_intro.content | markdownFormat | safe }}-
- {% for key in why.keys %}
+ {% for key in whyus_elements %}
- {{ key.title }}
- {{ key.content }} @@ -203,9 +188,9 @@ blog:
{% svg "brand/arrows" %} - {{ services.title }}
+{{ our_services.title }}
- {% for item in services.items %} + {% for item in our_services_items %}{{ item.prefix }} {{ item.title }}
diff --git a/src/pages/legal.md b/src/pages/legal.md index bc97310..9503a31 100644 --- a/src/pages/legal.md +++ b/src/pages/legal.md @@ -7,6 +7,4 @@ description: Imprint infos layout: page --- -Edit your details in `_data/personal.yaml` - {{ personal.address }} diff --git a/src/posts/2024/2024-12-21-prefect-1-to-2-migration.md b/src/posts/2024/2024-12-21-prefect-1-to-2-migration.md index a044341..8a02b1d 100644 --- a/src/posts/2024/2024-12-21-prefect-1-to-2-migration.md +++ b/src/posts/2024/2024-12-21-prefect-1-to-2-migration.md @@ -1,14 +1,16 @@ --- -title: 'Prefect 1.0 to 2.0 Migration: Motivation, Process and Workflow' -description: 'This article outlines the motivation and steps for migrating from Prefect 1 to Prefect 2. Additionally, it provides guidance on how to coordinate work between teams to ensure the migration is performed with zero downtime' +title: "Prefect 1.0 to 2.0 Migration: Motivation, Process and Workflow" date: 2024-12-21 +author: Alessio Civitillo +description: This article outlines the motivation and steps for migrating from Prefect 1 to Prefect 2. Additionally, it provides guidance on how to coordinate work between teams to ensure the migration is performed with zero downtime +tags: + - prefect --- - - Prefect 1.x was the original version of Prefect, a data orchestration platform designed to streamline the development, scheduling, and monitoring of workflows. It used a monolithic server that acted as an API layer and a task scheduler. Prefect 2.x brings several architectural changes that offer a more flexible, scalable, and cloud-native solution for orchestrating workflows. With the focus on Kubernetes, containers, and a more modular API, Prefect 2.x aims to be a better fit for modern cloud infrastructure and CI/CD pipelines. + ## Why Migrate? Scalability: Prefect 2.x offers better scalability through dynamic task mapping and execution in various environments (Kubernetes, Docker). @@ -16,6 +18,7 @@ Scheduling: Prefect 2.x introduces more powerful scheduling and retry logic. Cloud-Native: Enhanced integration with cloud-native tools, such as Kubernetes, and an improved deployment model. Flexibility: Granular control over deployments and orchestration. + 1. Key Differences Between Prefect 1 and Prefect 2 2.1 Architecture Changes Prefect 1.x used a single monolithic server and agent system for managing workflows and executions. @@ -65,6 +68,7 @@ def my_task(): with Flow("my_flow") as flow: my_task() ``` + Prefect 2.x: @@ -112,6 +116,7 @@ o In the Prefect UI, navigate to the storage block and set the correct product Set Notifications: + 1. Define Trigger Events: Use Prefect's event triggers or monitor task/flow state transitions. Specifically: o Failed state for deployments (e.g., using a StateHandler or flow/task callbacks). @@ -123,7 +128,7 @@ o Configure these in the Prefect UI under Blocks. o Example: Slack Notification Block to send messages on failures. -4. Common Migration Issues and How to Resolve Them +3. Common Migration Issues and How to Resolve Them 4.1 Task State Issues In Prefect 1.x, task states were relatively simple. In Prefect 2.x, the task state model is more granular, including states like Pending, Running, Failed, Success, and custom states. @@ -148,14 +153,13 @@ The transition from the old scheduler to the new scheduling system in Prefect 2. Solution: Double-check the cron syntax, and ensure that the IntervalSchedule and CronSchedule are configured correctly. -5. Conclusion +4. Conclusion Migrating from Prefect 1.x to Prefect 2.x can be an involved process, but the flexibility, scalability, and performance improvements offered by Prefect 2.x are well worth the effort. With better integration into cloud-native tools, an object-oriented approach to flows and tasks, and improved scheduling and retry logic, Prefect 2.x provides a future-proof solution for orchestrating modern data flows. - - 2 Zero-Downtime Migration Workflow Between Teams + 1. Planning Phase 1.1 Define the Migration Scope Identify the flows to be migrated, including their dependencies and critical components. @@ -196,10 +200,10 @@ o Use tools such as Prometheus, Grafana, or Datadog to track metrics. 3.1 Continuous Validation and Monitoring · Run Validation Tests (samples, cols name): After migrating, validate that all flows and services are functioning as expected. This includes end-to-end tests for data pipelines, integrations, and error handling. For example, within the ingestion layer, where data flows from the data source to ADLS, a parallel architecture can be implemented: Prefect 1 pointing to the Production ADLS, and Prefect 2 pointing to the Development ADLS. There are two levels of validation: -1. Path Validation +2. Path Validation o The paths where data is loaded into the Production and Development ADLS must be identical. o Note: If the file name includes a timestamp, remove the date portion before performing the comparison. -2. Data Frame Validation +3. Data Frame Validation o Ensure that the number of rows in the Development ADLS matches the order of the values extracted from the Production ADLS. o Verify that all columns in the Production data are present in the Development data. Any differences can be attributed to new columns added in the updated flow version. @@ -240,4 +244,3 @@ Once Prefect 2 is fully operational, stop Prefect 1 workflows and switch to Pref 6 Conclusion Zero-downtime migration requires meticulous planning, careful execution, and proactive monitoring. By adopting strategies such as incremental migration, dual-environment deployment, real-time monitoring, and automated rollback mechanisms, teams can successfully migrate workflows without impacting day-to-day operations. As recap, the chart shown below represents a state-flow of the migration process: -