Code

feat: make it work for serp#73

Code

Open

JossWhittle wants to merge 152 commits intomainSwanseaUniversityMedical/DARE-Airflow:mainfrom feat/make-it-work-for-serpSwanseaUniversityMedical/DARE-Airflow:feat/make-it-work-for-serpCopy head branch name to clipboard

+2,201-905

Commits

Commits on Apr 10, 2024

• working version - Fast DuckDB

  tekkisse

  committed
  8bde32c

  Copy full SHA for 8bde32c

• bit of tidying up

  tekkisse

  committed
  2b4e4c8

  Copy full SHA for 2b4e4c8

• auto run DAG & UTF-8 testing

  tekkisse

  committed
  3fa3709

  Copy full SHA for 3fa3709

• check in

  tekkisse

  committed
  07c1263

  Copy full SHA for 07c1263

• FIX: install minio to base image

  tekkisse

  committed
  5d0d76f

  Copy full SHA for 5d0d76f

• multi part download

  tekkisse

  committed
  934d2ca

  Copy full SHA for 934d2ca

Commits on Apr 11, 2024

• working version

  tekkisse

  committed
  1e816b1

  Copy full SHA for 1e816b1

• use updated docker container

  tekkisse

  committed
  d65fe71

  Copy full SHA for d65fe71

• tracking

  tekkisse

  committed
  24201f2

  Copy full SHA for 24201f2

Commits on Apr 12, 2024

• parquete file size tracking

  tekkisse

  committed
  1eb3973

  Copy full SHA for 1eb3973

Commits on Apr 13, 2024

• MVP

  tekkisse

  committed
  5fdca92

  Copy full SHA for 5fdca92

Commits on Apr 16, 2024

• FEAT: add jinja to docker image

  tekkisse

  committed
  18eb439

  Copy full SHA for 18eb439

• compute ledger working

  tekkisse

  committed
  b17010e

  Copy full SHA for b17010e

Commits on Apr 17, 2024

• tweaking

  tekkisse

  committed
  42896ad

  Copy full SHA for 42896ad

• turn debug off

  tekkisse

  committed
  35ae96b

  Copy full SHA for 35ae96b

• track end iceberg table

  tekkisse

  committed
  5079d8d

  Copy full SHA for 5079d8d

Commits on Apr 21, 2024

• add notifitcation

  tekkisse

  committed
  19fda0d

  Copy full SHA for 19fda0d

Commits on Apr 23, 2024

• make build

  tekkisse

  committed
  08d8393

  Copy full SHA for 08d8393

• .

  tekkisse

  committed
  d78c907

  Copy full SHA for d78c907

• made cooler

  tekkisse

  committed
  d0ae2a4

  Copy full SHA for d0ae2a4

Commits on Apr 25, 2024

• sort out docker compose start order

  tekkisse

  committed
  e2e7d96

  Copy full SHA for e2e7d96

Commits on Apr 28, 2024

• regex single for list

  tekkisse

  committed
  8b5fb30

  Copy full SHA for 8b5fb30

• master switch

  tekkisse

  committed
  c225b44

  Copy full SHA for c225b44

Commits on Apr 29, 2024

• working version hitting assetv3

  tekkisse

  committed
  08f68a2

  Copy full SHA for 08f68a2

Commits on Apr 30, 2024

• few error fixed

  tekkisse

  committed
  db63ec8

  Copy full SHA for db63ec8

• fix

  tekkisse

  committed
  e5e3ec0

  Copy full SHA for e5e3ec0

• added action

  tekkisse

  committed
  4f3ac35

  Copy full SHA for 4f3ac35

• create, append, replace options

  tekkisse

  committed
  f36a61a

  Copy full SHA for f36a61a

• MVP1

  tekkisse

  committed
  8f466bc

  Copy full SHA for 8f466bc

• use proxy

  tekkisse

  committed
  39a6b02

  Copy full SHA for 39a6b02

• bug fixes

  tekkisse

  committed
  9c49204

  Copy full SHA for 9c49204

Commits on May 1, 2024

• bug fixing

  tekkisse

  committed
  fe30b55

  Copy full SHA for fe30b55

• tidy up

  tekkisse

  committed
  b71fbde

  Copy full SHA for b71fbde

Commits on May 2, 2024

• turn proxy server on

  tekkisse

  committed
  4f6af6a

  Copy full SHA for 4f6af6a

• change default

  tekkisse

  committed
  5001c7c

  Copy full SHA for 5001c7c

• change logging

  tekkisse

  committed
  2cc7c8d

  Copy full SHA for 2cc7c8d

• .

  tekkisse

  committed
  246aedb

  Copy full SHA for 246aedb

Commits on May 7, 2024

• refactor functions

  tekkisse

  committed
  e516ad0

  Copy full SHA for e516ad0

Commits on May 8, 2024

• refactored into modules

  tekkisse

  committed
  c023310

  Copy full SHA for c023310

• full refactor

  tekkisse

  committed
  e2a6de0

  Copy full SHA for e2a6de0

• separate DAG's

  tekkisse

  committed
  b01c54d

  Copy full SHA for b01c54d

Commits on May 9, 2024

• extra logging

  tekkisse

  committed
  e314b8a

  Copy full SHA for e314b8a

• ISSUE WITH SERIALISING DICT OBJECTS

  tekkisse

  committed
  44e9d4c

  Copy full SHA for 44e9d4c

• tweaking

  tekkisse

  committed
  6185b0c

  Copy full SHA for 6185b0c

• add manual SED

  tekkisse

  committed
  8845877

  Copy full SHA for 8845877

Commits on May 10, 2024

• turn proxy on

  tekkisse

  committed
  5e4514c

  Copy full SHA for 5e4514c

• deployable version

  tekkisse

  committed
  4c7ff41

  Copy full SHA for 4c7ff41

• a

  tekkisse

  committed
  ce997a4

  Copy full SHA for ce997a4

• cope with daft hard coded values

  tekkisse

  committed
  5e29347

  Copy full SHA for 5e29347

• dam need to get more setting into config files

  tekkisse

  committed
  9561e33

  Copy full SHA for 9561e33

Commits on May 12, 2024

• bucket default configurations if not dataset configuration

  tekkisse

  committed
  d371af7

  Copy full SHA for d371af7

Commits on May 13, 2024

• allow for manual overide

  tekkisse

  committed
  cb689a4

  Copy full SHA for cb689a4

Commits on May 28, 2024

• fix(container): use base container with our root ca baked in

  alee-x

  committed
  ba70254

  Copy full SHA for ba70254

• update gitignore

  alee-x

  committed
  3306be8

  Copy full SHA for 3306be8

• fix(container): add chi root ca to trust store

  alee-x

  committed
  16f8b44

  Copy full SHA for 16f8b44

• small changes

  tekkisse

  committed
  c6fc7cb

  Copy full SHA for c6fc7cb

• Merge branch 'feat/make-it-work-for-serp' of https://github.com/SwanseaUniversityMedical/DARE-Airflow into feat/make-it-work-for-serp

  tekkisse

  committed
  c8bac8c

  Copy full SHA for c8bac8c

Commits on May 29, 2024

• switch to usding redis settings from Airflow connection

  tekkisse

  committed
  812a204

  Copy full SHA for 812a204

• remove db number (redis) from connection

  tekkisse

  committed
  d48b618

  Copy full SHA for d48b618

Commits on Jun 3, 2024

• fix: when getting the redis connection, if auth is enabled then use the auth for the redis client

  alee-x

  committed
  29f55d8

  Copy full SHA for 29f55d8

Commits on Jun 9, 2024

• change tracking info

  tekkisse

  committed
  23a98ea

  Copy full SHA for 23a98ea

• correct logic

  tekkisse

  committed
  77efc18

  Copy full SHA for 77efc18

• correct regex

  tekkisse

  committed
  2265d77

  Copy full SHA for 2265d77

Commits on Jun 10, 2024

• increase power of SED

  tekkisse

  committed
  7fc8829

  Copy full SHA for 7fc8829

• SED validation

  tekkisse

  committed
  f8abb94

  Copy full SHA for f8abb94

• add code to pull schema and max out

  tekkisse

  committed
  f15e6c9

  Copy full SHA for f15e6c9

• fix stupid error

  tekkisse

  committed
  9a58c83

  Copy full SHA for 9a58c83

• change way table refernced

  tekkisse

  committed
  6014996

  Copy full SHA for 6014996

• .

  tekkisse

  committed
  666c6be

  Copy full SHA for 666c6be

• remove schema name from table name

  tekkisse

  committed
  f94b955

  Copy full SHA for f94b955

• trying to make this work

  tekkisse

  committed
  03523bc

  Copy full SHA for 03523bc

• try again

  tekkisse

  committed
  baa9327

  Copy full SHA for baa9327

• correction

  tekkisse

  committed
  96365e3

  Copy full SHA for 96365e3

• try

  tekkisse

  committed
  d882d6a

  Copy full SHA for d882d6a

• .

  tekkisse

  committed
  b66fd5e

  Copy full SHA for b66fd5e

• deal with varchar

  tekkisse

  committed
  4eccdeb

  Copy full SHA for 4eccdeb

• correct if statement

  tekkisse

  committed
  be3ec1e

  Copy full SHA for be3ec1e

• better solution

  tekkisse

  committed
  353a83f

  Copy full SHA for 353a83f

• python error

  tekkisse

  committed
  ae253ea

  Copy full SHA for ae253ea

• whoops - missed that

  tekkisse

  committed
  6021796

  Copy full SHA for 6021796

• move marker

  tekkisse

  committed
  771d101

  Copy full SHA for 771d101

• fix/turn debug off

  tekkisse

  committed
  967a8c3

  Copy full SHA for 967a8c3

Commits on Jun 11, 2024

• turn debug back on

  tekkisse

  committed
  f8110d1

  Copy full SHA for f8110d1

• correct error

  tekkisse

  committed
  c4bbf66

  Copy full SHA for c4bbf66

• add time datatype into mapping

  tekkisse

  committed
  43cc96b

  Copy full SHA for 43cc96b

• try time stamp

  tekkisse

  committed
  6de1e4e

  Copy full SHA for 6de1e4e

• leave file for debigging

  tekkisse

  committed
  947e828

  Copy full SHA for 947e828

• rename DAGS

  tekkisse

  committed
  3638818

  Copy full SHA for 3638818

• switch duckdb to disk instead of memory

  tekkisse

  committed
  8d7c27c

  Copy full SHA for 8d7c27c

• tweaking

  tekkisse

  committed
  34c5703

  Copy full SHA for 34c5703

• lower case for file extension

  tekkisse

  committed
  0c88e90

  Copy full SHA for 0c88e90

Commits on Jun 12, 2024

• feat(dags-container): upgrade duckdb to v1.0.0, upgrade base container to use the same as DLM-dags (#76)

  alee-x

  authored
  6b0c604

  Copy full SHA for 6b0c604

• Merge branch 'main' into feat/make-it-work-for-serp

  alee-x

  authored
  f640682

  Copy full SHA for f640682

Commits on Jun 14, 2024

• fix: use airflow connection schema keys for the aws connection where possible

  alee-x

  committed
  d9a18a7

  Copy full SHA for d9a18a7

• fix: don't hardcode minio secure/ssl to false because then it doesn't work when we have a https minio

  alee-x

  committed
  7586eeb

  Copy full SHA for 7586eeb

• fix: correctly detect the connection protocol for minio and trino based on the connection details

  alee-x

  committed
  c108861

  Copy full SHA for c108861

• fix: don't hardcode the trino connection as though there no bloody authentication on it!

  alee-x

  committed
  4c4873f

  Copy full SHA for 4c4873f

• fix: we can't have a module with the same name as a package we need to import, also use trino.auth BasicAuthentication

  alee-x

  committed
  8f8a8bb

  Copy full SHA for 8f8a8bb

Commits on Jun 15, 2024

• cope with different etag

  tekkisse

  committed
  beb411f

  Copy full SHA for beb411f

• Merge branch 'feat/make-it-work-for-serp' of https://github.com/SwanseaUniversityMedical/DARE-Airflow into feat/make-it-work-for-serp

  tekkisse

  committed
  33c54d4

  Copy full SHA for 33c54d4

Commits on Jun 26, 2024

• Update docker-compose.yml

  tekkisse

  committed
  f072adc

  Copy full SHA for f072adc

• increase max connections

  tekkisse

  committed
  0040bc8

  Copy full SHA for 0040bc8

• remove auth as does not work with http

  tekkisse

  committed
  2baf179

  Copy full SHA for 2baf179

• options auth for trino

  tekkisse

  committed
  0de5efd

  Copy full SHA for 0de5efd

• correct code

  tekkisse

  committed
  745a1a8

  Copy full SHA for 745a1a8

• up max connections

  tekkisse

  committed
  4646f7f

  Copy full SHA for 4646f7f

• change port for mariadb

  tekkisse

  committed
  8fd0a8f

  Copy full SHA for 8fd0a8f

• switch ports

  tekkisse

  committed
  6761328

  Copy full SHA for 6761328

• .

  tekkisse

  committed
  f91a4e0

  Copy full SHA for f91a4e0

• .

  tekkisse

  committed
  a62c6cc

  Copy full SHA for a62c6cc

• switch hive server

  tekkisse

  committed
  51f8053

  Copy full SHA for 51f8053

• add second postgres

  tekkisse

  committed
  8b2ce46

  Copy full SHA for 8b2ce46

• .

  tekkisse

  committed
  aacedc2

  Copy full SHA for aacedc2

Commits on Jul 5, 2024

• change portal tables

  tekkisse

  committed
  29e8979

  Copy full SHA for 29e8979

• corrections

  tekkisse

  committed
  c26e6cc

  Copy full SHA for c26e6cc

• .

  tekkisse

  committed
  a0af32e

  Copy full SHA for a0af32e

• new table for portal

  tekkisse

  committed
  f504676

  Copy full SHA for f504676

Commits on Aug 2, 2024

• send schema request to rabbitmq

  tekkisse

  committed
  4abcc89

  Copy full SHA for 4abcc89

Commits on Aug 23, 2024

• use external DAGS

  tekkisse

  committed
  0bee53a

  Copy full SHA for 0bee53a

• correction

  tekkisse

  committed
  e2d6d8f

  Copy full SHA for e2d6d8f

• bump airflow build up

  tekkisse

  committed
  551a65e

  Copy full SHA for 551a65e

• real correction :-)

  tekkisse

  committed
  a0c7f29

  Copy full SHA for a0c7f29

Commits on Aug 27, 2024

• extra trino worker

  tekkisse

  committed
  b416746

  Copy full SHA for b416746

• upgrade trino

  tekkisse

  committed
  8df9ad2

  Copy full SHA for 8df9ad2

• change airflow image

  tekkisse

  committed
  8a12026

  Copy full SHA for 8a12026

• extra max threads

  tekkisse

  committed
  676b535

  Copy full SHA for 676b535

Commits on Aug 28, 2024

• up container version

  tekkisse

  committed
  c8dbe31

  Copy full SHA for c8dbe31

Commits on Sep 2, 2024

• changing rabbit

  tekkisse

  committed
  51e4b0a

  Copy full SHA for 51e4b0a

• use latest PR

  tekkisse

  committed
  cafde4b

  Copy full SHA for cafde4b

• shell scripts

  tekkisse

  committed
  bec25db

  Copy full SHA for bec25db

• correct spelling

  tekkisse

  committed
  7be9e0a

  Copy full SHA for 7be9e0a

• add portal details

  tekkisse

  committed
  0ba1348

  Copy full SHA for 0ba1348

• add restart script

  tekkisse

  committed
  130944c

  Copy full SHA for 130944c

• whoops - fix

  tekkisse

  committed
  3ad1bc0

  Copy full SHA for 3ad1bc0

• instructions

  tekkisse

  committed
  2222d2b

  Copy full SHA for 2222d2b

• add start firehose

  tekkisse

  committed
  faca597

  Copy full SHA for faca597

• wire up firehose

  tekkisse

  committed
  ac9503c

  Copy full SHA for ac9503c

Commits on Sep 3, 2024

• correct rabbitmq def

  tekkisse

  committed
  c7fa94b

  Copy full SHA for c7fa94b

• use correct DLM PR

  tekkisse

  committed
  5b447cc

  Copy full SHA for 5b447cc

• use local DAG's

  tekkisse

  committed
  d0a1e30

  Copy full SHA for d0a1e30

Commits on Sep 5, 2024

• a poke

  John-Vaughan

  committed
  c9e5a4b

  Copy full SHA for c9e5a4b

Commits on Sep 25, 2024

• added request_schema_save

  John-Vaughan

  committed
  b7d623c

  Copy full SHA for b7d623c

Commits on Oct 1, 2024

• update

  John-Vaughan

  committed
  156c6cb

  Copy full SHA for 156c6cb

Commits on Oct 8, 2024

• add postgres for nrdav2

  tekkisse

  committed
  c873662

  Copy full SHA for c873662

• Merge branch 'feat/make-it-work-for-serp' of https://github.com/SwanseaUniversityMedical/DARE-Airflow into feat/make-it-work-for-serp

  tekkisse

  committed
  1a4f488

  Copy full SHA for 1a4f488

Commits on Oct 10, 2024

• fix

  John-Vaughan

  committed
  db38ef7

  Copy full SHA for db38ef7

Commits on Oct 16, 2024

• added ledger

  John-Vaughan

  committed
  a3b01f0

  Copy full SHA for a3b01f0

Commits on Oct 21, 2024

• PG_MAX_CONNECTIONS?

  John-Vaughan

  committed
  50c59eb

  Copy full SHA for 50c59eb

• joss stuff

  John-Vaughan

  committed
  8171b49

  Copy full SHA for 8171b49

Commits on Oct 23, 2024

• update

  John-Vaughan

  committed
  046592c

  Copy full SHA for 046592c

Commits on Nov 11, 2024

• add

  John-Vaughan

  committed
  f733ef1

  Copy full SHA for f733ef1

Commits on Nov 12, 2024

• update

  John-Vaughan

  committed
  fe921c7

  Copy full SHA for fe921c7