Skip to content

Commit

Permalink
Add tembo_parquet_s3_fdw.git to Analytics (#81)
Browse files Browse the repository at this point in the history
In order to the data warehouse stack into Analytics, add Tembo's fork of
`tembo_parquet_s3_fdw` to the Analytics stack as required by support
pg_tier. Build the extension in its on `FROM` target and just copy its
files, to avoid installing unnecessary dependencies.

Exclude Postgres 17 from the build, since `parquet_s3_fdw` doesn't
support it yet, and the Analytics stack depends on on Hydra columnar
anyway, and it doesn't support Postgres 17, either.

Also remove pg_duckdb, as it is now available as a Trunk extension.
  • Loading branch information
theory committed Jan 22, 2025
1 parent f9c2075 commit bd378b1
Show file tree
Hide file tree
Showing 2 changed files with 40 additions and 19 deletions.
7 changes: 7 additions & 0 deletions .github/workflows/build_tembo_pg_slim.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -388,7 +388,9 @@ jobs:
CONTAINER_NAME: "analytics-cnpg"
steps:
- uses: actions/checkout@v4
if: ${{ matrix.pg_version != '17' }}
- name: Build Docker images based on conditions
if: ${{ matrix.pg_version != '17' }}
run: |
if [[ "${{ matrix.pg_version }}" == "14" || "${{ matrix.pg_version }}" == "15" ]]; then
echo "Skipping Analytics build for Postgres 14 or 15"
Expand All @@ -398,12 +400,14 @@ jobs:
docker build ./$CONTAINER_NAME --build-arg PG_VERSION=${{ matrix.pg_version }} --build-arg TAG=${{ needs.pre-build.outputs.short_sha }} -t $IMAGE_NAME
shell: bash
- name: Login to Tembo Quay
if: ${{ matrix.pg_version != '17' }}
uses: docker/login-action@v2
with:
registry: ${{ secrets.QUAY_REPOSITORY }}
username: ${{ secrets.QUAY_USER_TEMBO }}
password: ${{ secrets.QUAY_PASSWORD_TEMBO }}
- name: Push to Quay
if: ${{ matrix.pg_version != '17' }}
shell: bash
run: |
if [[ "${{ matrix.pg_version }}" == "14" || "${{ matrix.pg_version }}" == "15" ]]; then
Expand All @@ -418,14 +422,17 @@ jobs:
docker push ${{ secrets.QUAY_REPOSITORY }}/$IMAGE_NAME-$tag
done
- name: Configure AWS credentials for ECR
if: ${{ matrix.pg_version != '17' }}
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: ${{ secrets.GHA_IAM_ROLE }}
role-session-name: images-gha-docker-build-and-push
aws-region: "us-east-1"
- name: Install awscli
if: ${{ matrix.pg_version != '17' }}
uses: unfor19/install-aws-cli-action@v1
- name: Push to ECR
if: ${{ matrix.pg_version != '17' }}
shell: bash
run: |
if [[ "${{ matrix.pg_version }}" == "14" || "${{ matrix.pg_version }}" == "15" ]]; then
Expand Down
52 changes: 33 additions & 19 deletions analytics-cnpg/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,36 +1,50 @@
ARG PG_VERSION=16
ARG TAG=c19d507
ARG TAG=86080de

FROM quay.io/tembo/standard-cnpg:${PG_VERSION}-${TAG}
FROM quay.io/tembo/standard-cnpg:${PG_VERSION}-${TAG} AS build
USER root

WORKDIR /
WORKDIR /work

# Install build dependencies for parquet_s3_fdw and its dependent libraries
RUN apt-get update && apt-get install -y \
git \
g++ \
automake \
bison \
build-essential \
ccache \
cmake \
ninja-build \
flex \
git \
libboost-all-dev \
libcurl4-openssl-dev \
libevent-dev \
libprotobuf-dev \
libprotoc-dev \
libreadline-dev \
libssl-dev \
&& rm -rf /var/lib/apt/lists/*

# Clone and build parquet_s3_fdw
RUN git clone https://github.com/duckdb/duckdb && \
cd duckdb && \
GEN=ninja make

RUN git clone https://github.com/duckdb/pg_duckdb.git && \
cd pg_duckdb && \
make install
libtool \
libxml2-dev \
libxml2-utils \
libxslt-dev \
pkg-config \
protobuf-compiler \
xsltproc \
zlib1g-dev

# Clone and build Tembo's fork of parquet_s3_fdw (required for pg_tier).
RUN git clone https://github.com/tembo-io/tembo_parquet_s3_fdw.git \
&& (cd tembo_parquet_s3_fdw && git checkout v1.1.4) \
&& make -C tembo_parquet_s3_fdw USE_PGXS=1 install DESTDIR="$PWD/target"

# Copy the files to the final image.
FROM quay.io/tembo/standard-cnpg:${PG_VERSION}-${TAG}
USER root
COPY --from=build /work/target/./ /

# cache all extensions
RUN set -eux; \
cp -r $(pg_config --pkglibdir)/* /tmp/pg_pkglibdir; \
cp -r $(pg_config --sharedir)/* /tmp/pg_sharedir;

ENV LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

# Revert the postgres user to id 26
RUN usermod -u 26 postgres
USER 26

0 comments on commit bd378b1

Please sign in to comment.