@zsnmwy what do you do right now with the Serverless Env Prisma Edge ? - Windmill

rubenf

R

rubenfOP•7/4/23, 8:19 AM

I think the best is probably that we support prisma edge client in a dedicated lightweight http handler and that one generate the client locally and then sync it using the CLI. Then one can refer to it with local relative imports

Meow the Magnificent

M

Meow the Magnificent•7/4/23, 8:34 AM

Hello! I'm trying to programmatically disable buttons based on groups that ctx.username is part of. Wondering how do we retrieve the groups of a user in python?

rubenf

R

rubenfOP•7/4/23, 8:37 AM

@meowthemagnificent short of adding it to the context, the easiest will be to hit windmill api directly with the user token

rubenf

R

rubenfOP•7/4/23, 8:38 AM

https://app.windmill.dev/openapi.html#/operations/whoami

rubenf

R

rubenfOP•7/4/23, 8:39 AM

you can leverage BASE_INTERNAL_URL as baseUrl and fetch directly or use the windmill_api library to hit whoami

rubenf

R

rubenfOP•7/4/23, 8:40 AM

but I think we should add it to ctx, that would be more consistent

Meow the Magnificent

M

Meow the Magnificent•7/4/23, 8:41 AM

fast reply as always Ruben! Thanks alot! I'll check that out

rubenf

Rrubenf @zsnmwy what do you do right now with the Serverless Env Prisma Edge ?

zsnmwy

Z

zsnmwy•7/4/23, 9:05 AM

The usage is like the official docs. https://www.prisma.io/docs/data-platform/data-proxy
But the official data proxy is not open source. And the official region is far away.

Data Proxy infrastructure is currently available in two regions:

US East 1 (West Virginia, United States)
EU Central 1 (Frankfurt, Germany)

Data Proxy infrastructure is currently available in two regions:

US East 1 (West Virginia, United States)
EU Central 1 (Frankfurt, Germany)

So I deploy the unofficial data proxy in k8s. https://github.com/zsnmwy/prisma-data-proxy
I do some work to make it work with Deno.

---

Detail Flow

Prisma Edge Client -- Send GraphQL --> Prisma Data Proxy( Wrapper ) --- Forward The GraphQL Request To Prisma Query Engine ---> Prisma Query Engine --> DB

Prisma Edge Client

Write the Prisma Schema.
Build the Deno Client base on the schema. yarn prisma generate --data-proxyyarn prisma generate --data-proxy
Send GraphQL To Prisma Data Proxy

Prisma Data Proxy

Prisma Data Proxy starts a Prisma Query Engine base on config. Make it listen on the specific port.
/app/query-engine --datamodel-path ./prisma/schema.prisma --host 0.0.0.0 --enable-playground --port 4467 --enable-raw-queries --enable-metrics --dataproxy-metric-override --enable-telemetry-in-response/app/query-engine --datamodel-path ./prisma/schema.prisma --host 0.0.0.0 --enable-playground --port 4467 --enable-raw-queries --enable-metrics --dataproxy-metric-override --enable-telemetry-in-response
Prisma Query Engine will parse the Prisma schema and builds the GraphQL schema.
Prisma Data Proxy will accept requests from the Prisma Edge Client. It will check the requested permission. Then forward the request to Prisma Query Engine.

Prisma Query Engine

Accept the request - GraphQL from Prisma Data Proxy.
Query DB and return data.

Prisma

Data Proxy in the Prisma Data Platform provides database connection management and pooling, load balancing, scaling, and fault tolerance features so that you can manage and scale database connections for applications that use Prisma ORM. With Data Proxy, your application can scale and maintain predictable database performance because it also lim...

GitHub

GitHub - zsnmwy/prisma-data-proxy: Self-host Prisma Data Proxy.

Self-host Prisma Data Proxy. Contribute to zsnmwy/prisma-data-proxy development by creating an account on GitHub.

zsnmwy

Z

zsnmwy•7/4/23, 9:05 AM

Prisma Client Edge does not support some Prisma features. Like metrics. That means we need to remove the unsupported features.
Deno Client needs to rebuild on each DB version. And push them to S3. But is not easy to track the version in Windmill. Windmill will hash the code and keep it. But the schema and the Prisma Deno Client are changed. Then I need to change all Prisma import links.
The Prisma Migration Engine binary is not good for users. So I build another Nodejs image to migrate and track the database.

K8S Deployment

Init Container - init-db-migrate-deploy
Main Container - prisma-data-proxy
Prisma Query Engine export metrics about performance.

rubenf

R

rubenfOP•7/4/23, 9:06 AM

Woah that's super impressive but also seems like a lot of work to make prisma work

zsnmwy

Z

zsnmwy•7/4/23, 9:06 AM

import {
  Prisma,
  PrismaClient,
} from "http://xxxxx/prisma-edge-deno/0.2.2/deno/edge.ts";

type PrismaDataProxy = {
  url: string;
};
export async function main(
  prismaDataProxy: PrismaDataProxy,
) {
  const prisma = new PrismaClient({
    datasources: { db: { url: prismaDataProxy.url } },
  });
}

import {
  Prisma,
  PrismaClient,
} from "http://xxxxx/prisma-edge-deno/0.2.2/deno/edge.ts";

type PrismaDataProxy = {
  url: string;
};
export async function main(
  prismaDataProxy: PrismaDataProxy,
) {
  const prisma = new PrismaClient({
    datasources: { db: { url: prismaDataProxy.url } },
  });
}

rubenf

Rrubenf Woah that's super impressive but also seems like a lot of work to make prisma wo...

zsnmwy

Z

zsnmwy•7/4/23, 9:07 AM

But it's easy now. I build a lot of scripts.

rubenf

R

rubenfOP•7/4/23, 9:07 AM

I think an official integration for windmill would mandate the data-proxy

rubenf

R

rubenfOP•7/4/23, 9:07 AM

so the only thing would be to integrate the edge.ts generator in the CLI

rubenf

R

rubenfOP•7/4/23, 9:08 AM

so that you can resolve it with local imports

rubenf

R

rubenfOP•7/4/23, 9:09 AM

Could you share the script that you use to generate:

import {
  Prisma,
  PrismaClient,
} from "http://xxxxx/prisma-edge-deno/0.2.2/deno/edge.ts";

import {
  Prisma,
  PrismaClient,
} from "http://xxxxx/prisma-edge-deno/0.2.2/deno/edge.ts";

We could add it to the CLI and that would probably almost be sufficient

rubenf

R

rubenfOP•7/4/23, 9:09 AM

it would require to use the CLI ans sync on db changes but that's doable I suppose

zsnmwy

Z

zsnmwy•7/4/23, 9:09 AM

Now I just only to need build the image base on the base image.
More jobs are done on the base image.

FROM xxx/prisma-db/prisma-data-proxy:latest

ARG PRISMA_VERSION="4bc8b6e1b66cb932731fb1bdbbc550d1e010de81"
ENV OS="linux-musl"
ENV QUERY_ENGINE_URL="https://binaries.prisma.sh/all_commits/${PRISMA_VERSION}/${OS}/query-engine.gz"
ENV MIGRATION_ENGINE_URL="https://binaries.prisma.sh/all_commits/${PRISMA_VERSION}/${OS}/migration-engine.gz"

# install prisma
WORKDIR /app/prisma
# download query engine
RUN wget -O query-engine.gz $QUERY_ENGINE_URL && \
    gunzip query-engine.gz && \
    chmod +x query-engine
# download migration engine
RUN wget -O migration-engine.gz $MIGRATION_ENGINE_URL && \
    gunzip migration-engine.gz && \
    chmod +x migration-engine

ENV TZ Asia/Shanghai
RUN apk add alpine-conf tzdata tini --no-cache && \
    /sbin/setup-timezone -z Asia/Shanghai && \
    apk del alpine-conf

WORKDIR /app

COPY ./prisma ./prisma
COPY ./wait-for-it.sh .
RUN chmod a+x ./wait-for-it.sh

ENV MIGRATION_LOCK_FILE="/app/prisma/migration.lock"
ENV QUERY_ENGINE_PATH="/app/query-engine"
ENV MIGRATION_ENGINE_PATH="/app/migration-engine"
ENV PRISMA_SCHEMA_FILE="/app/prisma/schema.prisma"

EXPOSE 4466
CMD ["/app/main"]
ENTRYPOINT [ "tini", "--" ]

FROM xxx/prisma-db/prisma-data-proxy:latest

ARG PRISMA_VERSION="4bc8b6e1b66cb932731fb1bdbbc550d1e010de81"
ENV OS="linux-musl"
ENV QUERY_ENGINE_URL="https://binaries.prisma.sh/all_commits/${PRISMA_VERSION}/${OS}/query-engine.gz"
ENV MIGRATION_ENGINE_URL="https://binaries.prisma.sh/all_commits/${PRISMA_VERSION}/${OS}/migration-engine.gz"

# install prisma
WORKDIR /app/prisma
# download query engine
RUN wget -O query-engine.gz $QUERY_ENGINE_URL && \
    gunzip query-engine.gz && \
    chmod +x query-engine
# download migration engine
RUN wget -O migration-engine.gz $MIGRATION_ENGINE_URL && \
    gunzip migration-engine.gz && \
    chmod +x migration-engine

ENV TZ Asia/Shanghai
RUN apk add alpine-conf tzdata tini --no-cache && \
    /sbin/setup-timezone -z Asia/Shanghai && \
    apk del alpine-conf

WORKDIR /app

COPY ./prisma ./prisma
COPY ./wait-for-it.sh .
RUN chmod a+x ./wait-for-it.sh

ENV MIGRATION_LOCK_FILE="/app/prisma/migration.lock"
ENV QUERY_ENGINE_PATH="/app/query-engine"
ENV MIGRATION_ENGINE_PATH="/app/migration-engine"
ENV PRISMA_SCHEMA_FILE="/app/prisma/schema.prisma"

EXPOSE 4466
CMD ["/app/main"]
ENTRYPOINT [ "tini", "--" ]

rubenf

Rrubenf Could you share the script that you use to generate: ``` import { Prisma, Pr...

zsnmwy

Z

zsnmwy•7/4/23, 9:12 AM

I can write a detailed guide to help you.
It needs two repo to show you how to do it.

rubenf

R

rubenfOP•7/4/23, 9:15 AM

I think I don't follow everything but:

Would love to build a guide to integrate prisma
Improve windmill so that we can improve the prisma experience

zsnmwy

Z

zsnmwy•7/4/23, 9:16 AM

Too many details. I just picked the important part to say.

zsnmwy

Z

zsnmwy•7/4/23, 9:18 AM

But Windmill needs the data proxy. No matter what kind of data proxy.

rubenf

R

rubenfOP•7/4/23, 9:18 AM

Agreed, even for the native sql integration we lose so much performance on establishing the connection

rubenf

R

rubenfOP•7/4/23, 9:21 AM

I will probably strongly recommend using pgBouncer for postgres

rubenf

R

rubenfOP•7/4/23, 9:22 AM

(That's for the user scripts, windmill uses of postgres already uses efficient pooling)

zsnmwy

Z

zsnmwy•7/4/23, 9:27 AM

Prisma has a good user experience.
How about both support?
Now I make Prisma work with a self-host data proxy in Windmill.
May just need little jobs to integrate with Windmill.

rubenf

R

rubenfOP•7/4/23, 9:28 AM

the data proxy is always gonna be a long-runnin process which is something that should be decoupled from windmill

rubenf

R

rubenfOP•7/4/23, 9:29 AM

that and the fact that it's gonna be very workspace/folder specific so you would need one such process per workspace

rubenf

R

rubenfOP•7/4/23, 9:29 AM

that scale pretty terribly

zsnmwy

Z

zsnmwy•7/4/23, 9:32 AM

May need money to enable it on the online version.

zsnmwy

Z

zsnmwy•7/4/23, 9:33 AM

rubenf

R

rubenfOP•7/4/23, 9:38 AM

I think it's more that there is only thing we want to do the devops of

rubenf

R

rubenfOP•7/4/23, 9:38 AM

and that is windmill, not data proxy

Meow the Magnificent

MMeow the Magnificent Hello! I'm trying to programmatically disable buttons based on groups that ctx.u...

rubenf

R

rubenfOP•7/4/23, 9:58 PM

Added groups to ctx on latest

rubenf

R

rubenfOP•7/4/23, 10:13 PM

Not sure about willmind but yes that's pretty much the most common use-case for windmill

rubenf

R

rubenfOP•7/4/23, 10:15 PM

I mena yes but for that use-case I would recommend self-hosting

rubenf

R

rubenfOP•7/4/23, 10:15 PM

you can just set timeout to infinite

rubenf

R

rubenfOP•7/4/23, 10:16 PM

it's one of the env variable of the workers

rubenf

R

rubenfOP•7/4/23, 10:16 PM

I recommend reading the docs and README

Sindre

S

Sindre•7/5/23, 2:48 AM

https://www.windmill.dev/docs/reference#raw-payload

/app/query-engine --datamodel-path ./prisma/schema.prisma --host 0.0.0.0 --enable-playground --port 4467 --enable-raw-queries --enable-metrics --dataproxy-metric-override --enable-telemetry-in-response

/app/query-engine --datamodel-path ./prisma/schema.prisma --host 0.0.0.0 --enable-playground --port 4467 --enable-raw-queries --enable-metrics --dataproxy-metric-override --enable-telemetry-in-response

import {
  Prisma,
  PrismaClient,
} from "http://xxxxx/prisma-edge-deno/0.2.2/deno/edge.ts";

type PrismaDataProxy = {
  url: string;
};
export async function main(
  prismaDataProxy: PrismaDataProxy,
) {
  const prisma = new PrismaClient({
    datasources: { db: { url: prismaDataProxy.url } },
  });
}

import {
  Prisma,
  PrismaClient,
} from "http://xxxxx/prisma-edge-deno/0.2.2/deno/edge.ts";

type PrismaDataProxy = {
  url: string;
};
export async function main(
  prismaDataProxy: PrismaDataProxy,
) {
  const prisma = new PrismaClient({
    datasources: { db: { url: prismaDataProxy.url } },
  });
}

FROM xxx/prisma-db/prisma-data-proxy:latest

ARG PRISMA_VERSION="4bc8b6e1b66cb932731fb1bdbbc550d1e010de81"
ENV OS="linux-musl"
ENV QUERY_ENGINE_URL="https://binaries.prisma.sh/all_commits/${PRISMA_VERSION}/${OS}/query-engine.gz"
ENV MIGRATION_ENGINE_URL="https://binaries.prisma.sh/all_commits/${PRISMA_VERSION}/${OS}/migration-engine.gz"

# install prisma
WORKDIR /app/prisma
# download query engine
RUN wget -O query-engine.gz $QUERY_ENGINE_URL && \
    gunzip query-engine.gz && \
    chmod +x query-engine
# download migration engine
RUN wget -O migration-engine.gz $MIGRATION_ENGINE_URL && \
    gunzip migration-engine.gz && \
    chmod +x migration-engine

ENV TZ Asia/Shanghai
RUN apk add alpine-conf tzdata tini --no-cache && \
    /sbin/setup-timezone -z Asia/Shanghai && \
    apk del alpine-conf

WORKDIR /app

COPY ./prisma ./prisma
COPY ./wait-for-it.sh .
RUN chmod a+x ./wait-for-it.sh

ENV MIGRATION_LOCK_FILE="/app/prisma/migration.lock"
ENV QUERY_ENGINE_PATH="/app/query-engine"
ENV MIGRATION_ENGINE_PATH="/app/migration-engine"
ENV PRISMA_SCHEMA_FILE="/app/prisma/schema.prisma"

EXPOSE 4466
CMD ["/app/main"]
ENTRYPOINT [ "tini", "--" ]

FROM xxx/prisma-db/prisma-data-proxy:latest

ARG PRISMA_VERSION="4bc8b6e1b66cb932731fb1bdbbc550d1e010de81"
ENV OS="linux-musl"
ENV QUERY_ENGINE_URL="https://binaries.prisma.sh/all_commits/${PRISMA_VERSION}/${OS}/query-engine.gz"
ENV MIGRATION_ENGINE_URL="https://binaries.prisma.sh/all_commits/${PRISMA_VERSION}/${OS}/migration-engine.gz"

# install prisma
WORKDIR /app/prisma
# download query engine
RUN wget -O query-engine.gz $QUERY_ENGINE_URL && \
    gunzip query-engine.gz && \
    chmod +x query-engine
# download migration engine
RUN wget -O migration-engine.gz $MIGRATION_ENGINE_URL && \
    gunzip migration-engine.gz && \
    chmod +x migration-engine

ENV TZ Asia/Shanghai
RUN apk add alpine-conf tzdata tini --no-cache && \
    /sbin/setup-timezone -z Asia/Shanghai && \
    apk del alpine-conf

WORKDIR /app

COPY ./prisma ./prisma
COPY ./wait-for-it.sh .
RUN chmod a+x ./wait-for-it.sh

ENV MIGRATION_LOCK_FILE="/app/prisma/migration.lock"
ENV QUERY_ENGINE_PATH="/app/query-engine"
ENV MIGRATION_ENGINE_PATH="/app/migration-engine"
ENV PRISMA_SCHEMA_FILE="/app/prisma/schema.prisma"

EXPOSE 4466
CMD ["/app/main"]
ENTRYPOINT [ "tini", "--" ]