Is it possible to add the template by the user? I use Prisma to query the database. In Windmill - S

S

Sindre•7/4/23, 7:57 AM

I could misunderstand you, but are you asking for how to select a script that you as the user has created. if so you need to click on the "Select a script or flow" button.

Z

zsnmwyOP•7/4/23, 8:00 AM

I want to add the templates here.

Zzsnmwy Is it possible to add the template by the user? I use Prisma to query the databa...

Z

zsnmwyOP•7/4/23, 8:07 AM

I can share the workflow if anyone needs it.
Directing to connect the database in the serverless env is not a good solution.

R

rubenf•7/4/23, 8:08 AM

I'm actually reworking this right now and the Postgresql and HTTP will no more be really templates but full dedicated languages

R

rubenf•7/4/23, 8:08 AM

this is because the backend for fetch only scripts will be different and a lot more efficient and lightweight

R

rubenf•7/4/23, 8:08 AM

and for SQL same and we will not wrap it with typescript, it will be raw SQL

R

rubenf•7/4/23, 8:09 AM

Also we will add support for GraphQL by allowing to edit the query directly (will use the http engine under the hood)

R

rubenf•7/4/23, 8:09 AM

For Prisma, we might be able to do something nice, I do not know enough about it

R

rubenf•7/4/23, 8:10 AM

@zsnmwy what do you do right now with the Serverless Env Prisma Edge ?

R

rubenf•7/4/23, 8:19 AM

I think the best is probably that we support prisma edge client in a dedicated lightweight http handler and that one generate the client locally and then sync it using the CLI. Then one can refer to it with local relative imports

M

Meow the Magnificent•7/4/23, 8:34 AM

Hello! I'm trying to programmatically disable buttons based on groups that ctx.username is part of. Wondering how do we retrieve the groups of a user in python?

R

rubenf•7/4/23, 8:37 AM

@meowthemagnificent short of adding it to the context, the easiest will be to hit windmill api directly with the user token

R

rubenf•7/4/23, 8:38 AM

https://app.windmill.dev/openapi.html#/operations/whoami

R

rubenf•7/4/23, 8:39 AM

you can leverage BASE_INTERNAL_URL as baseUrl and fetch directly or use the windmill_api library to hit whoami

R

rubenf•7/4/23, 8:40 AM

but I think we should add it to ctx, that would be more consistent

M

Meow the Magnificent•7/4/23, 8:41 AM

fast reply as always Ruben! Thanks alot! I'll check that out

Rrubenf @zsnmwy what do you do right now with the Serverless Env Prisma Edge ?

Z

zsnmwyOP•7/4/23, 9:05 AM

The usage is like the official docs. https://www.prisma.io/docs/data-platform/data-proxy
But the official data proxy is not open source. And the official region is far away.

Data Proxy infrastructure is currently available in two regions:

US East 1 (West Virginia, United States)
EU Central 1 (Frankfurt, Germany)

Data Proxy infrastructure is currently available in two regions:

US East 1 (West Virginia, United States)
EU Central 1 (Frankfurt, Germany)

Data Proxy infrastructure is currently available in two regions:

US East 1 (West Virginia, United States)
EU Central 1 (Frankfurt, Germany)

Data Proxy infrastructure is currently available in two regions:

US East 1 (West Virginia, United States)
EU Central 1 (Frankfurt, Germany)

So I deploy the unofficial data proxy in k8s. https://github.com/zsnmwy/prisma-data-proxy
I do some work to make it work with Deno.

---

Detail Flow

Prisma Edge Client -- Send GraphQL --> Prisma Data Proxy( Wrapper ) --- Forward The GraphQL Request To Prisma Query Engine ---> Prisma Query Engine --> DB

Prisma Edge Client

Write the Prisma Schema.
Build the Deno Client base on the schema.
```
yarn prisma generate --data-proxy
```
yarn prisma generate --data-proxy
```
yarn prisma generate --data-proxy
```
yarn prisma generate --data-proxy
Send GraphQL To Prisma Data Proxy

Prisma Data Proxy

Prisma Data Proxy starts a Prisma Query Engine base on config. Make it listen on the specific port.
```
/app/query-engine --datamodel-path ./prisma/schema.prisma --host 0.0.0.0 --enable-playground --port 4467 --enable-raw-queries --enable-metrics --dataproxy-metric-override --enable-telemetry-in-response
```
/app/query-engine --datamodel-path ./prisma/schema.prisma --host 0.0.0.0 --enable-playground --port 4467 --enable-raw-queries --enable-metrics --dataproxy-metric-override --enable-telemetry-in-response
```
/app/query-engine --datamodel-path ./prisma/schema.prisma --host 0.0.0.0 --enable-playground --port 4467 --enable-raw-queries --enable-metrics --dataproxy-metric-override --enable-telemetry-in-response
```
/app/query-engine --datamodel-path ./prisma/schema.prisma --host 0.0.0.0 --enable-playground --port 4467 --enable-raw-queries --enable-metrics --dataproxy-metric-override --enable-telemetry-in-response
Prisma Query Engine will parse the Prisma schema and builds the GraphQL schema.
Prisma Data Proxy will accept requests from the Prisma Edge Client. It will check the requested permission. Then forward the request to Prisma Query Engine.

Prisma Query Engine

Accept the request - GraphQL from Prisma Data Proxy.
Query DB and return data.

Prisma

Data Proxy

Data Proxy in the Prisma Data Platform provides database connection management and pooling, load balancing, scaling, and fault tolerance features so that you can manage and scale database connections for applications that use Prisma ORM. With Data Proxy, your application can scale and maintain predictable database performance because it also lim...

GitHub

GitHub - zsnmwy/prisma-data-proxy: Self-host Prisma Data Proxy.

Self-host Prisma Data Proxy. Contribute to zsnmwy/prisma-data-proxy development by creating an account on GitHub.

Z

zsnmwyOP•7/4/23, 9:05 AM

Prisma Client Edge does not support some Prisma features. Like metrics. That means we need to remove the unsupported features.
Deno Client needs to rebuild on each DB version. And push them to S3. But is not easy to track the version in Windmill. Windmill will hash the code and keep it. But the schema and the Prisma Deno Client are changed. Then I need to change all Prisma import links.
The Prisma Migration Engine binary is not good for users. So I build another Nodejs image to migrate and track the database.

K8S Deployment

Init Container - init-db-migrate-deploy
Main Container - prisma-data-proxy
Prisma Query Engine export metrics about performance.

R

rubenf•7/4/23, 9:06 AM

Woah that's super impressive but also seems like a lot of work to make prisma work

Z

zsnmwyOP•7/4/23, 9:06 AM

import {
  Prisma,
  PrismaClient,
} from "http://xxxxx/prisma-edge-deno/0.2.2/deno/edge.ts";

type PrismaDataProxy = {
  url: string;
};
export async function main(
  prismaDataProxy: PrismaDataProxy,
) {
  const prisma = new PrismaClient({
    datasources: { db: { url: prismaDataProxy.url } },
  });
}

import {
  Prisma,
  PrismaClient,
} from "http://xxxxx/prisma-edge-deno/0.2.2/deno/edge.ts";

type PrismaDataProxy = {
  url: string;
};
export async function main(
  prismaDataProxy: PrismaDataProxy,
) {
  const prisma = new PrismaClient({
    datasources: { db: { url: prismaDataProxy.url } },
  });
}

import {
  Prisma,
  PrismaClient,
} from "http://xxxxx/prisma-edge-deno/0.2.2/deno/edge.ts";

type PrismaDataProxy = {
  url: string;
};
export async function main(
  prismaDataProxy: PrismaDataProxy,
) {
  const prisma = new PrismaClient({
    datasources: { db: { url: prismaDataProxy.url } },
  });
}

import {
  Prisma,
  PrismaClient,
} from "http://xxxxx/prisma-edge-deno/0.2.2/deno/edge.ts";

type PrismaDataProxy = {
  url: string;
};
export async function main(
  prismaDataProxy: PrismaDataProxy,
) {
  const prisma = new PrismaClient({
    datasources: { db: { url: prismaDataProxy.url } },
  });
}

Rrubenf Woah that's super impressive but also seems like a lot of work to make prisma wo...

Z

zsnmwyOP•7/4/23, 9:07 AM

But it's easy now. I build a lot of scripts.

R

rubenf•7/4/23, 9:07 AM

I think an official integration for windmill would mandate the data-proxy

R

rubenf•7/4/23, 9:07 AM

so the only thing would be to integrate the edge.ts generator in the CLI

R

rubenf•7/4/23, 9:08 AM

so that you can resolve it with local imports

R

rubenf•7/4/23, 9:09 AM

Could you share the script that you use to generate:

import {
  Prisma,
  PrismaClient,
} from "http://xxxxx/prisma-edge-deno/0.2.2/deno/edge.ts";

import {
  Prisma,
  PrismaClient,
} from "http://xxxxx/prisma-edge-deno/0.2.2/deno/edge.ts";

import {
  Prisma,
  PrismaClient,
} from "http://xxxxx/prisma-edge-deno/0.2.2/deno/edge.ts";

import {
  Prisma,
  PrismaClient,
} from "http://xxxxx/prisma-edge-deno/0.2.2/deno/edge.ts";

We could add it to the CLI and that would probably almost be sufficient

R

rubenf•7/4/23, 9:09 AM

it would require to use the CLI ans sync on db changes but that's doable I suppose

Z

zsnmwyOP•7/4/23, 9:09 AM

Now I just only to need build the image base on the base image.
More jobs are done on the base image.

FROM xxx/prisma-db/prisma-data-proxy:latest

ARG PRISMA_VERSION="4bc8b6e1b66cb932731fb1bdbbc550d1e010de81"
ENV OS="linux-musl"
ENV QUERY_ENGINE_URL="https://binaries.prisma.sh/all_commits/${PRISMA_VERSION}/${OS}/query-engine.gz"
ENV MIGRATION_ENGINE_URL="https://binaries.prisma.sh/all_commits/${PRISMA_VERSION}/${OS}/migration-engine.gz"

# install prisma
WORKDIR /app/prisma
# download query engine
RUN wget -O query-engine.gz $QUERY_ENGINE_URL && \
    gunzip query-engine.gz && \
    chmod +x query-engine
# download migration engine
RUN wget -O migration-engine.gz $MIGRATION_ENGINE_URL && \
    gunzip migration-engine.gz && \
    chmod +x migration-engine

ENV TZ Asia/Shanghai
RUN apk add alpine-conf tzdata tini --no-cache && \
    /sbin/setup-timezone -z Asia/Shanghai && \
    apk del alpine-conf

WORKDIR /app

COPY ./prisma ./prisma
COPY ./wait-for-it.sh .
RUN chmod a+x ./wait-for-it.sh

ENV MIGRATION_LOCK_FILE="/app/prisma/migration.lock"
ENV QUERY_ENGINE_PATH="/app/query-engine"
ENV MIGRATION_ENGINE_PATH="/app/migration-engine"
ENV PRISMA_SCHEMA_FILE="/app/prisma/schema.prisma"

EXPOSE 4466
CMD ["/app/main"]
ENTRYPOINT [ "tini", "--" ]

FROM xxx/prisma-db/prisma-data-proxy:latest

ARG PRISMA_VERSION="4bc8b6e1b66cb932731fb1bdbbc550d1e010de81"
ENV OS="linux-musl"
ENV QUERY_ENGINE_URL="https://binaries.prisma.sh/all_commits/${PRISMA_VERSION}/${OS}/query-engine.gz"
ENV MIGRATION_ENGINE_URL="https://binaries.prisma.sh/all_commits/${PRISMA_VERSION}/${OS}/migration-engine.gz"

# install prisma
WORKDIR /app/prisma
# download query engine
RUN wget -O query-engine.gz $QUERY_ENGINE_URL && \
    gunzip query-engine.gz && \
    chmod +x query-engine
# download migration engine
RUN wget -O migration-engine.gz $MIGRATION_ENGINE_URL && \
    gunzip migration-engine.gz && \
    chmod +x migration-engine

ENV TZ Asia/Shanghai
RUN apk add alpine-conf tzdata tini --no-cache && \
    /sbin/setup-timezone -z Asia/Shanghai && \
    apk del alpine-conf

WORKDIR /app

COPY ./prisma ./prisma
COPY ./wait-for-it.sh .
RUN chmod a+x ./wait-for-it.sh

ENV MIGRATION_LOCK_FILE="/app/prisma/migration.lock"
ENV QUERY_ENGINE_PATH="/app/query-engine"
ENV MIGRATION_ENGINE_PATH="/app/migration-engine"
ENV PRISMA_SCHEMA_FILE="/app/prisma/schema.prisma"

EXPOSE 4466
CMD ["/app/main"]
ENTRYPOINT [ "tini", "--" ]

FROM xxx/prisma-db/prisma-data-proxy:latest

ARG PRISMA_VERSION="4bc8b6e1b66cb932731fb1bdbbc550d1e010de81"
ENV OS="linux-musl"
ENV QUERY_ENGINE_URL="https://binaries.prisma.sh/all_commits/${PRISMA_VERSION}/${OS}/query-engine.gz"
ENV MIGRATION_ENGINE_URL="https://binaries.prisma.sh/all_commits/${PRISMA_VERSION}/${OS}/migration-engine.gz"

# install prisma
WORKDIR /app/prisma
# download query engine
RUN wget -O query-engine.gz $QUERY_ENGINE_URL && \
    gunzip query-engine.gz && \
    chmod +x query-engine
# download migration engine
RUN wget -O migration-engine.gz $MIGRATION_ENGINE_URL && \
    gunzip migration-engine.gz && \
    chmod +x migration-engine

ENV TZ Asia/Shanghai
RUN apk add alpine-conf tzdata tini --no-cache && \
    /sbin/setup-timezone -z Asia/Shanghai && \
    apk del alpine-conf

WORKDIR /app

COPY ./prisma ./prisma
COPY ./wait-for-it.sh .
RUN chmod a+x ./wait-for-it.sh

ENV MIGRATION_LOCK_FILE="/app/prisma/migration.lock"
ENV QUERY_ENGINE_PATH="/app/query-engine"
ENV MIGRATION_ENGINE_PATH="/app/migration-engine"
ENV PRISMA_SCHEMA_FILE="/app/prisma/schema.prisma"

EXPOSE 4466
CMD ["/app/main"]
ENTRYPOINT [ "tini", "--" ]

FROM xxx/prisma-db/prisma-data-proxy:latest

ARG PRISMA_VERSION="4bc8b6e1b66cb932731fb1bdbbc550d1e010de81"
ENV OS="linux-musl"
ENV QUERY_ENGINE_URL="https://binaries.prisma.sh/all_commits/${PRISMA_VERSION}/${OS}/query-engine.gz"
ENV MIGRATION_ENGINE_URL="https://binaries.prisma.sh/all_commits/${PRISMA_VERSION}/${OS}/migration-engine.gz"

# install prisma
WORKDIR /app/prisma
# download query engine
RUN wget -O query-engine.gz $QUERY_ENGINE_URL && \
    gunzip query-engine.gz && \
    chmod +x query-engine
# download migration engine
RUN wget -O migration-engine.gz $MIGRATION_ENGINE_URL && \
    gunzip migration-engine.gz && \
    chmod +x migration-engine

ENV TZ Asia/Shanghai
RUN apk add alpine-conf tzdata tini --no-cache && \
    /sbin/setup-timezone -z Asia/Shanghai && \
    apk del alpine-conf

WORKDIR /app

COPY ./prisma ./prisma
COPY ./wait-for-it.sh .
RUN chmod a+x ./wait-for-it.sh

ENV MIGRATION_LOCK_FILE="/app/prisma/migration.lock"
ENV QUERY_ENGINE_PATH="/app/query-engine"
ENV MIGRATION_ENGINE_PATH="/app/migration-engine"
ENV PRISMA_SCHEMA_FILE="/app/prisma/schema.prisma"

EXPOSE 4466
CMD ["/app/main"]
ENTRYPOINT [ "tini", "--" ]

Rrubenf Could you share the script that you use to generate: ``` import { Prisma, Pr...

Z

zsnmwyOP•7/4/23, 9:12 AM

I can write a detailed guide to help you.
It needs two repo to show you how to do it.

R

rubenf•7/4/23, 9:15 AM

I think I don't follow everything but:

Would love to build a guide to integrate prisma
Improve windmill so that we can improve the prisma experience

Z

zsnmwyOP•7/4/23, 9:16 AM

Too many details. I just picked the important part to say.

Z

zsnmwyOP•7/4/23, 9:18 AM

But Windmill needs the data proxy. No matter what kind of data proxy.

R

rubenf•7/4/23, 9:18 AM

Agreed, even for the native sql integration we lose so much performance on establishing the connection

R

rubenf•7/4/23, 9:21 AM

I will probably strongly recommend using pgBouncer for postgres

R

rubenf•7/4/23, 9:22 AM

(That's for the user scripts, windmill uses of postgres already uses efficient pooling)

Z

zsnmwyOP•7/4/23, 9:27 AM

Prisma has a good user experience.
How about both support?
Now I make Prisma work with a self-host data proxy in Windmill.
May just need little jobs to integrate with Windmill.

R

rubenf•7/4/23, 9:28 AM

the data proxy is always gonna be a long-runnin process which is something that should be decoupled from windmill

R

rubenf•7/4/23, 9:29 AM

that and the fact that it's gonna be very workspace/folder specific so you would need one such process per workspace

R

rubenf•7/4/23, 9:29 AM

that scale pretty terribly

Z

zsnmwyOP•7/4/23, 9:32 AM

May need money to enable it on the online version.

Z

zsnmwyOP•7/4/23, 9:33 AM

R

rubenf•7/4/23, 9:38 AM

I think it's more that there is only thing we want to do the devops of

R

rubenf•7/4/23, 9:38 AM

and that is windmill, not data proxy

MMeow the Magnificent Hello! I'm trying to programmatically disable buttons based on groups that ctx.u...

R

rubenf•7/4/23, 9:58 PM

Added groups to ctx on latest

R

rubenf•7/4/23, 10:13 PM

Not sure about willmind but yes that's pretty much the most common use-case for windmill

R

rubenf•7/4/23, 10:15 PM

I mena yes but for that use-case I would recommend self-hosting

R

rubenf•7/4/23, 10:15 PM

you can just set timeout to infinite

Is it possible to add the template by the user? I use Prisma to query the database. In Windmill - S

Similar Threads

Similar Threads

Similar Threads