invakid404•4mo ago

PHP scripts are slow to run

I have a relatively simple PHP script with a single Composer dependency as a part of a bigger flow that runs often, and its runtime dominates the entire flow, taking >0.6s to complete. It appears that Windmill installs the dependency every time it runs, which would be my guess why it takes so long. Is there a way to make Windmill cache it?

8 Replies

rubenf•4mo ago

Normally it is cached but will let @Hugo respond on that (he is OOTO, will be back tomorrow). Also if you give us reproduction steps with your imports, it's easier for us to investigate

invakid404•4mo ago

This is a slightly trimmed down version of my script:

<?php

// require:
// willdurand/email-reply-parser
use EmailReplyParser\Parser\EmailParser;

function main(string $content, bool $plain) {
  $parser = new EmailParser();

  $parsedEmail = $parser->parse($content);
  $fragment = current($parsedEmail->getFragments());

  $result = trim($fragment->getContent());
  $result = preg_replace('/\n{3,}/', "\n\n", $result);

  return $result;
}

<?php

// require:
// willdurand/email-reply-parser
use EmailReplyParser\Parser\EmailParser;

function main(string $content, bool $plain) {
  $parser = new EmailParser();

  $parsedEmail = $parser->parse($content);
  $fragment = current($parsedEmail->getFragments());

  $result = trim($fragment->getContent());
  $result = preg_replace('/\n{3,}/', "\n\n", $result);

  return $result;
}

I've deployed it as a standalone script and it still takes >0.5s to run each time, and I can see Composer logs in the output on every run

rubenf•4mo ago

also can you confirm it's slow after having been cached by all workers for that, verify that one of the run has been on the same worker more than once

invakid404•4mo ago

I ran it ~20 times and I only have 3 workers, all the runs have consistently taken >0.5s

invakid404•4mo ago

here are the logs for one of them: https://img.qilin-qilin.ts.net/2024-07-22_08-44-31_JYiaz.webp

invakid404•4mo ago

the logs appear to be identical on all runs for context, I am testing this on a self-hosted windmill deployment with the app itself and all the runners running on the same machine/container I haven't done any complex configuration I don't seem to be having caching issues with bun/deno scripts

Hugo C.•4mo ago

Composer install is run on each run using the composer lockfile which is generated on deployment. We use standard composer caching (cache folder is /tmp/windmill/cache/composer) and you will get speed gain if the package is already in there. You can test that by running rm -rf /tmp/windmill/cache/composer/files/willdurand inside the worker containers (which should make the script >2x slower). The script is still "slow" because of the composer install but the packages are cached. Bun caching is better optimized and allows us to skip bun install. We might consider in the future to do something similar for php/composer (cc @rubenf wdyt?)

rubenf•4mo ago

for sure, depending on how composer work, we can apply the same approach, have a tar containing the symlinks to the cache, and cache that tar based on lockfile hash