IceCactus•12mo ago

Why does this code sometimes work but 95% of the time it doesnt.

import * as wmill from "windmill-client"
import { Client } from "pg"
import { from as copyFrom } from "pg-copy-streams"

// Defining the type for Postgres resource
type Postgresql = {
  host: string,
  port: number,
  user: string,
  dbname: string,
  sslmode: string,
  password: string,
  root_certificate_pem: string
}

export async function  main(csvFile: string, postgresqlResource: Postgresql) {

  //Start up db connection
  //Creating a new postgres client
  const client = new Client({
    user: postgresqlResource.user,
    database: postgresqlResource.dbname,
    password: postgresqlResource.password,
    host: postgresqlResource.host, // Changed from hostname to host
    port: postgresqlResource.port
  });

let stream;
const time = new Date().toISOString();


  try {
    await client.connect();

    //test to make sure connection is working.
    //await client.query("INSERT INTO inventory (sku,warehouse_id,time,qty) VALUES ('test',2,now(),25)");


    stream = await client.query(copyFrom('COPY inventory (sku,warehouse_id,time,qty) FROM STDIN'));

    let numOfRows = 0;   

    stream.write('partnumber' + '\t' + '2' + '\t' + time + '\t' + '43' + '\n');
    //await stream.end();
    
  }
  catch (e)
  {
    console.log(e);
  }
  finally
  {
    stream.end();  
  }

Disclaimer: Im learning javascript. Im trying to bulk import into postgres using BUN. For some reason, randomly this work work like i expect but 95% of the time i run the script it won't. If i uncomment the insert, that will work 100% of the time.

17 Replies

rubenf•12mo ago

What's the error?

IceCactus•12mo ago

There is no error I wish there was at least it would give me a clue. I think there is something wrong with the async code, thats why it randomly works sometimes, but I cant seem to figure out what it is.

rubenf•12mo ago

you're likely missing an await somewhere can you format your code correctly, I cannot read it use backticks

IceCactus•12mo ago

Ive been trying to solve this for like 7 hours now, i have tried awaiting everything, then nothing etc. Whats wierd, is I will add an await, then try it and it will load into the postgres table. I will literally wait 5 seconds and rerun the same script, and it wont work, no errors

rubenf•12mo ago

import * as wmill from "windmill-client"
import { Client } from "pg"
import { from as copyFrom } from "pg-copy-streams"

// Defining the type for Postgres resource
type Postgresql = {
  host: string,
  port: number,
  user: string,
  dbname: string,
  sslmode: string,
  password: string,
  root_certificate_pem: string
}

export async function  main(csvFile: string, postgresqlResource: Postgresql) {

  //Start up db connection
  //Creating a new postgres client
  const client = new Client({
    user: postgresqlResource.user,
    database: postgresqlResource.dbname,
    password: postgresqlResource.password,
    host: postgresqlResource.host, // Changed from hostname to host
    port: postgresqlResource.port
  });

let stream;
const time = new Date().toISOString();


  try {
    await client.connect();

    //test to make sure connection is working.
    //await client.query("INSERT INTO inventory (sku,warehouse_id,time,qty) VALUES ('test',2,now(),25)");


    stream = await client.query(copyFrom('COPY inventory (sku,warehouse_id,time,qty) FROM STDIN'));

    let numOfRows = 0;

    stream.write('partnumber' + '\t' + '2' + '\t' + time + '\t' + '43' + '\n');
    //await stream.end();

  }
  catch (e)
  {
    console.log(e);
  }
  finally
  {
    stream.end();
  }

import * as wmill from "windmill-client"
import { Client } from "pg"
import { from as copyFrom } from "pg-copy-streams"

// Defining the type for Postgres resource
type Postgresql = {
  host: string,
  port: number,
  user: string,
  dbname: string,
  sslmode: string,
  password: string,
  root_certificate_pem: string
}

export async function  main(csvFile: string, postgresqlResource: Postgresql) {

  //Start up db connection
  //Creating a new postgres client
  const client = new Client({
    user: postgresqlResource.user,
    database: postgresqlResource.dbname,
    password: postgresqlResource.password,
    host: postgresqlResource.host, // Changed from hostname to host
    port: postgresqlResource.port
  });

let stream;
const time = new Date().toISOString();


  try {
    await client.connect();

    //test to make sure connection is working.
    //await client.query("INSERT INTO inventory (sku,warehouse_id,time,qty) VALUES ('test',2,now(),25)");


    stream = await client.query(copyFrom('COPY inventory (sku,warehouse_id,time,qty) FROM STDIN'));

    let numOfRows = 0;

    stream.write('partnumber' + '\t' + '2' + '\t' + time + '\t' + '43' + '\n');
    //await stream.end();

  }
  catch (e)
  {
    console.log(e);
  }
  finally
  {
    stream.end();
  }

rubenf•12mo ago

Use the pipeline function I would say: https://www.npmjs.com/package/pg-copy-streams

npm

pg-copy-streams

Low-Level COPY TO and COPY FROM streams for PostgreSQL in JavaScript using. Latest version: 6.0.6, last published: 4 months ago. Start using pg-copy-streams in your project by running npm i pg-copy-streams. There are 114 other projects in the npm registry using pg-copy-streams.

rubenf•12mo ago

and await on it

IceCactus•12mo ago

I tried that too, same thing. That didnt work at all. stream.write() actually worked once in a while

Sindre•12mo ago

stream.write('partnumber' + '\t' + '2' + '\t' + time + '\t' + '43' + '\n'); This part of your code might be causing issues. If stream.write returns false, it means the internal buffer is full, and you should wait for the 'drain' event before writing more data. However, if you're only writing once and then ending the stream, you might not encounter this issue. Ending the Stream: You are ending the stream with stream.end(). This should be done only after all writes are complete. It’s better to attach a callback to the 'finish' event of the stream to know when all data has been flushed and the stream has ended. stream.end(); Error Handling with Streams: You should add error handling for the stream. Attach an 'error' event listener to the stream to catch and handle any errors that occur during the stream operation.

stream.on('error', (err) => {
  console.error('Stream encountered an error:', err);
});

stream.on('error', (err) => {
  console.error('Stream encountered an error:', err);
});

Resource Management: Ensure that the client is disconnected and the stream is closed properly in the finally block. This ensures that resources are always cleaned up, even if an error occurs.

    finally {
      stream.end();
      await client.end();
    }

    finally {
      stream.end();
      await client.end();
    }

In summary, the main concern in your code is ensuring that the stream operations are handled correctly. You need to attach appropriate event listeners for 'drain', 'finish', and 'error' events on the stream to manage the flow of data and catch any errors. Additionally, make sure that all resources are properly cleaned up in a finally block. __ from chatGPT 4. Do not know if it's correct or not, but usally it's quite good, at least if you go back and forth a few times.

IceCactus•12mo ago

stream.on('error', (err) => { console.error('Stream encountered an error:', err); }); I had that in the code too...never executed

rubenf•12mo ago

what I can tell you is that you have to make sure you have an await somewhere that await for your execution to be done otherwise your code will return immediately and windmill will exit early

IceCactus•12mo ago

I think that is whats happening i just cant find it

rubenf•12mo ago

I think so too, so you need to wait until your stream is done and that's what await pipeline is for

rubenf•12mo ago

https://stackoverflow.com/questions/33599688/how-to-use-es8-async-await-with-streams

Stack Overflow

How to use ES8 async/await with streams?

In https://stackoverflow.com/a/18658613/779159 is an example of how to calculate the md5 of a file using the built-in crypto library and streams. var fs = require('fs'); var crypto = require('cryp...

IceCactus•12mo ago

I wish i could use c# in windmill

rubenf•12mo ago

you can install c# and call it from bash 😄

IceCactus•12mo ago

lol, thats exactly what im about to do...i was really trying to learn JS though and I'm stubborn haha