Why does this code sometimes work but 95% of the time it doesnt.
import * as wmill from "windmill-client"
import { Client } from "pg"
import { from as copyFrom } from "pg-copy-streams"
// Defining the type for Postgres resource
type Postgresql = {
host: string,
port: number,
user: string,
dbname: string,
sslmode: string,
password: string,
root_certificate_pem: string
}
export async function main(csvFile: string, postgresqlResource: Postgresql) {
//Start up db connection
//Creating a new postgres client
const client = new Client({
user: postgresqlResource.user,
database: postgresqlResource.dbname,
password: postgresqlResource.password,
host: postgresqlResource.host, // Changed from hostname to host
port: postgresqlResource.port
});
let stream;
const time = new Date().toISOString();
try {
await client.connect();
//test to make sure connection is working.
//await client.query("INSERT INTO inventory (sku,warehouse_id,time,qty) VALUES ('test',2,now(),25)");
stream = await client.query(copyFrom('COPY inventory (sku,warehouse_id,time,qty) FROM STDIN'));
let numOfRows = 0;
stream.write('partnumber' + '\t' + '2' + '\t' + time + '\t' + '43' + '\n');
//await stream.end();
}
catch (e)
{
console.log(e);
}
finally
{
stream.end();
}
Disclaimer: Im learning javascript. Im trying to bulk import into postgres using BUN. For some reason, randomly this work work like i expect but 95% of the time i run the script it won't. If i uncomment the insert, that will work 100% of the time.17 Replies
What's the error?
There is no error
I wish there was at least it would give me a clue.
I think there is something wrong with the async code, thats why it randomly works sometimes, but I cant seem to figure out what it is.
you're likely missing an await somewhere
can you format your code correctly, I cannot read it
use backticks
Ive been trying to solve this for like 7 hours now, i have tried awaiting everything, then nothing etc. Whats wierd, is I will add an await, then try it and it will load into the postgres table. I will literally wait 5 seconds and rerun the same script, and it wont work, no errors
Use the pipeline function I would say: https://www.npmjs.com/package/pg-copy-streams
npm
pg-copy-streams
Low-Level COPY TO and COPY FROM streams for PostgreSQL in JavaScript using. Latest version: 6.0.6, last published: 4 months ago. Start using pg-copy-streams in your project by running
npm i pg-copy-streams
. There are 114 other projects in the npm registry using pg-copy-streams.and await on it
I tried that too, same thing. That didnt work at all.
stream.write() actually worked once in a while
stream.write('partnumber' + '\t' + '2' + '\t' + time + '\t' + '43' + '\n');
This part of your code might be causing issues. If stream.write returns false, it means the internal buffer is full, and you should wait for the 'drain' event before writing more data. However, if you're only writing once and then ending the stream, you might not encounter this issue.
Ending the Stream: You are ending the stream with stream.end(). This should be done only after all writes are complete. It’s better to attach a callback to the 'finish' event of the stream to know when all data has been flushed and the stream has ended.
stream.end();
Error Handling with Streams: You should add error handling for the stream. Attach an 'error' event listener to the stream to catch and handle any errors that occur during the stream operation.
Resource Management: Ensure that the client is disconnected and the stream is closed properly in the finally block. This ensures that resources are always cleaned up, even if an error occurs.
In summary, the main concern in your code is ensuring that the stream operations are handled correctly. You need to attach appropriate event listeners for 'drain', 'finish', and 'error' events on the stream to manage the flow of data and catch any errors. Additionally, make sure that all resources are properly cleaned up in a finally block.
__
from chatGPT 4. Do not know if it's correct or not, but usally it's quite good, at least if you go back and forth a few times.stream.on('error', (err) => {
console.error('Stream encountered an error:', err);
}); I had that in the code too...never executed
what I can tell you is that you have to make sure you have an await somewhere that await for your execution to be done
otherwise your code will return immediately and windmill will exit early
I think that is whats happening i just cant find it
I think so too, so you need to wait until your stream is done
and that's what await pipeline is for
Stack Overflow
How to use ES8 async/await with streams?
In https://stackoverflow.com/a/18658613/779159 is an example of how to calculate the md5 of a file using the built-in crypto library and streams.
var fs = require('fs');
var crypto = require('cryp...
I wish i could use c# in windmill
you can install c# and call it from bash 😄
lol, thats exactly what im about to do...i was really trying to learn JS though and I'm stubborn haha