WSO2 streaming integrator - read large files (one million of records) and write to another file
I want to read one million records from a CSV file and write to another CSV file with WSO2 streaming integrator. The problem is the slow speed of reading and writing data, almost 10 per second. WSO2 says "Read a 100GB file with 10 million lines in just 20 mins" but I found no sample or guide to do that. can anyone help me?
Solution
Finally I found some example here https://ei.docs.wso2.com/en/7.2.0/streaming-integrator/ref/performance-analysis-results/ and with these example my speed is ok.
batch.size.max and buffer.size are useful parameters here.
- wxmaxima wxdraw , wxdraw2d, wxdraw3d. How to change output to a *.png file instead of terminal computer screen
- How to reference batch files stored in project from code
- How can I create an empty file at the command line in Windows?
- What are .nsh files? Can't find any docs for it
- C program — trouble compiling files together
- There are some way to print error log to an external file using g++ C++ compiler? (C++)
- FileNotFoundException: /storage/emulated/0/Android
- Flutter Solve: 'Offset is outside the bounds of the DataView' error only on Firebase Hosting
- How to get Files and Form data using the Request object in FastAPI?
- How to create a file -- including folders -- for a given path?
- Which reasons could make ShellExecute fail?
- Normal.dotm equivalent in Excel for referencing the same single VBA code
- Opening a file to read and write, create it if it doesn't exist
- How to avoid platform based NoSuchFileException when using java.nio.file.Paths?
- __FILE__ macro shows full path
- I can't move files through C++ (both MoveFile, MoveFileEx, filesystem::rename)
- Download a csv file from harvard dataverse
- Jasmine unit test for file size with TypeScript in Angular
- How can I inspect page resources in Firefox?
- Output not as expected in fwrite
- Check if file is open for writing by another task
- Dynamic Gnuplot graphs
- Read lines from UART using fgets and select
- Why can't a second process read the file which allows reading via FileShare.Read?
- Non-blocking (NIO) reading of lines
- Service to write files with specific group and permissions C#
- How to read a file from internet?
- how to copy a file from current folder to another folder in C#
- How to append multiple files in R
- How do I get the path and name of the python file that is currently executing?