WSO2 streaming integrator - read large files (one million of records) and write to another file
I want to read one million records from a CSV file and write to another CSV file with WSO2 streaming integrator. The problem is the slow speed of reading and writing data, almost 10 per second. WSO2 says "Read a 100GB file with 10 million lines in just 20 mins" but I found no sample or guide to do that. can anyone help me?
Solution
Finally I found some example here https://ei.docs.wso2.com/en/7.2.0/streaming-integrator/ref/performance-analysis-results/ and with these example my speed is ok.
batch.size.max and buffer.size are useful parameters here.
- How to read a local file in Javascript (Running from an Electron App)
- How to check whether a file or directory exists?
- filesystem error: cannot increment recursive directory iterator
- Extract parts from sql sentences with pyhton script
- Best way to resolve file path too long exception
- Android; Check if file exists without creating a new one
- How to check if a file contains only zeros in a Linux shell?
- How can I split a large text file into smaller files with an equal number of lines?
- Unix command line : How to get the total size of modified files in the last 30 days
- Is it possible to read a file in parallel by extending this function?
- How can I use getimagesize() with $_FILES['']?
- New line added to file when writing from StringIO
- How to implement a pythonic equivalent of tail -F?
- How do I load a file from resource folder?
- Linux - How to track all files accessed by a process?
- Searching for a file name inside a folder of jar files
- How to Download a File after POSTing data using FastAPI?
- Read a file backwards line by line using fseek
- How to create password protect zip file in nodejs using node-7z-archive package
- The bulk download from AWS s3 is failing
- Check if a file exists, and if so create a new file with a higher number
- SAP FM EPS2_GET_DIRECTORY_LISTING file mask
- How to select the first file in a directory?
- Why do I get "Pickle - EOFError: Ran out of input" reading an empty file?
- how to make a library with libtool if there are sources with the same filename in several directories
- Encrypting (large) files in PHP with openSSL
- Download file from github in CLI?
- How do I submit a "file" input without submit button using JavaScript?
- How do I change the file creation date of a Windows file?
- ReactJS - How to detect when cancel is clicked on file input?