s3 - how to get fast line count of file? wc -l is too slow
Does anyone have a quick way of getting the line count of a file hosted in S3? Preferably using the CLI, s3api but I am open to python/boto as well. Note: solution must run non-interactively, ie in an overnight batch.
Right no i am doing this, it works but takes around 10 minutes for a 20GB file:
aws cp s3://foo/bar - | wc -l
UPDATE 2024: Amazon S3 Select is no longer available to new users.
Here's two methods that might work for you...
Amazon S3 has a new feature called S3 Select that allows you to query files stored on S3.
You can perform a count of the number of records (lines) in a file and it can even work on GZIP files. Results may vary depending upon your file format.
Amazon Athena is also a similar option that might be suitable. It can query files stored in Amazon S3.
- ModuleNotFoundError: No module named 'imp'
- cdk api gateway route53 lambda custom domain name not working
- couldn't get current server API group list: the server has asked for the client to provide credentials error: You must be logged in to the server
- Bash with AWS CLI - unable to locate credentials
- Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect)
- InvalidClientTokenId: The security token included in the request is invalid. status code: 403
- s3 - how to get fast line count of file? wc -l is too slow
- AWS PostgreSQL `Cannot find version X for aurora`
- Cannot connect to Amazon Keyspaces with cqlsh
- Can't connect to phpmyadmin in Bitnami instance hosted by AWS
- AWS Amplify - 404 error when deploying Nextjs App
- How can I connect to my aurora serverless v2 database instance?
- Why do I get a CORS error on AWS Api Gateway & Lambda POST?
- How can I toggle between kubernetes context of various deployed apps?
- Redis Instance Reuse in AWS Lambda
- Error when using createmodel function for Sagemaker
- AWS Amplify Auth with Cognito User Pool not returning nonce or at_hash claim in JWT id_token
- Best practice for id_token vs. access_token use in AWS Lambda
- SQSlistener not receiving messages
- Drasi (Change data processing) not working as expected for "StoredProc" reaction
- Terraform - s3 static website wont deploy, telling me I dont have permissions when I do
- Can I pause an ECS service instead of deleting it?
- Unable to install mysqlclient package on EC2 instance
- AWS cannot signed CloudFront urls PEM error
- How can I import Psycopg2 into a Lambda Function on AWS?
- How to limit the access to EC2 from NLB only
- Redshift COPY query hangs. (Running but not submitted) (Table size just 30 rows)
- HttpClient.get retrieving index.html instead of an asset file
- AWS CDK -- Cannot find module '@aws-cdk/aws-ec2'
- Get GitHub git branch for AWS CodeBuild