What is the best way to archive structured data in low cost storage that can be accessed using API?
I have a few hundred GBs of data in a MySQL database that I would like to archive in a low-cost cloud storage. At the same time, I would still like the data to be easily accessible and retrieve using API. I am looking for a technology-independent solution (not tied to any particular database engine).
The frequency of retrieval is expected to be very low but I would like to keep the "cost" (that is the effort) of retrieval to be low as well. I can live with a higher latency to retrieve the data.
I was thinking of using Spark/Databricks (to provide the structured API interface) with long term storage solutions such as Azure Data Lake Storage Gen2 (to provide the low cost storage).
Is this the optimal solution or are there better alternatives?
Yes, Azure Data Lake Storage Gen2 (ADLS2) is the best cloud service to store the data in archive with low cost.
Check the detailed pricing here.
To archive the data in ADLS2, while creating the storage account, in Advanced tab select Cool access tier. Also check mark Enable hierarchical namespace option.
Advanced -> Access tier -> Cool
You can easily authenticate and access Azure Data Lake Storage Gen2 (ADLS Gen2) storage accounts using an Azure storage account access key. Apache Spark provide multiple APIs (RDD, DataFrame, DataSet) which can help you to access and transform the data as per your requirement.
Refer to learn more about APIs: Databricks with ADLS Gen2, RDDs vs DataFrames and Datasets
Note: To retrieve the data you must change the ADLS access tier to Hot.
- can not access to database in moodle
- How to create a filter in power BI to filter rows from a table if the particular column contains certain text?
- Make database column value dependant on another column
- How to Autocomplete several Tables with DBEdit
- Why is my LiveData not getting retrieved from the Room Database and update automatically?
- MySQL - Kill all sleeping connections
- What is the difference between these 2 queries?
- MongoDB run loop to add incremental index value in array
- SQLite as a production database for a low-traffic site
- When i run npm run dev command show error
- Most Efficient Way to... Unique Random String
- Why is my PHP code not correctly updating my CRUD application?
- How to fix the issue: "Values of types "BINARY(255)" and "CHARACTER VARYING(255)" are not comparable" while autopopulating data into H2 database?
- How to map OO inheritence to a relational database scheme?
- Why don't DBMS's support ASSERTION
- Thunderl Cient- Error: User validation failed: accountStatus: `SUSPEND` is not a valid enum value for path `accountStatus`., roles.0:
- Which column type for storing the year field in a table with rows of yearly data
- Access a MongoDB Collection Using String in PyMongo
- What is Database Normalisation/Normalization?
- Check for changes to a SQL Server database?
- Import some database entries through PHPMyAdmin with overwrite
- Is the primary key automatically indexed in MySQL?
- Add new field based on the value of the existing field - MongoDB Aggregate
- Modify and delete tables and apply to DBContext
- Composite Primary Keys : Good or Bad?
- Query that returns the size of a table in a SQLite database
- Declare local variables in PostgreSQL?
- How to migrate data from MongoDB to SQL-Server?
- Custom sorting a dataframe based on multiple columns
- How add unique key to existing table (with non uniques rows)