All in all, data management is a balance between Financial and Operational.
Storing the right kind of data on the right storage type is essential to your business and can help you achieve, for example, faster software solutions or better data-driven decisions.
To get started, try to determine the most important 20% of your data for your business.
Here is a view of your data temperature distribution:
Targeted repartition of data:
Nowadays, most companies are challenged with large data collection growth, which can sometimes go out of hand.
This leads us to potential issues that should be addressed appropriately, such as:
- Handling growth
- Filtering out irrelevant data
- Managing costs by storing data in the right place
Most companies have more cold data than hot data, and fail to optimize storage type with data temperature.
The following storages types are available within Outscale Cloud:
- BSU (Block Storage Unit) is a block storage, that is, a virtual hard drive disk you can attach to your instances (VMs). You can also create snapshots from BSU volumes, which can help manage data. For more information, see About Volumes.
- Ephemeral storage is very fast storage located physically on your instance (VM). You cannot create snapshots from it, and data is erased each time you stop the instance. For more information, see About Ephemeral Storage.
- OSU (Object Storage Unit) is an autonomous service, not related to your instances, and where data can be access via HTTP. For more information, see About OSU.
Depending on your current needs and usage, your data can go through temperature changes that determine which storage type is right for you.
You start the business day with your application installed an instance (VM). You also store your data and operating system on different disk partitions.
You create a 100 GiB BSU volume, attach it to your instance, and store your database on it. Because the database needs a quick access to the disk, the recommended storage type here is a classic disk mounted on your instance. That way, your data is "hot", in live production, close to your application and instantly available.
Integrity of such data is not covered by SLAs. You cannot roll back in case of issue on the disk.
Ephemeral disk can be a good alternative of BSU for non persistency data such as your temporary cache directory for MySQL/MariaDB.
End of the Day Data
At the end of the business day, you want to store the current state of your data and create a backup of it. You do not want this data available for your application, but you want it available in case you need to recover the disk.
To do so, you can create a snapshot of the current volume. Data stored on the snapshot is colder than the data running on the attached volume: you cannot access it directly as a volume disk for your application.
A snapshot only takes a few seconds to create and has a 99.9999% integrity guarantee. You can set up a rolling backup strategy where snapshots older than seven days are automatically deleted and replaced with the current-day snapshot.
At that point, BSU, ephemeral storage and snapshots are on the same IaaS platform as your business resources. The data is ready to go for your business in just a few seconds.
Your rolling backup strategy is set up, and you are about to delete the seven-day old snapshot. However, you are legally obligated to keep this data for one year.
This data is cold: it currently has no value for your business and might never go into live production. You do not need it to be available instantly, but you might need to access it eventually.
Here, the recommended and cheaper solution is to archive the data on OSU. Because it is network-based, storing the data is slower than on BSU while still ensuring reliability.