All in all, data management is a balance between Finances and Operations.
Storing the right type of data on the right type of storage is essential to your business and can help you achieve, for example, faster software solutions or better data-driven decisions.
To get started, try to determine the 20% of data that is most important for your business.
You should aim to distribute your data temperature according to the diagram below:
Targeted repartition of data
Nowadays, most companies have to deal with the rise of large data collections, a phenomenon that proves to be challenging.
This leads us to potential issues that should be addressed appropriately, such as:
- Handling growth
- Filtering out irrelevant data
- Managing costs by storing data in the right place
Generally, companies possess more cold data than hot data, and fail to optimize the different storage types according to data temperature.
The following storage types are available within the 3DS OUTSCALE Cloud:
- BSU (Block Storage Unit) is a block storage solution, that is, a virtual hard drive disk you can attach to your instances. You can also create snapshots from BSU volumes, which can help manage data. For more information, see About Volumes.
- OSU (Object Storage Unit) is an autonomous service, not related to your instances, and where data can be accessed via HTTP. For more information, see About OSU.
Depending on your current needs and usage, your data can go through temperature changes that determine which storage type is appropriate.
You start the business day with your application installed on an instance. You also store your data and operating system on different disk partitions.
You create a 100 GiB BSU volume, attach it to your instance, and store your database on it. Because the database needs a quick access to the disk, the recommended storage type here is a classic disk mounted on your instance. That way, your data are "hot", in live production, close to your application and instantly available.
The integrity of such data is not covered by service level agreements (SLAs). You cannot roll back in case an issue occurs on the disk.
At the end of the business day, you want to store the current state of your data and create a backup of them. You do not want these data available for your application, but you want them available in case you need to recover the disk.
To do so, you can create a snapshot of the current volume. Data stored on the snapshot are colder than the data running on the attached volume: you cannot access them directly as a volume disk for your application.
A snapshot only takes a few seconds to create and has a 99.9999% integrity guarantee. You can set up a rolling backup strategy where snapshots older than seven days are automatically deleted and replaced with the current-day snapshot.
At that point, BSU and snapshots are on the same IaaS platform as your business resources. The data are available in just a few seconds.
Your rolling backup strategy is set up, and you are about to delete the seven-day-old snapshot. However, you are legally obligated to keep these data for one year.
These data are cold: they currently have no value for your business and might never go into live production. You do not need them to be available instantly, but you might need to access them eventually.
Here, the recommended and cheaper solution is to archive the data via OSU. Because it is network-based, storing the data is slower than with BSU, while still ensuring reliability.