The freedom of the cloud comes with the responsibility of FinOps. Businesses need to adopt an operating model in which cloud engineers are responsible for the cost of their solutions. “Pay as you go” pricing models that have shifted CAPEX to OPEX need proper governance to bring accountability to cloud spend.
FinOps principles:
Teams need to collaborate
Everyone takes ownership for their cloud usage
A centralized team drives FinOps
Reports should be accessible and timely
Decisions are driven by business value of cloud
Take advantage of the variable cost model of the cloud.
Our FinOps generic stack
Infracost
Cloud cost estimates for Terraform in pull requests. This powerful tool gives you full visibility on the cost of your terraform static resources.
GCP Pricing Calculator
Easy-to-use tool for quick estimation of costs on Google Cloud.
Billing Export to BigQuery
Source of truth for all your costs on Google Cloud. You can then generate all kind of analysis from this source table and plug it into datamarts.
Cost optimization dashboards
Custom-made dashboards to understand and optimize your Google Cloud costs.
Resource tagging and labelling
Labelling and tagging your resources based on cost centers and departments allows for owners hip of resources and reporting. We use dedicated terraform module to easily label and tag every single resource.
GCP Budget Alerts
Cost can skyrocket in a matter of hours with usage-based resources (i.e. bigquery “on-demand” pricing model) and having budget alerts notify you as soon as a cost outburst happens.
Resource commitments
Committing to a resource for a long period of time (1 to 3 years) can drastically reduce your cost (up to 60%).
Quotas
Drastic solution to limit usage on certain resources. Recommended on non-production projects to safeguard against wrong manipulation from your dev team.
FinOps with data
SQL optimization: Filter early on, don’t use “ORDER BY” statements, avoid explosion of joins, etc. The list of SQL best-practices is long and applying those can significantly reduce your costs. It is also of major importance to use incremental loads for every new data load instead of full refresh. And if you are on BigQuery, you should leverage partitioning and clustering; setting those properly will make your downstream queries much faster.
BigQuery flex slots: By default you will be on the “on demand” pricing model at 5 USD per Tb of data processed. In case you have workloads running at regular intervals, flex slots can be a great option to reduce costs. You can switch those on and off dynamically and make great savings as compared to the “on-demand” pricing model.
Bucket lifecycle: Not frequently accessed data should reside on a cold data storage and this can be achieved with lifecycle management policies. Based on the frequency of access to your data, it should be stored in an appropriate
Those are just some examples and based on your data product (Dataflow, Vertex AI, etc.), specific cost optimization techniques exist.
There are actually plenty of tools and techniques to reduce your costs and it is first a matter of creating a FinOps culture within your company. Then cost optimization becomes part of your development, not as an obstacle but as a supporting activity to leverage at best your resources.
Why Astrafy ?
- As a small company that is continuously experimenting internally on a ton of technologies, we have been forced to adopt a FinOps culture to optimize costs at best. We have put our hands dirty on many cost optimization tools and techniques. We know what it takes to get where we are and are in a good position to educate other companies on FinOps.
- As a small company that is continuously experimenting internally on a ton of technologies, we have been forced to adopt a FinOps culture to optimize costs at best. We have put our hands dirty on many cost optimization tools and techniques. We know what it takes to get where we are and are in a good position to educate other companies on FinOps.
- Our experience allows us to quickly detect areas for cost optimization and to have cost “quick wins” in a matter of days.