Costs and Insights Info
This document refers to the creation of the structure and assembly of this dashboard and other connected dashboards. This is our repository that contains all the code needed to create structures.
Structure, Orchestration and Flow
Basically our structure was created in lambda functions using ECR images.
In all the functions of the repository we have a file called docker.txt, which contains the step by step on how to create a local ECR image for testing and how to deploy it to production too. In addition to the libraries to be installed that the function depends on and which is found in requirements.txt and will be installed through a Dockerfile file.
Currently our structure is orchestrated with the AWS tool called Step Functions, and here is our pipeline there.
Basically, we created the structure of the necessary cost tables for snowflake and then, in parallel, we created the structure of the aws cost tables along with the slack notifications and event structure about the customers.
The flow above is triggered by the EventBridge here.
About the codes, basically all the functions just run a set of queries, mainly for the snowflake calculations, where we separate them by large queries and in separate files for debugs if necessary, in the others, you just need to understand the query and update it if necessary.
Grafana Dashboards
All dashboards in grafana are created based on these tables which are created and updated in these lambdas function. All are found in INTERNAL.INTERNAL and must have ownership as SYSADMIN, with the exception of the aws tables.
For example for this dashboard, if I enter the dashboard written CURRENT_MONTH and click on EDIT:
I will see a query below running on snowflake via plugin.
And with that you can manipulate your results and structures.
Slack Integration
Basically to create an integration with slack and the application in lambda function, first an application was created in slack. The generated token was saved in this secret to be used in our application via API by the python slack_sdk library. Our apps: https://api.slack.com/apps/A03TTGKC5L1
And the combination of the python library for snowflake with this and other libraries that you can find here in the requirements.txt, we managed to get the results and handle the alerts. Here is an example identical to our case of how to use it.
https://www.pragnakalp.com/create-slack-bot-using-python-tutorial-with-examples/