Tag: gcp
-
Tracking Consumption History in BigQuery Tables: A Guide to Retrieving the Latest Access Date
In today’s post, I’ll share insights into tracing the consumption history of BigQuery tables. This activity isn’t just for the curious; it’s a necessary step in optimizing the data management within BigQuery and ensuring its relevance. By understanding table utilization, organizations can streamline storage and computation costs, deprecate unused data, and offer more up-to-date datasets.… Read more
-
Dataform Explored: Harnessing Its Power and Addressing Opportunities for Improvements
In an earlier post, I spotlighted Dataform’s transformative capabilities, emphasizing its potential to reshape the data transformation and pipelining landscape for teams. Like all sophisticated tools, a deeper examination reveals areas where refinement could enhance the user experience. In this piece, I’ll share the challenges I’ve encountered and suggest improvements to augment Dataform’s effectiveness. Areas… Read more
-
Google Dataplex: An Introduction and Its Advantages
While exploring Google’s tools recently, I bumped into something called Google Dataplex. As most of us know, data plays a huge role in today’s businesses. Every day, big and small companies are making and using tons of data. But with all this data flowing in and out, handling it becomes a big challenge. That’s where… Read more
-
The High Level Benefits of Using Dataform with BigQuery
Introduction In my journey as someone who transitioned from being a data analyst to a data engineer, I had the opportunity to explore and pilot Dataform and DBT for my organization. These tools offered exciting features that bridged the gap between data analysts and software engineers, with a minimal learning curve. In this post, I… Read more
-
Why I Chose Google Cloud Professional Data Engineer Certification
The Spark: My Initial Interest in Google Cloud Professional Data Engineer Certification In recent weeks, I’ve been pondering over the decision to earn the Google Cloud Professional Data Engineer certification. Curiously, I found limited narratives on why many choose this path. So, I embarked on a journey of introspection to document my motivations. In case… Read more
-
How to create an alert for BigQuery Slowness or Bottleneck
Why do we want to create an alert for Slow BigQuery As a Data Engineer and someone who supports the users, it is not great when you start your morning with Slack channel messages about BigQuery or other databases being slow or hanging. I will list some steps to create an alert to detect BigQuery… Read more