Volga — Open-source Feature Engine For Real-time AI — Part 1
Author(s): Andrey Novitskiy Originally published on Towards AI. This is the first part of a 2-post series describing the background and motivation behind Volga. For technical details, see the second part. Volga river TL;DR Volga is an open-source, self-serve, scalable data/feature calculation …
Unlocking the Gates to Success: Dive into SQL Interview Questions from Leading MAANG Companies
Author(s): Kamireddy Mahendra Originally published on Towards AI. “Consistent practice is the key to unlocking success in clearing any coding interview.” Concepts used: Window functions, CTE, Joins, Subqueries, and GROUP BY Photo by Christian Wiediger on Unsplash Q1. Assume you’re given a …
Simplify Your Data Engineering Journey: The Essential PySpark Cheat Sheet for Success!
Author(s): Kamireddy Mahendra Originally published on Towards AI. “ It is not important to complete tasks blindly. It is important to complete tasks more efficiently with more effectiveness” Photo by Markus Winkler on Unsplash Yes, It is important to understand before getting …
Revolutionising Machine Learning: Achieving Top 4% in Kaggle with AutoGluon in Just 7 Lines of Code
Author(s): Daniel Voyce Originally published on Towards AI. Autogluon Forecasting Since starting a new Data Engineering role at Slalom _build, I realized I needed to refresh my ML experience as it was a couple of years out of date. A couple of …
Deletion Vectors in Delta Tables: Speeding Up Operations in Databricks
Author(s): Muttineni Sai Rohith Originally published on Towards AI. Traditionally, Delta Lake supports only the Copy-On-Write paradigm, in which underlying data files are changed anytime a file has been written. Example: When a single row in a file is deleted, the entire …
Understanding Data Lineage: From Source to Destination
Author(s): Muttineni Sai Rohith Originally published on Towards AI. I went to a restaurant yesterday, “Anthera.” After eating my fourth or fifth piece of pepper chicken, which, by the way, was delicious, I started to be amazed by our capability to digest …
Data Cleaning in Python
Author(s): Louis Adibe Originally published on Towards AI. Master data cleaning in Python using the Panda libraryScott Graham on Unsplash Today, I will show you how to implement data cleaning using pandas. The dataset used in this publication comes from open-rice Hongkong …
Understanding SCD — Slowly Changing Dimensions
Author(s): Saniya Parveez Originally published on Towards AI. Introduction In the dynamic realm of data management, the concept of Slowly Changing Dimensions (SCD) emerges as a crucial paradigm. SCD constitutes a fundamental principle in the field of data warehousing and database administration, …
Orchestrate Machine Learning Pipelines with AWS Step Functions
Author(s): ????Mike Shakhomirov Originally published on Towards AI. Advanced-Data Engineering and ML Ops with Infrastructure as CodePhoto by Markus Winkler on Unsplash This story explains how to create and orchestrate machine learning pipelines with AWS Step Functions and deploy them using Infrastructure …
Chat with Your BigQuery Data
Author(s): Benedict Neo Originally published on Towards AI. made with excalidraw Large language models (LLMs) have shown extraordinary ability in understanding natural language and generating code. One popular use-case for code generation is in Text-To-SQL tasks, where the goal is to automatically …
What does “Garbage in, garbage out” mean in solving real business problems?
Author(s): Zijing Zhu Originally published on Towards AI. and how to avoid it with a practical workflow Photo by Gary Chan on Unsplash This member-only story is on us. Upgrade to access all of Medium. In today's business landscape, relying on accurate …
How to Query Pandas Dataframe With the query() Method?
Author(s): Chetan Ambi Originally published on Towards AI. Syntax This member-only story is on us. Upgrade to access all of Medium. Image by author If you have been using pandas for data analysis and are still not aware of the Pandas query …
It’s a Great Time To Be a Data Engineer
Author(s): Keith McNulty Originally published on Towards AI the World’s Leading AI and Technology News and Media Company. If you are building an AI-related product or service, we invite you to consider becoming an AI sponsor. At Towards AI, we help scale …
Diving Into Data Pipelines — Foundations of Data Engineering
Author(s): Towards AI Team An Introduction to Data Pipelines. How, What, When, and Why? Continue reading on Towards AI » Published via Towards AI …
Lakehouse and the evolution of Data Lake
Author(s): Bruno Cordeiro Simplifying data infrastructure and accelerating innovation Continue reading on Towards AI — Multidisciplinary Science Journal » Published via Towards AI …
Recent Posts
