Excited to share the success of my latest project: data engineering with Uber's dataset! Used Python, pandas, SQL for insights. Leveraged Google Cloud, BigQuery, Looker Studio for robust analysis. Discovered prime pickup spots, fare rates insights. Thanks to the amazing team! 📊💻🔍 #DataEngineering #UberData #GoogleCloud Overview This project encompasses a comprehensive data engineering venture centered around Uber's expansive dataset. Leveraging Python, pandas, and SQL, we delved deep into data transformation and wrangling to extract invaluable insights.
Key Highlights Technologies Used: Python, pandas, SQL, Google Cloud, BigQuery, Looker Studio, Lucid Chart Insights Uncovered: Prime pickup locations, detailed fare rates analysis Tools Utilized: Google Cloud and BigQuery for seamless data management and execution of intricate queries Looker Studio for data visualization, uncovering patterns and trends Lucid Chart for developing an ERD (Entity-Relationship Diagram) to understand data structures and relationships Getting Started
To get started with this project, follow these steps:
Step-1: import the .csv file to python then did data wrangling in pandas.
Step-2: In the lucid chart I made the ERD of Uber data with the primary key and foreign keys connected.
Step-3: Then open the account on google cloud and made new VM instance and connected through SSH and connected my mage.ai transformed data to Google BigQuery.
Step-4: installed mage.ai on google VM and did data loading and data transforming and data exporting to Google Big Query in Mage.ai
Step-5: Then wrote some SQL script to get the meaningful data insight and made some joins in the script.
Step-6: Then Created account on Looker studio and then connected Google BigQuery To Looker Studio to get the data directly.
Step-7: After getting all the data in Looker studio I created data cards, maps and chart to get the meaningful insight from the data.
Explore the insights and findings generated.
Dashboard Link:- https://shorturl.at/fgJ35
License This project is licensed under the MIT License.