Hello, I'm André! 👋
About Me
I am a Data Engineer passionate about harnessing the power of data to drive decision-making and innovation. When I'm not delving into data, you might find me exploring the great outdoors or experimenting with my Raspberry Pi collection.
Skills
- 📊 Data Engineering: Proficient in SQL, Python, and data warehousing concepts.
- 🔄 ETL Processes: Experienced in building efficient ETL pipelines.
- ☁️ Cloud Technologies: Familiar with cloud services like AWS, GCP, or Azure.
- 🗄️ Database Management: Comfortable with SQL.
- 🛠️ Tools: Proficient in Git and Docker.
My Projects
AWS Data Pipeline Project: Emulating Pinterest's Data Handling
A sophisticated project aimed at emulating Pinterest's data processing mechanisms for effective management of vast volumes of user-generated content. This project utilized AWS services, Apache Kafka, Databricks, and more, focusing on real-time analytics and batch data processing.
- Repository: Link to GitHub repository for this project
- Technologies: Python, SQL, Spark, AWS (Kinesis, S3, MSK, MWAA, RDS), Apache Kafka, Databricks, Apache Airflow
- Key Features:
- Simulated real-world user interactions with custom Python scripts.
- Utilized Databricks for batch and real-time data processing.
- Automated workflows with AWS Managed Workflows for Apache Airflow (MWAA).
- Configured API Gateway for data ingestion.
Multinational Retail Data Centralisation
This project aims to centralize and streamline data management in the retail sector. It features Python scripts for data extraction, cleaning, and database utilities, focusing on handling diverse data formats and sources. Key technologies used include Python, Pandas, Tabula, SQLAlchemy, PostgreSQL, and Docker.
- Repository: Link to GitHub repository for this project
- Technologies: Python, Pandas, Tabula, Requests, Boto3, SQLAlchemy, YAML, PostgreSQL, Docker
- Key Features:
- Efficient data extraction from databases, PDFs, APIs, and Amazon S3.
- Comprehensive data cleaning and transformation.
- Integration with PostgreSQL database for centralized data management.
Computer Vision Rock Paper Scissors
A fun and interactive project that combines computer vision and game logic. Using Python, OpenCV, and a pre-trained Keras model, this application recognizes hand gestures in real-time to play Rock Paper Scissors against the computer.
- Repository: Link to GitHub repository for this project
- Technologies: Python, NumPy, OpenCV, Keras, Keyboard
- Key Features:
- Real-time gesture recognition from webcam footage.
- Classifies hand gestures into Rock, Paper, Scissors, or false reading.
- Game logic to manage wins, losses, and overall game state.
Connect with Me
I'm always open to connecting with fellow data enthusiasts and tech professionals. Check out my repositories and let's get in touch!