Job Summary:
Location : Bangalore
Experience: 1to 6years
As a Data Engineer at INITO, you will play a pivotal role in designing and implementing robust data pipelines that serve as the backbone for our data-driven decision-making process. Working closely with Business Analysts, Data Scientists, and Machine Learning Engineers, you will ensure the integration, transformation, and delivery of data across our platforms, creating a single source of truth that supports complex analytics and machine learning models. We are seeking a passionate and skilled individual who is adept at leveraging cutting-edge technology to solve complex data challenges and enhance our business capabilities.
Responsibilities:
- Design, develop, and maintain robust data pipelines using Airflow, DBT, and Python within a BigQuery environment.
- Implement and maintain data integrations with various sources and targets using tools like Airbyte, Elementary, and Metaflow, with a focus on optimizing data storage and retrieval in BigQuery.
- Write advanced SQL queries and scripts for data manipulation and retrieval in BigQuery.
- Develop and optimize data models and schemas in Preset and other visualization tools, tailored for performance in BigQuery.
-Utilize Pyspark for big data processing tasks, ensuring compatibility with BigQuery's infrastructure.
-Work with APIs, Dockers, and GCP services to enhance and automate data workflows, particularly focusing on integration with BigQuery.
-Apply best practices in continuous integration and continuous deployment (CI/CD) environments using testing and pipeline validation techniques.
-Employ asynchronous architectures, queues, caching strategies, and distributed computing to enhance system performance.
- Write custom code in Python and utilize Jinja within DBT for dynamic pipeline generation, specifically designed for efficiency in BigQuery.
- Adopt design patterns and engineering best practices to ensure code scalability and maintainability.
Key outcomes:
Develop a Unified Data Foundation - Spearhead the creation of a single source of truth that centralizes our data assets, ensuring consistency and reliability across the board.
Expertise in Cutting-Edge Technology - Utilize the state-of-the-art DBT stack and automate reports, optimizing our data processing workflows for efficiency and accuracy.
Drive Impact Across Teams - Your work will directly influence the success of our Business Analysts, Data Scientists, and Machine Learning Engineers by providing them with the crucial data they need to excel.
Optimize Business Logic - Design and manage data tables that support and enhance business logic functions, delivering tailored data solutions that drive strategic decision-making.