1. Data Pipeline Development & Automation

• Design, build, test, and maintain scalable ETL/ELT data pipelines to extract, transform, and load data from multiple sources (SAP ERP, CRM, databases, APIs, flat files, cloud services)

• Automate data workflows using orchestration tools (such as Apache Airflow, Azure Data Factory, AWS Glue)

• Implement incremental data loading and real-time streaming solutions where applicable

• Prepare and manage data for ML/AI model training, including feature engineering, data labeling pipelines, and model-ready dataset construction

2. Data Architecture & Infrastructure

• Build and optimize data warehouses, data lakes, and data marts to support analytics and reporting needs

• Design and implement lakehouse architectures (such as Databricks, Delta Lake, Apache Iceberg) to support both analytics and AI/ML workloads

• Ensure data infrastructure is scalable, secure, and cost-effective across cloud platforms (such as AWS, Azure, GCP)

• Manage database performance tuning, indexing, partitioning, and query optimization

3. Data Quality & Governance

• Implement data validation, cleansing, and quality checks to ensure accuracy and consistency

• Establish data lineage, metadata management, and data cataloging practices

• Enforce data governance policies, security protocols, and compliance standards (GDPR, data privacy)

• Implement data observability practices using tools (such as Monte Carlo, Great Expectations) for anomaly detection and pipeline health monitoring

4. AI/ML Data Support & MLOps

• Support MLOps practices including pipeline automation, model monitoring, and data versioning for deployed AI/ML models

• Collaborate with the Data Science Team to prepare datasets for model training, including data cleaning, data transformation, and feature engineering

• Build and maintain pipelines for LLM/GenAI workflows, including vector databases, embedding pipelines, and prompt data management

5. Collaboration & Stakeholder Engagement

• Work closely with data analysts, data scientists, business intelligence teams, and business stakeholders to understand data requirements

• Translate business needs into technical data solutions and provide guidance on best practices

• Document data processes, architectures, and workflows for knowledge sharing and onboarding

6. Cloud & Tool Management

• Leverage cloud-based data services (such as AWS S3, Redshift, RDS, Azure Synapse, Databricks, Snowflake) for data storage and processing

• Utilize version control (Git), CI/CD pipelines, and infrastructure-as-code (Terraform, ARM templates) for deployment

• Evaluate and recommend new tools and technologies to improve data engineering workflows

7. Performance Monitoring & Troubleshooting

• Monitor data pipeline performance, identify bottlenecks, and optimize for speed and reliability

• Troubleshoot and resolve data-related issues, ensuring minimal downtime

• Implement data observability and pipeline health monitoring to proactively detect issues