Job Overview:
We are seeking a highly skilled Data Engineer with expertise in Artificial Intelligence (AI) to join our dynamic team. This role involves designing, building, and maintaining robust data pipelines while integrating AI models into data processing workflows. The ideal candidate will have a strong background in data engineering, machine learning, and AI, enabling them to work on complex data challenges and implement intelligent solutions.
Key Responsibilities:
- Data Pipeline Development: Design, build, and maintain scalable data pipelines to collect, process, and store large datasets from various sources using technologies such as Apache Kafka, Apache Spark, and ETL tools.
- AI Model Integration: Collaborate with data scientists and AI engineers to deploy machine learning models into data pipelines, ensuring seamless integration and efficient model performance.
- Data Architecture Design: Develop and implement data architecture and infrastructure that supports both batch and real-time data processing.
- Data Quality and Governance: Ensure data accuracy, completeness, and consistency across the organization by implementing data quality checks and governance policies.
- Performance Optimization: Optimize data storage and processing frameworks for speed, efficiency, and scalability, using techniques like indexing, partitioning, and caching.
- Cloud Infrastructure: Utilize cloud platforms (AWS, Azure, GCP) to deploy and manage data pipelines and AI models, ensuring reliability and scalability.
- Automation and Monitoring: Automate data workflows and set up monitoring systems to ensure smooth operations and quick resolution of any issues.
- Collaboration: Work closely with AI engineers, data scientists, and software developers to support AI-driven projects and enhance data-driven decision-making processes.
- Documentation: Create and maintain comprehensive documentation for data workflows, pipelines, and AI model integration processes.
- Research and Innovation: Stay updated with the latest trends in data engineering and AI, bringing innovative solutions to improve data processing and AI implementation.
