Deskripsi pekerjaan
- Collaborate with Data Engineer to design and implement scalable and reliable systems for ingestion, processing, and analysis of large disparate data sets from diverse sources.
- Improve existing and create new data infrastructure components to better automate extraction, transformation, loading, and other data management processes.
- Develop tools and applications to proactively measure, monitor, and improve data quality and consistency during loading and analysis processes.
- Analyze and improve efficiency, reliability, and scalability of data infrastructure and processes.
- Work with data team to define and promote best practices for data management and analysis, and to build and improve systems to implement and support these practices.
- Maximizing the reliability and performance of services used in data infrastructure.
- 4+ years of relevant work experience in data infrastructure and software engineering.
- Strong scripting and development proficiency in two or more languages (Python preferred, experience in Airflow DAG is a plus).
- Professional experience with multiple relational and non-relational database systems.
- Desire and readiness to take on greater ownership of and responsibility for data infrastructure.
- Professional experience retrieving, parsing, cleaning, and transforming data from multiple formats (e.g. XML, JSON, CSV, PDF, etc.) and delivery mechanisms (e.g. files, streams, APIs).
- Experience with cloud infrastructure services, especially Google Cloud Platform.
- Experience with development pipeline and continuous development with Git and its automation.
- Experience with Kubernetes and has managed the deployment and services inside Kubernetes.
- Familiarity with basic cloud networking.
