Main Purpose of the Job:

The Data Engineer is responsible for providing the right data, to the right place, in the right format. They will work with business owners to transform data into valuable resources through the design and development of data processing pipelines that are fully automated, resilient, maintainable and comply with Dense Air’s data governance and information security policies.

The Data Engineer is responsible for designing and implementing data flows, business logic, metadata management and logging to ensure that data is reliably available to stakeholders on time and to the highest quality.

The Data Engineer will work with peers in the Data and Analytics group, collaborate with business stakeholders and other technical teams to provide end-to-end data capabilities, ensuring Dense Air’s tactical and strategic business objectives are met.

The Data Engineer will bring with them a skillset that demonstrates their abilities as a cloud data engineer, GCP-centric but open to working with other cloud service providers where appropriate. They will share their knowledge both intra and inter teams and be able to participate and contribute to the currently evolving micro-service architecture, advising where relevant how to leverage the appropriate resources available to achieve the company’s ambitions.

Key Responsibilities:

• Analysis and profiling of source data to enable its effective integration and use
• Design and develop data processing pipelines that process huge volumes of 3rd party, internal and client data.
• Working with technical peers, design software that optimizes Dense Air’s on-premises and cloud footprints.
• Ensure data engineering best practices are introduced into the development process, resulting in the creation of high value data assets.
• Ensure that data governance and information security is included in all solutions by design and that relevant metadata is collected and stored in appropriate repositories.
• Design and build software that is flexible, scalable, maintainable and of high quality.
• Continually monitor data ingest and usage patterns and provide feedback into software optimization, improvements and new features.
• Participate in the full development life cycle, including agile ceremonies and code reviews.

Education and Experience:

• Three years working in a data engineering capacity, in a medium or large sized organisation.
• Strong knowledge of software programming languages such as SQL and Python
• Experience of micro-service architecture and containerization and appreciate the full life cycle of their development process
• Experience of working with serverless cloud technologies such as Google’s Cloud Run, Cloud Functions, Apigee
• Experience of working within a big data environment, with knowledge of batch orchestration frameworks such as GCP DataFlow and various storage technologies, such as GCS, blob, RDBMS, NoSQL
• Experience of working with GIS and spatial databases or extensions such as PostgreSQL with PostGIS, Oracle Spatial, MS SQL Server Spatial, GeoMesa and GeoTrellis.
• Experience of leveraging Cloud compute such as GCP BigQuery for querying Big Data at scale and how to optimise data models to improve performance, scaling and cost.
• Experience of common data management concepts and patterns such as ETL, Meta Data Management, Data Quality and Lineage.
• Understanding of DevOps, CI-CD pipelines, automated testing frameworks, test and design led development and other associated software engineering best practices.
• Technical degree in Computer Science, Computer Engineering or relevant field preferable.

Personal Qualities

• Outstanding problem solver and team player.
• Passionate about providing best-in-class data to our stakeholders.
• Able to document designs with strong attention to detail.
• An excellent planner able to schedule and prioritise work and react positively to issues occurring.
• Enthusiastic about development, sharing experience and best practice and able to write clear and maintainable software.
• Able to present information in a concise way to stakeholders.

Equality and Diversity

Dense Air is an equal opportunities employer. We have an inclusive culture, and respect for everyone, regardless of gender, ethnicity, religion, age, sexual orientation, disability or other characteristics