• Location: England, Greater Manchester, Manchester
  • Salary: £50000 - £70000 per annum
  • Technology: Google Cloud Jobs
  • Job Type: Permanent
  • Date Posted: 23rd Apr, 2021
  • Reference: Dataeng12


Job Description

I am working with a fast growing a hugely exciting AI & ML agency who are recruiting for a data engineer who will be working on our data pipeline architecture, with the aim of providing clean, usable data to Business Analysts and Data Scientists. You will be responsible for helping to build modular pipeline components and ensure technical documentation is created and maintained.


Role & Responsibilities
    • Helping design, build, maintain and operate the data pipeline.
    • Defining and building modular data pipeline components.
    • Ensuring that solid development practices, such as proper use of source control, full testing processes and automated deployment mechanisms, are followed
    • .Collaborate with data scientists and business analysts to discover where business value can be found within the data available.
    • Maintaining existing systems and supporting migration to new data pipeline architecture.Training Client Analytics and Implementation teams in how to implement and support clients.
    • Acting as a subject matter expert on all aspects of the data pipeline.Identifying potential performance issues, bottlenecks and pain points and recommend new and creative ways of resolving them.



Skills & Qualifications
    • Distributed data processing, for example Spark
    • NoSQL Databases, such as MongoDB or Couchbase
    • Cloud computing platforms, such as Google Cloud Platform or AWS
    • Pipeline orchestration, for example Airflow
    • Technical understanding of infrastructure components, their dependencies, and interactions between servers, virtual systems, networks, databases, web applications, etc


Experience
  • 3+ years of proven experience using Python to build data pipelines, including familiarity with python's core big data / data science libraries: e.g. pandas, pyspark, scikit-learn etcSolid understanding of database design and SQL
  • Experience working in cross functional agile teams, particularly teams including Data Scientists, Software Engineers and Business Analysts
  • The ability to communicate complicated technical solutions to non technical users
  • Take ownership of feature development and ongoing maintenance
  • Technical understanding of infrastructure components, their dependencies, and interactions between servers, virtual systems, networks, databases, web applications, etc


Similar Jobs

GCP Architect
England, Greater Manchester, Manchester