Senior Data Engineer

Job details
Posting date: 12 April 2021
Salary: £34,000 to £37,000 per year
Hours: Full time
Closing date: 12 May 2021
Location: Ilford
Company: Tekbright Systems Ltd
Job type: Permanent
Job reference: TBSLTD /IT-01/2021/SDE-01

Apply for this job


Job Posted Date: 12/04/2021
Job Expiration Date: 12/05/2021

TEKBRIGHT SYSTEMS LTD currently looking for a full-time Senior Data Engineer to join our team.
This is a great opportunity for individuals seeking to work on, and with, a dynamic software development team in a scrum/agile environment.


• Agile environment will be using agile tools like Jira boards with user stories and tasks
• Design and develop big data-related solutions
• Design and processing structured and unstructured data in the ETL process
• Building distributed in-memory application using Spark and Spark QL
• Responsible for the cycle of the Enterprise Data Warehouse building including Requirement Analysis, ETL, and Testing.
• Involve in preparing design, unit, and integration test documents.
• Handling the XSD and XML files generation through the Informatica ETL process.
• Creating the Informatica mappings, sessions, and workflows as per the technical design document
• Extract, Load, and Transform Data via using tools(Informatica, flume, Sqoop ) and script languages(eg: python)
• Maintain effective communications with the project office on all aspects of personal scheduling and project scheduling.
• Building distributed, scalable, and reliable data pipelines to ingest and process data at a large scale in real-time
• design and develop batch jobs using spark
• Analysed data using Hadoop components using HIV and Pig
• Ability to write shell scripts to extract data from Unix servers into Hadoop HDFS
• Ability to adapt AVRO format for entire data ingestion for faster operation and less space utilization.
• build, deploy and monitor on Cloud Environment.
• Setup dev principles and Delivery standards.
• Identify gaps in the data process and drive improvements.

• Bachelor’s degree in a related field.
• Overall It experience must be 7+ years
• must have a minimum of 5 years of experience in Big data and Python/Java/scala
• Strong interpersonal skills including mentoring, coaching, collaborating, and team building
• Knowledge of Project and Software Development Life Cycle Methodologies
• multiple projects and meet deadlines within a fast-paced environment
• Promote cooperation and commitment within the team by assisting and working collaboratively with others
• OBIEE/Power BI
• SQL/ PLSQL, Stored Procedures
• Experience in Informatica ETL, OLAP
• Experience in Hadoop, hdfs, Yarn, MapReduce, Hive, Pig, Sqoop, Oozie, Flume
• Experience in Spark, Spark SQL, Spark Streaming, Spark ML
• Experience in MQ Messaging / Rabit MQ, Confluent Kafka, Schema Registry, Kafka connector, Ksql, Kstreams ..etc
• Experience in JSON, Avro, Parquet file formats
• Microservices Architecture
• Cassandra, Hbase, Elastic Search, or any NO SQL database experience
• RDBMS: Oracle/ SQL Server/Sybase
• Monitoring tools: ELK, Grafana, App Dynamics, Prometheus
• Matrix Query Language, V6 database administration, Oracle
• Data Model using Business Components, Matrix Query Language
• Experience in CI/CD pipeline, Groovy Jenkins, Linux shell scripts.
• any of Cloud technologies: Azure, GCP, AWS.
• Python/R/Scala Knowledge and Experience is a plus
• Dev Ops experience: Jenkins CI/CD, Ansible, helm, terraform
• Ability to self-direct
• Excellent communication skills.

Apply for this job