Senior Big Data developer
12 April 2021
£34,000 to £37,000 per year
12 May 2021
Tekbright Systems Ltd
Apply for this job
Job Posted: 12/04/2021
Job Expiration Date: 12/05/2021
TEKBRIGHT SYSTEMS LTD currently looking for a full-time Senior Big Data Developer to join our team.
This is a great opportunity for individuals seeking to work on, and with, a dynamic software development team in a scrum/agile environment.
• Agile environment will be using agile tools like Jira boards with user stories and tasks
• Design and develop big data-related solutions
• Maintain and tune the performance of Spark Applications
• Design and processing structured and unstructured data in the ETL process
• Running Hadoop streaming jobs to process terabytes of data
• Building distributed in-memory application using Spark and Spark QL
• Manage and schedule spark jobs on a Hadoop cluster using Apache Falcon
• Involve in preparing design, unit, and integration test documents.
• Extract, Load, and Transform Data via using tools( Informatica, flume, Sqoop ) and script languages(eg: python)
• Maintain effective communications with the project office on all aspects of personal scheduling and project scheduling.
• Building distributed, scalable, and reliable data pipelines to ingest and process data at a large scale in real-time
• design and develop batch jobs using spark
• Analysed data using Hadoop components using HIV and Pig
• Ability to write shell scripts to extract data from Unix servers into Hadoop HDFS
• Ability to adapt AVRO format for entire data ingestion for faster operation and less space utilization.
• Build, deploy and monitor on Cloud Environment.
• Setup dev principles and Delivery standards.
• Identify gaps in the data process and drive improvements.
• Bachelor’s degree in a related field.
• Overall It experience must be 7+ years
• must have a minimum of 5 years of experience in Big data and Python/Java/scala
• Strong interpersonal skills including mentoring, coaching, collaborating, and team building
• Knowledge of Project and Software Development Life Cycle Methodologies
• multiple projects and meet deadlines within a fast-paced environment
• Promote cooperation and commitment within the team by assisting and working collaboratively with others
• Experience in Hadoop, hdfs, Yarn, MapReduce, Hive, Pig, Sqoop, Oozie, Flume
• Experience in Spark, Spark SQL, Spark Streaming, Spark ML
• Experience in MQ Messaging / Rabit MQ, Confluent Kafka, Schema Registry, Kafka connector, Ksql, Kstreams ..etc
• Experience in JSON, Avro, Parquet file formats
• Microservices Architecture
• Cassandra, Hbase, Elastic Search or any NO SQL database experience and any RDBMS databases
• Monitoring tools: ELK, Grafana, App Dynamics, Prometheus
• Matrix Query Language, V6 database administration, Oracle
• Data Model using Business Components, Matrix Query Language
• Experience in CI/CD pipeline, Groovy Jenkins, Linux shell scripts.
• any of Cloud technologies: Azure, GCP, AWS.
• Python/R/Scala Knowledge and Experience is a plus
• Dev Ops experience: Jenkins CI/CD, Ansible, helm, terraform
• Ability to self-direct
• Excellent communication skills.
Apply for this job