Abser Ali S
Chennai
Summary
Databricks certified data engineer professional with 14 years of software development experience. Specialized in big data processing and solutions for 6 years, focusing on optimizing ETL performance and reliability using advanced databricks platforms. Expertise in Azure Data Factory, Azure Databricks, Delta Lake, PySpark, Streaming, and Spark SQL. Delivering efficient and effective solutions in the field of data engineering.
Overview
15
15
years of professional experience
1
1
Certification
Work History
Data Solution Architect
DXC Technology
10.2022 - Current
- Migrated legacy mainframe systems to modern cloud-based platforms for increased efficiency and scalability to stands with market and competitors
- Improved ETL performance by optimizing delta lake tables with liquid clustering, z ordering, auto compact, AQE, recommended coding techniques and data modeling.
- Evaluated emerging technologies in the field of data engineering, recommending and implementing them into the pipeline
- Data engineering architecture design and supports for our passenger & cargo data processing portfolio
- Reduced operational costs through optimizing cluster configuration and storage
- Built a Data Governance model on top of unity catalog for AA data system.
- Effectively used databricks APIs for running notebooks, jobs, cluster creation, etc.
- Built a cicd pipeline using azure devops & github and automated the deployment process
- Developed DLT pipeline for delta table change data feed.
- Optimized power bi query workload by 80% with materialized view on delta lake house
- Built an efficient streaming pipeline with statefull aggregator operator & watermarking for live flight status dashboard.
Senior Data Engineer
FIS Global Business Solutions
04.2021 - 09.2022
- Developed data pipelines by implementing advanced ETL processes for data from various sources and format
- Optimized ETL pipelines by following databricks recommended performance tuning techniques and coding standards.
- Improved data quality by addressing duplicate records, null values, threshold check in data pipeline.
SQL Developer
NEC Corporation
05.2015 - 03.2021
- Developed efficient stored procedures for complex business logic, reducing processing time and improving system stability.
- Evaluated new technologies related to SQL development for potential implementation opportunities, facilitating innovation within the team's workflow.
Senior Software Engineer
SolarWinds
11.2014 - 04.2015
- Developed applications for storage and network monitoring
Senior Software Engineer
FIS Global Business Solutions
07.2010 - 11.2014
- Designed and developed batch application for prepaid card fee processing and settlement.
Education
B.Tech/B.E. - Computer Science
Dhaanish Ahmed College of Engineering
Chennai, IN04-2010
Skills
- SQL
- Azure Databricks
- Pyspark
- Spark Streaming
- CICD Pipeline
- Azure DevOps
- Azure Data Factory
- Power BI
- MS SQL
- Github
- ADLS
- Delta Live Table & AutoLoader
Certification
- Databricks Certified Data Engineer Professional
- Databricks Certified Spark Developer 3.0
- AWS cloud practitioner
ACCOMPLISHMENTS
- Delivered FATS Mainframe migration project on time with UAT issue fixed and user documentation
- Optimized critical on-demand reporting job with query refactoring & optimization, dataframe cache, cluster configuration tune & delta lake optimization
- Developed a source control & code repository for databricks jobs in github and created CICD pipeline on databricks asset bundle for deployment
- Migrated ADF pipelines those were running notebooks through interactive cluster to job cluster using databricks job APIs
Awards
- Received Champion Award for FATS mainframe migration project and it was challenging to rewrite cobol & pl1 into pyspark data frame equivalent.
- Received Kudos Award for the innovation & implementation of databricks jobs deployment through databricks asset bundle workflow.
Languages
English
Professional Working
Timeline
Data Solution Architect
DXC Technology
10.2022 - Current
Senior Data Engineer
FIS Global Business Solutions
04.2021 - 09.2022
SQL Developer
NEC Corporation
05.2015 - 03.2021
Senior Software Engineer
SolarWinds
11.2014 - 04.2015
Senior Software Engineer
FIS Global Business Solutions
07.2010 - 11.2014
B.Tech/B.E. - Computer Science
Dhaanish Ahmed College of Engineering
Similar Profiles
Syed NadeemuddinSyed Nadeemuddin
Professional 2 Networks at DXC Technology (Was transitioned from AT&T to DXC)Professional 2 Networks at DXC Technology (Was transitioned from AT&T to DXC)
Henry Arthur VincentHenry Arthur Vincent
Client Capability Lead at DXC Technology, DXCClient Capability Lead at DXC Technology, DXC
Nikhil C TNikhil C T
Business Analyst / Manual QA Tester at DXC Technology/Fidelis Technology Services Pvt LtdBusiness Analyst / Manual QA Tester at DXC Technology/Fidelis Technology Services Pvt Ltd
Usha SrinivasanUsha Srinivasan
Pre-Primary School Teacher at St. Mark's Matric Higher Secondary SchoolPre-Primary School Teacher at St. Mark's Matric Higher Secondary School
LOKESH KUMAR CLOKESH KUMAR C
Senior CAD/BIM Technician at Systra MVA Consulting INDIA Pvt LtdSenior CAD/BIM Technician at Systra MVA Consulting INDIA Pvt Ltd