Sumit Mittal

One more video added to this series for Data Engineers. • Why is ONE Spark Task So Slow? youtube.com/shorts/bByPB8kWe… • Your Spark job suddenly became 5x slower? youtube.com/shorts/7wMromBRZ… • Your production pipeline failed at 2 AM? youtube.com/shorts/X3ubwwPXG… • Two candidates have similar resumes. Only one gets hired. Who would you choose? youtube.com/shorts/Z4KE7Qg9H… • Parquet is already fast. So why does Delta Lake exist? youtube.com/shorts/q51_dMa1l… Before watching each one, pause for a moment and think about your answer. That's how real learning happens. Which scenario should I turn into the next shorts?

If you are a Data Engineer, here are 4 deep-dive articles worth saving. Real experiments. Real numbers. Spark UI observations. And lessons you can use when things go wrong in production. Save this post. You might want to come back to these. - What happens when you read 40 Billion Records with Databricks Disk Cache? medium.com/@sumitmittal-tren… - How did AQE reduce a Spark job from 45 minutes to 21 minutes? medium.com/@sumitmittal-tren… - Parquet is already fast. So why does Delta Lake exist? medium.com/@sumitmittal-tren… - Why did 29,200 small files become a performance problem for Spark? trendytech.in/the-small-file… And the next experiment? A Spark join that refused to get faster even after adding more compute. That's what I'll break down next. Until then, what production Spark problem should I experiment with next?

Here are Top 7 playlists for Data Engineers 1. Python for Data Engineers - youtube.com/watch?v=Q4nbwea2… 2. SQL - youtube.com/watch?v=zAOUpVM6… 3. SQL Advanced - youtube.com/watch?v=zg9GNdX-… 4. SQL Leetcode - youtube.com/watch?v=qZfMYcHH… 5. Git & Git Hub for Data Engineers - youtube.com/watch?v=C8MBkyqw… 6. Data Engineering mock Interviews - youtube.com/watch?v=HDNxFiRr… 7. Azure Data Engineer Project - youtube.com/watch?v=d3Vw3VtK… I hope this will help all the aspiring Data Engineers. PS~ My new Data Engineering Batch is starting this Saturday. DM to know more! #bigdata #databricks #apachespark

When the going gets tough, the tough gets going. Yes, this has been one of the common qualities of my students who; Never give up, keep working hard, don’t fear transitioning, cross all hurdles and reach new growth levels. Here are two hardworking students who recently joined new companies with amazing growth. Mounya Bheemreddy joined Genpact as Senior Data Engineer. Rohith Cheedella joined TCS as Data Engineer. What fueled their growth was a strong command of PySpark, Databricks and Azure Data Factory. I wish you both much more success ahead in this exciting Data Engineering domain. My new batch of most popular, Ultimate Data Engineering Masters Program is starting tomorrow. DM me for more details.

I will be coming Live Tomorrow for a Free Career Guidance Session, 31st Oct @ 10:30 AM (IST) If you belong to any of these domains, this session can become a life changer for you. - Software Development - Testing, QA - ETL, Support - App. Development - Data Analytics & Other Traditional Technologies Having trained 30000+ professionals over last 5 years who successfully transitioned to Data Engineering, they also once felt stuck working on traditional tech stacks. Most common queries professionals from any IT domain have are; - Prerequisites which I should know - What about my past experience - What growth can I expect - How much time is required - What should be the roadmap I will be addressing all these queries or any other specific doubt of yours, related to Data Engineering transition Live in this session. 👉 Register here (Limited slots): us06web.zoom.us/meeting/regi… Date: 31st Oct 2026 Time: 10:30 AM (IST) Also Join WhatsApp Community Group for Session Updates 👇 chat.whatsapp.com/FlMsJVIJnG… Looking forward to meeting you in the Session. Note - If you are a Trendytech student then you can skip this, as you already know all of this :)

Proud to have impacted careers in Data Engineering & AI across 23+ countries All the credit goes to my students who believed in me, especially when I haven’t met them in person. Sharing some recent successes of my students, who joined new companies as Data Engineers in various parts of the globe. UAE – Ashwinth Dharumaseelan joined Ali & Sons Holding as Data Engineer Ireland - Haritha Reddy joined Johnson & Johnson as Lead Data Engineer USA – Sachin D. joined The World Bank Group as Lead Data Engineer Argentina – Imam Abdullah Khan joined Craft Labs as Data Engineer (Remote) Ireland – Sharayu Zodpe joined JP Morgan Chase as Data Engineer 2 Many of my students have now started another upskilling journey with me, focusing on Agentic AI & FDE training. Having trained more than 30000 professionals in last 5 years, I again share this milestone with all my beloved students for their love, support and trust.

My Ultimate Data Engineering program has been producing solid results over years. It is a very proud moment for me, seeing my students getting new Data Engineering roles every day and joining new companies with amazing growth. I am sharing two brilliant students who recently achieved a new career milestone. Vishal Gupta joined HCL Tech as Lead Data & AI Engineer Shivam Gupta joined HashedIn by Deloitte as Data Engineer Mastering Databricks and having a solid thought process around, Optimization techniques, Structured streaming and building efficient data pipelines made them unstoppable. Data Engineers today act as the backbone and play a very critical role in this evolving AI era. If you are looking for a solid career ahead, do explore Data Engineering as your next upskilling goal.

The demand for Data Engineers Is soaring - and these brilliant students have turned opportunities into offers. With lot of work happening around AI, a significant increase in demand is noticed for Data Engineers having; A very clear though process, in-depth understanding of internals, expertise in Databricks and optimization techniques. Congratulations to these hard-working and very enthusiastic students on their new joining. Anil Kumar Reddy Kancham joined as Lead Data Engineer at Persistent Systems Naveen Kumar Medisetti joined TCS as Senior Data Engineer Vishal Sharma joined Navikenz as Data Engineer Wishing all of you amazing journey ahead in this very promising Data Engineering domain. To everyone feeling stuck or not sure about their future in traditional IT roles; Transition to Data Engineering is 100% possible. 1000s of my students have done it. You too definitely can. You just need to start.

A solid preparation with Databricks expertise got him this new role with amazing hike. Congratulations, Harsh Jain, on joining Coforge as Senior Data Engineer. What makes his journey special is; - His realization to get into this data engineering domain for a better future - Commitment towards learning new tech stack with a zeal to grow - His not giving up attitude, topped up with hard work A strong hold on fundamentals, clarity around internals of Pyspark and mastering Azure cloud; helped him grab multiple opportunities and earn appreciation for his work. Once again, wishing you the very best in this new role. Seeing lots of my students sharing their success is the greatest achievement for me as a teacher. Wishing each of you success in whatever big goal you are pursuing. The biggest step is to start. Once you begin, stay consistent - the results will definitely follow someday.

Starting my teaching career with 4 students to now hosting 3rd season of Student Success Celebration at Taj, Bengaluru It was a very tough call to shortlist 125 students from 600+ applicants. The Invitees include students who have cracked Top companies, achieved exponential growth, secured multiple offers, amazing CTCs, and more. I am very excited to host the Grand Achievers Celebration on 10th October 2026. Every student of mine is special, but I had to make this call due to space constraints. It all started with a small rented room, lots of hope, and trust in myself. Having now trained 30,000+ IT professionals, this was all possible because of; My students who trusted me and their families who supported them to take a step ahead, and reach greater heights. The motto has always remained the same: Making Solid Data & AI Engineers. Let’s make this event a memorable one with lot of Fun, Networking and Celebration.

An airline's chatbot promised a customer a refund that didn't exist. Guess who paid? ⠀ The airline. ⠀ A man had just lost his grandmother. ⠀ He needed to fly, fast. So he went to Air Canada's website and asked the chatbot a simple question: does the airline offer bereavement fares? ⠀ The chatbot was helpful. Confident. Clear. ⠀ Book your ticket now, it said. Then claim a partial refund within 90 days. ⠀ So he did. ⠀ Then he applied for the refund. ⠀ The airline said no. That's not our policy. ⠀ The chatbot had invented it. ⠀ And then came the most surprising part. ⠀ In court, the airline argued that the chatbot was a separate entity, responsible for its own actions. ⠀ In other words: don't blame us. Blame the bot. ⠀ In 2024, the court rejected that argument. The airline had to pay. ⠀ If your AI says it, you said it. ⠀ This is the part of AI agents that demos never show you. ⠀ An LLM doesn't look up facts. It predicts the most likely next words. Without being connected to your real policies and data, it can sound completely sure of itself and still be completely wrong. ⠀ Put that LLM inside an agent that talks to customers, and every confident mistake becomes your company's mistake. ⠀ So if you're building your first AI agent, my advice is simple: ⠀ Start internal. ⠀ Build it for your own team first. An HR policy assistant. A tool that answers questions from internal documents. An agent that watches for failures and alerts the team. ⠀ When an internal agent gets it wrong, a colleague catches it. When a customer-facing agent gets it wrong, you might end up in court. ⠀ Make your mistakes where they're cheap. ⠀ Then, once it's grounded, tested and guarded, let it talk to customers. ⠀ Would you trust an AI agent to talk to your customers today? My new Agentic AI Engineering Program is starting on 17th Oct. DM to know more!

1 Bold Decision -> Solid Data Engineering career -> New Joining A big congratulations to these very hard-working students, who changed their career trajectory with upskilling. Renukaprasad D S joined CGI as Senior Data Engineer Pranay Sonkusare joined as Senior Data Engineer at Capgemini Abhishek Tiwari joined TCS as Senior Data Engineer Their amazing hike, reflects their strong fundamentals and expertise around PySpark and Databricks. Going beyond just learning tools, they focused on optimization techniques, developing thought process, and effective utilization of resources. With no looking back, they worked on building and managing end-to-end data pipelines. This switch takes their careers to new heights. To everyone – Do access the future of the domain you are working in and take relevant steps today towards upskilling for a secured and brighter future.

I am hosting a Free Live Career Guidance Session Tomorrow, 3rd October @ 11:30 AM (IST) Being into training IT Professionals from last 5 years and helping more than 30000 people transition successfully to Data Engineering; I exactly know where do people get stuck in taking that 1 Life Changing Decision. Most common queries professionals from any IT domain have are; - Prerequisites which I should know - What about my past experience - What growth can I expect - How much time is required - What should be the roadmap I will be addressing all these queries or any other specific doubt of yours, related to Data Engineering transition Live in this session. Who should attend this Session ? ANYONE Working In IT Domain and want to build a promising career in Data Engineering, This Session is for YOU 👉 Register here (Limited slots): us06web.zoom.us/meeting/regi… Date: 3rd Oct 2026 Time: 11:30 AM (IST) Also Join WhatsApp Community Group for Session Updates 👇 chat.whatsapp.com/FtCYBEheco… Looking forward to meeting you in the Session! Note - If you are a Trendytech student then you can skip this, as you already know all of this :)

I will be coming Live Tomorrow for a Free Career Guidance Session, 3rd Oct @ 11:30 AM (IST). Being into training IT Professionals from last 5 years and helping more than 30000 people transition successfully to Data Engineering; I exactly know where do people get stuck in taking that 1 Life Changing Decision. Most common queries professionals from any IT domain have are; - Prerequisites which I should know - What about my past experience - What growth can I expect - How much time is required - What should be the roadmap I will be addressing all these queries or any other specific doubt of yours, related to Data Engineering transition Live in this session. Who should attend this Session ? ANYONE Working In IT Domain and want to build a promising career in Data Engineering, This Session is for YOU 👉 Register here (Limited slots): us06web.zoom.us/meeting/regi… Date: 3rd Oct 2026 Time: 11:30 AM (IST) Also Join WhatsApp Community Group for Session Updates 👇 chat.whatsapp.com/FtCYBEheco… Looking forward to meeting you in the Session! Note - If you are a Trendytech student then you can skip this, as you already know all of this :)

Think you are ready for a Senior Data Engineer interview? Here is what companies like Mastercard, Databricks, Visa, IBM and others are actually asking - a compilation of recent interview questions shared by my students. 1. Mastercard - lnkd.in/gDVaMdkz 2. Databricks - lnkd.in/gBWx8VcJ? 3. Visa - lnkd.in/gQWrPqr3? 4. EY - lnkd.in/gVEmTdBP? 5. PwC - lnkd.in/gq-jSZfc? 6. Deloitte - lnkd.in/g85s9w_b 7. IBM - lnkd.in/gWaSCRAJ 8. Tiger Analytics - lnkd.in/gkqWac8m? 9. LTI Mindtree - lnkd.in/gg5iH7KT? 10. Persistent - lnkd.in/g9n7_82c? I hope this will help you in cracking these interviews smoothly. My New Batch of Most Popular, "Ultimate Data Engineering Masters Program" is starting this Saturday 3rd October 2026. DM to know more!

Indian Air Force -> State Bank of India -> Senior Data Engineer With changing industry dynamics, one who is flexible and ready to adapt to changes will not only grow but also ensure a promising future. The willingness to learn new technology and putting in 100% effort ensured a smooth transitioning for this very hard-working and fearless student towards newer heights. Congratulations, Naveed Shaikh, on joining RIB Software as Senior Data Engineer. A strong fundamental knowledge and expertise in; PySpark, Distributed storage, Azure Databricks, Data Factory, AWS EMR, Glue, S3 and more helped him build a solid career in this domain. I am sure, journey of Naveed would inspire many who doubt themselves towards taking the first step towards upskilling and transition to something which is more relevant.

A strong command of PySpark and Databricks elevated their careers to the next level as solid Data Engineers. One decision to align your skills with a domain holding significant potential and a bright future can change the entire dynamic of your career and life. A big congratulations to these sincere and very hard-working students on their new roles at TCS with great hike. Surabhi Roy joined as Lead Data Engineer Vaibhav Mane joined as Senior Data Engineer Their confidence not only made them perform their best on complex data projects but also made them enjoy their work, gain appreciations and grow.

When I see my brilliant students reaching Architect positions, the feeling is different. Congratulations Krishnaraj Rajappan on joining as Senior Data Architect at TEKsystems. His deep understanding of internals, thought process around optimization and focus on Databricks and Performance Tuning, made him stand out and grow to this level. This is what differentiates a normal Data Engineer from a Top Data Engineer. Today, when I see thousands of my students, working at Top product companies in very good positions; I feel truly content and happy that my teachings are creating such an impact on their careers. With increasing Data and many developments around AI, the role of Data Engineers has become more prominent. Do explore building a solid career in this domain.

Starting his career as System Engineer, this brilliant student now joined Siemens Energy as Lead Data Engineer Vishal Sharma kept upskilling and gaining practical insights with in-depth learning around; - PySpark, Large scale cloud migration, - Azure Databricks, Data Factory, - Optimisation techniques, Structured Streaming and more. His zeal for growth and sincerity enabled him to seize opportunities and advance in this promising domain. A big congratulations to you Vishal. Wishing you the very best in this role. To everyone who are uncertain about their career’s future; One step towards transition can take your career years ahead with multi-fold growth.

3 Journey to amazing growth and success in Data Engineering. Proud of my students who have put in all their hard work, efforts and upskilled for promising roles in this domain. Shubhandu Sharma joined MathCo as Manager – Data Engineering Ayushi Jain joined Persistent Systems as Lead Data Engineer HARSHAVARDHAN REDDY SADHU joined TMUS Global Solutions as Senior Data Engineer A structured learning path from; - Building strong fundamentals around Distributed storage and processing, - In-depth knowledge of PySpark Internals, - Mastering Azure Databricks, Data Factory, - Adding AWS cloud services as 2nd cloud knowledge, - To creating and managing complex data pipelines, They went unstoppable. Congratulations to each of you; I wish you the very best in your new roles and bigger responsibilities. To everyone finding stuck with growth or are uncertain about their role ahead in traditional tech stacks; Do explore Data Engineering to build a promising and secured future ahead.