What did I learn while working as a data scientist?

My experience as a data scientist at a large startup. Why is real-world A.I/M.L so different from academia? How do we appropriately integrate business and science/engineering?

3 min readAug 9, 2022

Learning Phase One

I worked as a data scientist at Swiggy for 1 year, a large Indian food delivery startup (around 20 million active users per month). Unlike other teams (such as recommendations, etc) where if the prediction is a bit off then the teams can quickly look into and push changes, I worked in the logistics team where every second was essential. I enjoyed every second of the intense and highly rewarding work. I would also like to thank my mentors for believing in me and guiding me at every step possible.

With all the innovations happening in data science/A.I field almost every day, one can lose sight of why is it so difficult to extract practical value from a wonderfully written research paper. This is not just valid for the data science field but almost any field.

A.I/M.L is extremely fuzzy with its two driving pillars, engineering, and science. A.I is almost identical to biology, while it seems extremely simple, the processes occur in a highly complex and efficient manner.

At Swiggy, I realized the importance of business value in data science where every second was essential. I realized what it means to be simple but complex. Understanding the business problem was 10x more important than deciding on the algorithm. While we could have applied any advanced algorithm, we needed explainability and robustness. Thinking over the business problem may sound boring but to truly harness the power of data science and Artificial Intelligence understanding the data is absolutely critical.

If I had an hour to solve a problem I’d spend 55 minutes thinking about the problem and five minutes thinking about solutions.
- Einstein

Learning Phase Two

In my next Data Science and Artificial intelligence endeavor, I worked for a medical dental company for about 1.5 years where I truly became one with the data. With a huge influx of data collected by the machines, my job was to implement algorithms to better understand/predict machine failures. To understand the data is to accurately identify the use cases by asking the right questions. Asking the right questions is 100 times more difficult than finding the correct answer.

A lot of times we are made to think that business and engineering/science are two different things but in reality and particularly in this field, they are the same thing. It is as if it is the same coin with one side being business and the other side being science.

Learning Phase Three

I also worked for a number of research labs during my undergrad (I just graduated this May!). Unlike the industry, there was a clear distinction where the main aim of the project was to find the right algorithm for the already cleaned and defined data. Particularly this involved beating a specified benchmark. This is extremely different from the industry as a lot of times we encounter data drift, i.e the model that was getting a substantial accuracy does not work for a production environment.

Conclusion

Data Science or A.I is not about having a Ph.D. or being the smartest person. It’s about asking the right questions and staying humble. Particularly A.I is seen as an engineering science where implementing ideas is just as important as inventing them. No matter what anyone tells you, DS is not glamorous, it is a lot of applied grunt work with a focused business goal. Implementing these techniques in your business requires a lot more than building a complex M.L model.

Need Help?

Need help turbocharging your business with cutting-edge A.I/data science research in a practical way? Data Kapoor is there for you! We welcome you to become part of our 7+ happy customers!