A Product Engineering Skill That No One Talks About

A Guide To Becoming A Better Software Product EngineerPhoto by Jen Theodore on UnsplashAnyone can build a product, but building and maintaining a product that progresses in the market requires much effort. Software engineers love problem-solving and bu…

A Guide To Becoming A Better Software Product Engineer

Photo by Jen Theodore on Unsplash

Anyone can build a product, but building and maintaining a product that progresses in the market requires much effort. Software engineers love problem-solving and bug fixing is something they face throughout their coding journey. Bug fixing is fun and games until it happens in the production environment and customers are being impacted in real-time.

Incident Management is an unpopular soft skill that every software product engineer must possess

Case Study

Mary works for an e-commerce company that sells shoes online. The applications in the organization are built as microservices and they have over 10 services with 5+ engineers working in each service. Mary is a senior engineer and there are 3 junior engineers and 1 intern on her team. Mary reports to her tech lead who also manages a few other microservices.

During a routine weekly release, someone in Mary’s team pushed a code to production that allows customers to edit the price of the shoe and complete the purchase with the edited price. It’s 6 pm on a Friday and Mary’s manager calls her to inform her that something’s wrong in the production environment.

How can Mary solve this problem?

How can you improve your incident management skills?

There is no definite answer to this as it can vary from industry to industry. Based on my personal experiences, I’ve compiled 10 important points that software product engineers can follow during an incident.

  1. Understand the problem
    First and foremost, Mary should spend some time understanding what exactly the problem is. It’s okay to spend a reasonable amount of time here until Mary completely understands the problem. It’s better than spending time, solving the wrong problem.
  2. Stay calm. Relax, take some time to process
    Mary was just about to begin her weekend and the call could’ve been very frustrating. You never know when incidents could happen. What’s important is, to enter the incident war room with a fresh mind. Mary decided to calm down, hydrate herself and then enter the war room.
  3. You might be a jack of all trades but reach out to the right people before it’s too late
    Mary has been an engineer all her career and she has been one of the best performers in the organization. This does not mean that she should debug everything by herself. She found out the commits that could’ve caused the impact and requested for the junior developers to join the call.
    She also keeps her manager and tech lead informed on the status of the issue, the next steps, and an eta for the fix.
  4. Communicate clearly, have empathy
    Mary explains the issue to her peers calmly and proposes a fix
  5. Follow a no-blame culture
    Though Mary is personally not happy with the incident, she does not vent it out to her peers. She believes that this is part of the job and that mistakes can happen.
  6. Analyze the impact
    While the other members of the team are working on a fix, Mary decides to find out how many customers have placed orders by bypassing the actual price.
    She passes these customers’ details to the operations team to check if these orders can be rectified in parallel.
  7. Find how to stop the impact from worsening immediately
    Mary comes up with a plan to stop the platform from allowing any more orders for the time being. She informs the front-end engineers to display a maintenance banner.
  8. Find a fix, test it thoroughly, apply the fix
    Once the junior developers code the fix, Mary personally reviews the code thoroughly, and requests to test it in the QA environments. Once all the test checks have passed, she requests for deployment and to pull down the maintenance banner.
  9. Observe and Monitor
    Once the fix is deployed, Mary requests production testing and personally monitors a few orders that were placed after this. After confirming that everything is fine, she hands over the monitoring responsibility to a support engineer and signs off.
  10. Find the RCA, and take ownership of the postmortem. See how you can avoid such incidents in the future
    The following week, she regroups with her peers to identify how this edge case was missed and how this can be avoided in the future. She suggests raising alerts in case of price discrepancies, throwing errors if the values are tampered and writing more unit tests to test all the scenarios.
https://memegenerator.net

Conclusion

Incident management is not widely spoken in the software engineering community and I believe this is a soft skill that can be developed only by facing many incidents. In certain organizations, the tech lead fully holds the responsibility of incident management. If you’re a software product engineer aspiring to become a tech lead, you could start working on your incident management skills by shadowing your senior peers.

https://media.makeameme.org/

Regardless of having the best engineers and using the best tools, you could still face incidents. Use every opportunity and learn from it to groom yourself for the better.


A Product Engineering Skill That No One Talks About was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.


Print Share Comment Cite Upload Translate
APA
Muralish Clint | Sciencx (2024-03-29T00:50:30+00:00) » A Product Engineering Skill That No One Talks About. Retrieved from https://www.scien.cx/2022/09/27/a-product-engineering-skill-that-no-one-talks-about/.
MLA
" » A Product Engineering Skill That No One Talks About." Muralish Clint | Sciencx - Tuesday September 27, 2022, https://www.scien.cx/2022/09/27/a-product-engineering-skill-that-no-one-talks-about/
HARVARD
Muralish Clint | Sciencx Tuesday September 27, 2022 » A Product Engineering Skill That No One Talks About., viewed 2024-03-29T00:50:30+00:00,<https://www.scien.cx/2022/09/27/a-product-engineering-skill-that-no-one-talks-about/>
VANCOUVER
Muralish Clint | Sciencx - » A Product Engineering Skill That No One Talks About. [Internet]. [Accessed 2024-03-29T00:50:30+00:00]. Available from: https://www.scien.cx/2022/09/27/a-product-engineering-skill-that-no-one-talks-about/
CHICAGO
" » A Product Engineering Skill That No One Talks About." Muralish Clint | Sciencx - Accessed 2024-03-29T00:50:30+00:00. https://www.scien.cx/2022/09/27/a-product-engineering-skill-that-no-one-talks-about/
IEEE
" » A Product Engineering Skill That No One Talks About." Muralish Clint | Sciencx [Online]. Available: https://www.scien.cx/2022/09/27/a-product-engineering-skill-that-no-one-talks-about/. [Accessed: 2024-03-29T00:50:30+00:00]
rf:citation
» A Product Engineering Skill That No One Talks About | Muralish Clint | Sciencx | https://www.scien.cx/2022/09/27/a-product-engineering-skill-that-no-one-talks-about/ | 2024-03-29T00:50:30+00:00
https://github.com/addpipe/simple-recorderjs-demo