Portfolio of Keith Elder

Process To Change A Typo on QuickenLoans.Com or MyQL or Rocket

by Keith Elder

The Back Story

During a leadership meeting I was listening to leaders talk about our process and making things better. But really as I sat and listened, I came to the conclusion that leaders hadn't really taken a step back to see just how crazy it has gotten and how painful it is. So I spoke up used an example of all the things that would have to happen to fix just a single typo on our front-end web sites. In 2004 or 2005 we could just walk over to a server engineer that had access and fix it, on the spot. But today we are burdened with so many other things and process. After the meeting I was asked to write my example down.

I started this document with the pieces I could outline and then I sourced input from a few other team members closer to the example on the front-end teams. Below is the result of that work with contributions from those listed below. If you've found this document, or have had this document shared to you, please feel free to edit it to capture things that are missing or format the content better.

The Setup

We are going to pretend a typo has made it into production on our public facing client web sites. This typo is within the code base of the site and not within the content management system. The following are all of the steps and estimated times it would take to fix a simple typo on a page within our web site(s).

QuickenLoans.Com

  1. Put a task in task tracker (JIRA/TFS) by the BA or PA
    1. 2 minutes
  2. Prioritized, conversation with Engineer
    1. 1-5 minutes if Engineer is around
    2. 15-60 minutes if Engineer is in a meeting
  3. Engineer pulls latest code
    1. few seconds
  4. Engineer find location of the data
    1. Sr. Engineer - less than 30 seconds
    2. New Engineer - 1-3 minutes
  5. Opens file, fixes.
    1. 10 seconds
  6. Commits to git.
    1. 20-30 seconds
  7. Engineer goes to task tracker and drags item to "Ready for Test" lane.
    1. 30 seconds
  8. Engineer goes into HAL and pushes to test.
    1. 5 minutes (get into hal, build, push)
  9. Engineer manually notifies QA the fix is in test.
    1. 1-60 minutes depending if Engineer is in meeting or at lunch or whatever once testing gets done
  10. QA certifies build in test
    1. 1-15 minutes
      1. screen typo found might require information to be entered to get to it
  11. Engineers pushes build to beta environment
    1. 5 minutes (get into hal, build, push)
  12. Build gets certified by QA re-tests. Now ready for PROD deployment.
    1. 1-15 mintues
  13. QA change review request has to be created in Cherwell.
    1. 5-20 minutes
  14. Change review has to be approved by leader.
    1. 1 minute to 1 day
  15. Change review has to be approved by Senior Leader.
    1. 1 minute to 1 day
  16. Assuming the deployment is going to happen on the same day. BA would reach out to Keymasters in HipChat letting them know they needed someone to do a deploy. Keymasters would verify the change review had been approved.
    1. 1 minute to several hours depending on availability of Keymasters
  17. Keymasters deploy (if Rocket or Myql, this is to out of production servers)
    1. 15 - 30 minutes
  18. QA verifies build in production
    1. 1-15 minutes

MyQL.Com

  1. Put a task in task tracker (JIRA/TFS) by the BA or PA
    1. 2 minutes
  2. New work meeting held to prioritize new story
    1. 15-30 minute meeting, once a week. Or discussion item in 15 minute daily standup.
  3. Prioritized, conversation with Engineer
    1. 1-5 minutes if Engineer is around
    2. 15-60 minutes if Engineer is in a meeting
  4. Engineer pulls latest code
    1. few seconds
  5. Engineer find location of the data
    1. Sr. Engineer - less than 30 seconds
    2. New Engineer - 1-3 minutes
  6. Opens file, fixes.
    1. 10 seconds
  7. Commits to git.
    1. 20-30 seconds
  8. Engineer goes to task tracker and drags item to "Ready for Test" lane.
    1. 30 seconds
  9. Engineer goes into HAL and pushes to test.
    1. 5 minutes (get into hal, build, push)
  10. Engineer manually notifies QA the fix is in test.
    1. 1-60 minutes depending if Engineer is in meeting or at lunch or whatever once testing gets done
  11. QA certifies build in test
    1. 1-15 minutes
  12. Engineers pushes build to beta environment
    1. 5 minutes (get into hal, build, push)
  13. Build gets certified by QA re-tests. Now ready for PROD deployment.
    1. 1-15 mintues
  14. QA change review request has to be created in Cherwell.
    1. 5-20 minutes
  15. Change review has to be approved by leader.
    1. 1 minute to 1 day
  16. Change review has to be approved by Senior Leader.
    1. 1 minute to 1 day
  17. Assuming the deployment is going to happen on the same day. BA would reach out to Keymasters in HipChat letting them know they needed someone to do a deploy. Keymasters would verify the change review had been approved.
    1. 1 minute to several hours depending on availability of Keymasters
  18. Keymasters deploy (if Rocket or Myql, this is to out of production servers)
    1. 15 - 30 minutes
  19. QA verifies build in production
    1. 1-15 minutes
  20. Keymasters rotate load balancers so that new code is now live to clients.
    1. 1-10 minutes
  21. QA monitors logs for errors
    1. 15-30 minutes
  22. Keymasters deploy new build to out of production servers.
    1. 1-15 minutes
  23. QA verifies build on out of production servers
    1. 1-15 minutes

Rocket

  1. Put a task in task tracker (JIRA/TFS) by the BA or PA
    1. 2 minutes
  2. New work meeting held to prioritize new story
    1. 15-30 minute meeting, once a week. Or discussion item in 15 minute daily standup.
  3. Prioritized, conversation with Engineer
    1. 1-5 minutes if Engineer is around
    2. 15-60 minutes if Engineer is in a meeting
  4. Engineer pulls latest code
    1. few seconds
  5. Engineer find location of the data
    1. Sr. Engineer - less than 30 seconds
    2. New Engineer - 1-3 minutes
  6. Opens file, fixes.
    1. 10 seconds
  7. Commits to git.
    1. 20-30 seconds
  8. Engineer goes to task tracker and drags item to "Ready for Test" lane.
    1. 30 seconds
  9. Engineer goes into HAL and pushes to test.
    1. 5 minutes (get into hal, build, push)
  10. Engineer manually notifies QA the fix is in test.
    1. 1-60 minutes depending if Engineer is in meeting or at lunch or whatever once testing gets done
  11. QA certifies build in test
    1. 1-15 minutes
  12. Engineers pushes build to beta environment
    1. 5 minutes (get into hal, build, push)
  13. Build gets certified by QA re-tests. Now ready for PROD deployment.
    1. 1-15 minutes
  14. QA change review request has to be created in Cherwell.
    1. 5-20 minutes
  15. Change review has to be approved by leader.
    1. 1 minute to 1 day
  16. Change review has to be approved by Senior Leader.
    1. 1 minute to 1 day
  17. Assuming the deployment is going to happen on the same day. BA would reach out to Keymasters in HipChat letting them know they needed someone to do a deploy. Keymasters would verify the change review had been approved.
    1. 1 minute to several hours depending on availability of Keymasters
  18. Keymasters deploy (if Rocket or Myql, this is to out of production servers)
    1. 15 - 30 minutes
  19. QA verifies build in production
    1. 1-15 minutes
  20. Keymasters rotate load balancers so that new code is now live to clients.
    1. 1-10 minutes
  21. QA monitors logs for errors
    1. 15-30 minutes
  22. 3 days later... QA change review request has to be created in Cherwell.
    1. 5-20 minutes
  23. Change review has to be approved by leader.
    1. 1 minute to 1 day
  24. Change review has to be approved by Senior Leader.
    1. 1 minute to 1 day
  25. Team schedules deploy with keymasters
    1. 1 minute to several hours depending on availability of Keymasters
  26. Keymasters deploy new code to out of production servers
    1. 1-15 minutes
  27. QA Verifies build
    1. 1 minute if frontend only, 1-30 minutes if windows services involved.