Case Study: Streamlining Legacy Processes

Client

HR startup providing services to small business owners

Project Description

The client, a prominent startup providing HR services to small employers, relied on a single-tenant instance of enterprise compliance software to file payroll tax forms with federal, state, and local agencies. The existing process synchronized payroll data from the production Rails environment to the compliance software every morning, but it was highly unreliable. Manual intervention was required after a crash, and the lengthy resolution process often forced the operations team to stay late to file forms before end-of-day deadlines. This problem was exacerbated during periods of high system volume, such as early pre-holiday pay periods and end-of-month contractor payrolls.

Solution

• Process Management/Monitoring Solution: Developed a process management and monitoring solution to oversee the sync processes within the production Rails environment.
• Modernized Sync Processes: Wrote replacement workflows to synchronize the files between systems, incorporating retry attempts, result aggregation, and in-progress monitoring.
• Improved Observability API: Built a custom integration with the enterprise software so that we could better understand the progress of the sync and alert on failures more rapidly.

Results

• Improved Reliability: The new workflow code gained the ability to self-recover in most common error cases.
• Automated Monitoring and Alerts: Monitoring mechanisms triggered pager alerts to the operations team and engineers when the workflow was unable to recover automatically, ensuring swift response to potential issues.
• Empowered Engineers: The improved monitoring empowered the engineering team to proactively understand their system and address other issues within the sync workflow before they became serious.
• Zero Infrastructure Changes: The new sync process was seamlessly deployed to production within the existing Rails app, so no infrastructure changes were required.
• Reduced Labor Costs: The improved workflow reduced frequency of error from multiple times a week to less than once a month, minimizing nights of required overtime and allowing the operations team to reclaim their time.

Conclusion

Our implementation of a robust workflow significantly improved the reliability of the compliance software sync for the client. The operations team gained the ability to confidently manage the critical daily deadline, resulting in a substantial reduction in required overtime. The engineering team became empowered to proactively resolve issues.