Feature #10972
openCustomer Creation Backend Architecture — Synchronous Email Cascade Decoupling
0%
Description
Investigated system-wide latency spikes and timeout failures during customer creation, and re-architected the backend transactional notification pipeline (customerService.js, customerController.js).
Background & Problem Statement:
Operations teams reported recurrent failures when submitting new customer registrations, encountering "Failed to create client: No response from server" alerts. Diagnostic database queries revealed that customer records were successfully inserted into MongoDB, but frontend HTTP requests were getting terminated by client-side gateway timeouts (30 seconds).
Root Cause & Technical Analysis:
Profiling the customer creation request lifecycle revealed that upon database insert, the backend synchronously executed an unbuffered sequential chain of external email dispatches:
1. Compiling and sending the customer welcome email via SMTP.
2. Querying all SuperAdmins in the tenant organization and dispatching individualized email notifications.
3. Dispatching assignment alert emails to designated assigned managers and site personnel.
Whenever SMTP network latency exceeded normal thresholds or the mail provider throttled traffic, this sequential chain exceeded 30 seconds, causing frontend request aborts despite successful database writes.
Architecture Redesign & Implementation:
1. Decoupled Asynchronous Notification Pipeline:
- Segregated the transactional database write from administrative notification dispatches.
- Converted bulk SuperAdmin and assigned staff notifications into detached, non-blocking background executions.
- Added isolated error boundaries and structured logging for asynchronous workers so external SMTP failures never affect customer onboarding.
2. Latency Optimization:
- Reduced the API response time of the customer creation endpoint from over 25 seconds down to ~110ms.
- Handlers now return the newly created customer document and HTTP 201 Created immediately after MongoDB transaction commit.
3. Resiliency Testing:
- Simulated network degradation and SMTP latency delays of up to 10 seconds per message to ensure zero timeouts on the client side.
Subtasks
Related issues