Feature #10491
openDeployment, My usage feature and web scrapper application Investigation
100%
Description
Analyzed the current web scraper application
Reviewed the application's resource consumption and runtime behavior.
Observed high memory usage, including cases where memory increased to around 350 MB.
Observed high CPU utilization, sometimes reaching 100%+, causing the application to become unresponsive.
Investigated memory/cache usage
Researched what could be occupying memory during bulk scraping/email-related processing.
Looked into application-level caching, temporary data, queues/batches, and memory buildup during large workloads.
Considered whether memory is being released properly after processing.
Investigated performance improvements
Researched ways to improve scraper throughput and stability.
Considered batch processing, concurrency control, worker processes, memory limits, and resource optimization.
Evaluated how to prevent the scraper from consuming excessive CPU/memory during bulk operations.
Analyzed application failure/recovery behavior
Investigated the scenario where the application appears to be running, but APIs stop responding.
Identified a potential issue where MongoDB failure/reconnection problems can leave the application in a running state while API requests are no longer processed correctly.
Researched better health checks and automatic recovery mechanisms.
Researched Docker deployment
Evaluated whether containerizing the web scraper with Docker would improve deployment, isolation, resource management, and recovery.
Considered Docker-based restart policies and container resource limits.
Evaluated Kubernetes
Researched whether Kubernetes is necessary for the web scraper.
Compared Docker vs Kubernetes based on the application's current scale and requirements.
Concluded that Kubernetes may be unnecessary at the current stage if the application can be effectively managed with Docker/PM2 and proper monitoring.
Considered production reliability improvements
Researched health checks/readiness checks.
Considered automatic container restart on application failure.
Considered monitoring CPU/memory usage and application/API availability.
Looked at ways to prevent MongoDB connectivity issues from making the application appear healthy when APIs are actually unavailable.
implemented "my usage" feature in pos fe.
Deployed code in test environment
Tested pos Fe with team members.
Subtasks
Related issues
Updated by Sai Priyatham Sadineni 28 days ago
- Status changed from New to Resolved