SquareOps
Get an architecture review
CASE STUDYSaaS

Migration of MongoDB & Elasticsearch to AWS

How BatchService moved 12TB+ of MongoDB and Elasticsearch off Atlas and Elastic Cloud onto AWS EKS, cutting $30K/month hosting costs.

January 29, 20252 min read

On this page

Client Overview

BatchService.com, a leading property technology company since 2018, provides data-driven tools for real estate professionals. Their products, including lead generation and skip tracing platforms, serve a diverse clientele, from solo entrepreneurs to tech firms.

To enhance control, cost efficiency, and scalability, BatchService aimed to migrate its MongoDB and Elasticsearch workloads from Mongo Atlas and Elastic Cloud to AWS on Elastic Kubernetes Service (EKS).

Challenges

Massive Data Volumes:

  • MongoDB clusters with 0.5 TB to 1 TB per cluster.
  • Elasticsearch clusters storing 12 TB+ of data across staging and production.

High Hosting Costs:

  • Monthly expenses exceeded $30,000, increasing with data growth.

Limited Flexibility:

  • Restricted hardware configurations, backup, and recovery options.

Scalability Issues:

  • Managed service tiers imposed scaling limitations.

Compliance Requirements:

  • Needed HIPAA, PCI-DSS compliance and enhanced data security.


BatchService sought a self-managed AWS architecture that ensured high availability, cost savings, and operational efficiency.

Solution

SquareOps, an AWS Advanced Tier Partner, executed a full-scale migration strategy, ensuring a seamless transition with zero data loss.

Infrastructure & Architecture Upgrade

  • MongoDB Migration: Deployed high-availability replica sets on AWS EKS.
  • Elasticsearch Migration: Used Elastic Cloud on Kubernetes (ECK) for shard optimization and performance scaling.

Zero-Downtime Data Migration

  • MongoDB: Live migration via replica set synchronization to ensure data consistency.
  • Elasticsearch: Migrated 12 TB+ of data using snapshot and restore, plus real-time reindexing.
    Migrated 15+ production clusters and 10 staging clusters with zero downtime.

Performance Monitoring & Backup Strategy

  • Grafana Dashboards: Monitored 200+ performance metrics (query latency, disk IOPS, replica lag).
  • Advanced Backups: Implemented collection-level backups for MongoDB and index-level backups for Elasticsearch, encrypted and stored in AWS S3.

Security & Compliance Enhancements

  • End-to-End Encryption: Secured data at rest & in transit using AWS KMS and TLS.
  • Access Control: Applied role-based access control (RBAC) & audit logging for HIPAA & PCI-DSS compliance.

Cost Optimization

Migrated workloads to self-managed EKS clusters, cutting hosting costs by 27% (from $30,000 to $22,000/month).

Results & Business Impact

  • Seamless Migration: Transferred 12 TB+ of data, 5+ billion documents, zero data loss.
  • 30% Faster Query Performance: Improved database response times and ensured 99.99% uptime.
  • Cost Reduction: Achieved $8,000 monthly savings ($96,000 annually).
  • Operational Resilience: Integrated automated failovers & real-time observability via custom monitoring dashboards.
  • Scalability: Enabled dynamic cluster scaling, ensuring future-proof growth.

Conclusion

BatchService's migration to AWS successfully enhanced control, security, and cost efficiency while ensuring seamless scalability. The collaboration with SquareOps underscores their expertise in large-scale cloud migrations, ensuring high-performance, cost-effective, and future-ready solutions.

Thinking about migrating your databases to AWS? Contact SquareOps today!

FAQs

BatchService case study: FAQs

SquareOps moved BatchService's MongoDB clusters to high-availability replica sets on Amazon EKS using live migration through replica set synchronization, which keeps data consistent while the old and new clusters run side by side. In total, 15+ production clusters and 10 staging clusters moved with zero downtime and zero data loss.

Elasticsearch now runs on Amazon EKS with Elastic Cloud on Kubernetes (ECK), which also allowed shard optimization. SquareOps moved the 12 TB+ of data with snapshot and restore, plus real-time reindexing to catch up on new writes. Across MongoDB and Elasticsearch, more than 5 billion documents moved with no data loss.

BatchService's hosting costs for MongoDB and Elasticsearch fell by 27% after moving from Mongo Atlas and Elastic Cloud to self-managed clusters on Amazon EKS. Query performance also improved by 30%, uptime is 99.99%, and clusters can now scale without the limits of the managed service tiers.

Data is encrypted at rest and in transit with AWS KMS and TLS. Role-based access control and audit logging cover HIPAA and PCI-DSS needs. MongoDB gets collection-level backups and Elasticsearch index-level backups, encrypted and stored in Amazon S3. Grafana dashboards track 200+ metrics such as query latency, disk IOPS and replica lag.

Planning a similar project?

Start with a free architecture review. The engineers who run engagements like this one look at your setup and tell you what they would change first.

Newsletter
Cloud ops notes, monthly

Kubernetes upgrades, cost findings and incident write-ups from real engagements.