Nepal’s Largest E-commerce System: Journey from Monolith to Distributed Architecture in AWS

About the Customer

Sastodeal (sastodeal.com) is one of the leading B2C Marketplace eCommerce companies in Nepal with millions of customers and thousands of vendors/partners all over Nepal.

Their goal is to ensure that all the users and customers can buy products at affordable prices, get their products delivered as assured, and on time regardless of their location.

During Covid, when the nations went through many months of lockdowns, they have been one of the key players to ensure people get to buy items online. On the other hand, they provide a platform for our vendors to cater to millions of customers without additional cost to their business.

Customer Challenge

The eCommerce application was hosted on a single Magento Cloud Platform. Magento Cloud was more expensive and had no provisions for auto-scaling and customer support was only provided via emails from their ticketing system.

The major challenges faced by the customer were:

• Usage of high CPU/Memory server. At low traffic periods, this server was staying idle incurring costs. Whereas there were unpredictable high traffic periods where the server would become very slow and unresponsive.
• Downtime during product releases. Since the application was hosted on a single server, there was more effort during the product release cycle including possible downtime.
• The storage in the server was not scalable and took more effort in provisioning it manually.
• Lack of security for the application and the database. The instance was accessible from the public internet and susceptible to external breaches. WAF and VPN were not used.
• Monitoring was not adequate which delayed the detection of server failures.

The SastoDeal team wanted to re-architect the infrastructure, use distributed computing by separating applications from a single server, improve security, use multiple environments (Staging and Production), use Infrastructure as a Code, and follow other standard DevOps practices.

They decided to use AWS cloud as it provided a solution to most of the above requirements out of the box with little configuration.

Partner Solution Details with Architecture Diagram

Fig: Sastodeal Architecture

An AWS VPC with a public and private subnet was created for the application. Only Load Balancer was placed in the public subnet which has HTTP and HTTPS ports opened. Application server, Elasticache, and Database are all hosted in the private subnet and can only be connected either through AWS Client VPN or through the load balancer. NAT gateway was configured to allow communication from private subnet to other AWS resources and internet gateway to allow communication from VPC to the internet.

All user requests coming to the Load balancer were routed through AWS WAF. Common rules which protect against SQL injection, cross-site scripting, controlling bot traffics, etc. were set up to protect the web application. The Load balancer then directs the requests to application servers. All the different applications from the single server such as Application Server, Storage, Database Server, ElasticCache, ElasticSearch and RabbitMQ were decoupled into separate resources in AWS Cloud.

Most of the customers for this website are from Nepal, so the application servers were deployed in the Mumbai, India region for low latency and in two availability zones for high availability. The application server was made stateless and added to the Auto Scaling groups. When the CPU utilization of the server is high, during periods of high traffic, new servers are created and added to the pool. When the traffic is low, the servers were set to minimum saving on the costs. EFS volumes were used instead of EBS volumes, as the data in volume needed to be shared across all application servers.

AWS Aurora DB, was configured in place of the application MySQL database. Between application servers and the database, we configured Redis cache so the response to the frequent database queries is faster and lowers the load to the database. The replicas for both the database and cache were created in the second availability zone. Application is now using Reader and Writer for these databases which is then easier to scale.

AWS managed services were used where applicable to reduce the operation overhead of managing these resources. The application used RabbitMQ and ElasticSearch for message queue and search functionality respectively. We configured managed services of AWS MQ and AWS ElasticSearch for them. The application was using a third party email service to send customer emails, and we configured AWS SES to be used instead. We set up a cloudwatch dashboard that collects the CPU and memory utilization metrics for all the instances available in Auto Scaling Group dynamically. Various alarm rules are also created for Redis, Message Queue, RDS, etc. to ensure that issues are visible to the operational team.

We used CloudFormation templates to deploy resources following infrastructure as a code best practice. This helped us create identical staging and production environments. For access management, we created several user groups such as “billing”, “administrator” and “DevOps” and provided a demo to SastoDeal engineering team on how they can manage IAM groups, users, and policy. The root account user is not used.

The application is using Gitlabs CI/CD workflow. When developers commit code in customers’ gitlab, this code is deployed in a lightsail instance and a custom script is executed through the gitlab process. This custom bash script includes AWS CLI commands which creates new Amazon Machine Images(AMIs) with updated application code and then updates the launch configuration of the Auto Scaling group to use this image. Thus, after the application code is committed, the deployment is fully automated.

Timeline

The engagement started on 2nd November 2020, and the production workloads were switched on 5th December 2020. Genese Solution continued to provide support until 21st Jan 2020 by the DevOps engineers.

AWS Services

• Amazon VPC
• Amazon VPC NAT Gateway
• Amazon EC2 Auto Scaling
• Amazon EC2
• Amazon Lightsail
• Amazon Elastic File System (EFS)
• Amazon ElastiCache
• Amazon Aurora
• Amazon MQ
• Amazon Elasticsearch Service
• Amazon Cloudfront
• Amazon SES
• Amazon Cloudwatch
• Internet Gateways
• AWS Client VPN
• AWS WAF
• AWS CloudTrail
• Elastic Load Balancing

Results and Benefits

Customer had a single server application which was rearchitected and deployed in AWS Cloud following DevOps guidelines.

• We’ve managed to achieve Zero downtime deployment
• We’ve managed to achieve Autoscale with Application instances
• We’ve managed to separate Reader and Writer DBs for our Application
• We’ve managed to achieve Autoscale on Reader instances
• All resources are now split so we can dedicate resources to only those which require more rather than previously adding to the entire stack.

1. e.g:- RabbitMQ on AWS we still have ” T ” instances running while EC2 is on
   C6g.x.Large

• On Call Support is available with live chat. Previously with Magento Cloud Commerce it was only via Emails which took at least a few hours to respond.

Security of the application was achieved by using WAF and by placing most of the resources in the private subnets. The development team uses VPN to connect to private subnets. So far, no security incidents have occurred.

About the Partner

Genese Business Solution Pvt Ltd is an IT Software Development and Consulting Firm. It is a part of the UK based Genese Solution Ltd Company and serves clients in Europe and Asia. Genese is Nepal’s first and only Advanced Amazon Consulting Partner to provide Amazon Web Services (AWS) Consulting, Training and Certification. Genese also has partnerships with other companies like Microsoft, Google, Nagios, VMware, Neo4j, Barracuda, TrendMicro, Neo4j and PSI.

Genese is an integrated and specialized firm for Cloud Computing Ecosystem and Web/Software Development. Genese believes in Human Capital as its core strength to deliver the highest international level of support to our valued clients. It AIMS to be Nepal’s No.1 Cloud Computing Consulting Firm with the highest number and quality of internationally certified cloud consulting engineers.