Like the title suggests, My team and I managed to make an E-Commerce website for a Paris-based startup.
This blog contains some of the architectural and engineering decisions that went into its making, for which we address the different problems we had to solve and also the Trade-offs that we had to consider.
The AWS Architecture looked like this:
But after a while we migrated to another IaaS platform and ended up running most services on one single high resource compute machine.
Why We Chose an API-First Architecture
Server-Side Rendering has to do with using some scripting language such as PHP or EJS in order to generate an HTML file that includes data from the server and Endpoints. This makes the Front-End deeply coupled with the Backend since it is quite literally generated by the Backend.
Client-Side Rendering on the other hand separates the Front-End and Back-end, in a way that we could choose any Front-End Framework/Platform and then make it communicate with the backend using an agreed-upon Protocol Standard (such as REST API) on specific API Endpoints that the backend offers.
And for that specific reason we went with Client-Side Rendering and chose React for the Front-End, since changing the framework itself, the programming language, or even where the Front-End code itself exists (on a CDN, a Mobile phone etc) won't really require that big of a change on the Backend Side.
Authentication Architecture: JWTs, Cookies, and Sessions
some of you might have heard of Cross-Site Scripting (XSS), it has to do with JavaScript executing malicious code (usually introduced by a threat actor) in order to exfiltrate any important data that is accessible using JavaScript (such as from LocalStorage).
The problem with this attack is that if you happen to store information that can be used to impersonate you, such as API keys or Authentication Tokens, then your client would be at risk.
So as a mitigation http-only cookies are used in order for them to be auto-included with every request to our Backend without exposing access to them via Javascript.
Sessions must be configured on the Backend but alongside that we also used JWT, why ? because they serve as a cryptographic container for information related to the Authenticated user, such as their role for example (if they're an 'editor', 'client', 'superadmin' etc...)
We primarily use Session Rotation in the case of a breach which will force all clients to re-authenticate making the stolen session ids invalid for any attackers.
But in reality both JWT and Sessions can be rotated, in our case it would be redundant and introduce unnecessary complexity.
RBAC was used to secure Endpoints, given our use case it was the right one to use, but if the website had a more granular control of rights, then we would have used ABAC (Attribute-Based Access Control)
Modeling Inventory Across Multiple Stocks
Our client has multiple stocks in which he stores his merchandise, at first we weren't given that information so we had already made the app use only 1 Stock during checkout.
However due to the Database model being made in a way that allows flexible re-modeling, we were easily able to make it work in a way that a Single product can be outsourced from one or multiple stocks using different quantities, all in the same checkout without making any data discrepancies.
Modeling the Order Lifecycle as a State Machine
Finite State Machines are typically used to have strict rules on what a specific entity can execute while it's in a specific state, but we've used them conceptually to conceptualize the general checkout process
Securing Database Access Through SSH Tunnels
The approach is a lot of existing E-commerce websites built on top of CMS platforms is allowing access to the database via a website interface known as "PhpMyAdmin", the problem is that it introduces a new attack surface.
Because attackers could attempt to brute force the credentials to the database or in the worst case possible they might find a vulnerability in the admin portal software and use it to gain unauthorized access to the database.
The solution we used was SSH Tunneling, we deny any traffic that goes to the database except our Linux server, it is the only source of traffic that is allowed. So if we want to access the database from our Local development machines, we have to connect via SSH creating a tunnel, then making the Linux machine forward the traffic to the database, serving the role of a bridge.
Hardening a Production Linux Server
We took different measures to make sure that our Machine is generally secure against threat actors, in other words we "hardened" our Server.
We blocked all the ports except 80 and 443. Port 80 (HTTP) itself responds with redirection response to the HTTPS URL (running on 443) and of course Port 22 for SSH remote access.
Why We Used an ORM and Database Migrations
in our case, using an ORM was primarily useful for 2 distinct reasons:
- It provided a modifiable schema (that mapped Database objects to JS objects)
- It provided a way to interact with the database & perform migrations
Those two features are life saviors especially in a Live production system like an E-commerce website, Systems grow in size and complexity and without migrations you might make a change that is irreversible or introduce data consistency problems and be unable to tell what went wrong, with migrations you get provided a history of what you modified/added.
Plus you also get access to perform queries programmatically.
The Cost of Neglecting Regression Testing
Testing was one of the things that we initially neglected but we were proved that without it we were prone to introducing problems/bugs by fixing other bugs, something that Regression Testing (aka running the same tests after each modification) would have easily solved.
Designing a Hybrid Image Storage Strategy
When we talk about Object Storage we mean any sort of files that are dynamically added/removed, Front-end Build files are not considered Objects that we have to store in an S3 Bucket etc. because they don't change as often plus they're very light in size and usually fit better in a CDN.
Images of Products however are good candidates for Object Storage because they can be removed, modified, or added. Hence why we came up with a solution to not rely fully on an external Object Storage service like AWS S3 or Cloudinary and not fully host them ourselves.
We made a hybrid approach of Hashing the file content once uploaded, this type of hashing is usually referred to as "Content-Addressable", because it's the data content of something that makes you derive its hash, then making a copy both locally and remotely, locally the file gets overwritten but storage isn't wasted because we're only replacing the existing file with the same name.
The same happens for the Remote service using our own Wrapper abstraction.
Designing a Fallback Strategy for External Storage
As for the images that we talked about, they get fetched at first from the external storage endpoints but if nothing comes back then we alternatively use an exposed public-facing folder to serve the same exact images, that way we don't get affected if some problem with the external storage platform occurs.
Checkout Consistency, Transactions, and Race Conditions
One of the most important challenges in this project was related to Data Consistency, the checkout process specifically had components that can corrupt the state of our data if not handled carefully.
For example, a Checkout process involved taking care of Database INSERT queries that - if a problem occurs - we will end up with partial writes which make data invalid and inconsistent, the solution was to use Database Transactions which allowed us to perform multiple queries and if they all go through, we will "commit" the changes and end up with consistent data, otherwise a "rollback" will occur and nothing will be written.
Another thing we used is Triggers, they automatically reduce any quantity from any product on any stock upon the confirmation of payment.
Race conditions were the trickiest problems to spot but luckily we stumbled across only one instance of it and it was involving emails, asynchronous programming does introduce problems like these so one should be very careful about the flow of execution.
Another decision worth noting had to do with Payment Gateways and Delivery Platforms, see the problem with both of these is that between one company and another the data received changes.
So instead of making database tables for different services, we only generalized the core columns and put all the service-specific ones in a JSON column.
Diagnosing Production Outages Caused by Memory Pressure
Another instance in which we learned something entirely new was when we figured out at the early stages of the project why the Linux machine was sometimes becoming fully unresponsive, even to SSH.
Turns out the problem was that it had no Swap, Swap is a last resort for when RAM is full, Swap represents a chunk of secondary memory (aka SSD or HDD) and it gets used when the server receives a big spike of data that has to be stored in memory.
And without it the machine freezes because it ran out of memory.
Why We Put Nginx in Front of Express
We needed to serve static assets and terminate HTTPS without exposing the Express application directly. We therefore placed Nginx at the entry point, allowing it to handle incoming connections and forward application requests to Express
Separating Production and Testing Environments
We used a Production and Testing environments, this way we had a neat way to test out any API endpoints (using test sandbox API keys) and make sure that the app functions the way we want it to be.