🚀 Comprehensive Tutorial: Integrating Redis Caching with a Node.js Project Using Docker Compose

Outline

🛠️ Prerequisites
📂 Project Structure
📝 Setting Up Environment Variables
🐳 Configuring Docker Compose
📄 docker-compose.yml
📄 docker-compose.dev.yml
📄 docker-compose.prod.yml
📄 docker-compose.ref.yml
🐋 Creating the Dockerfile
📦 Installing Dependencies
🔗 Connecting Node.js to MongoDB and Redis
🗃️ Implementing Caching Logic with Redis
🚀 Running the Application
🧪 Testing the Setup
🧹 Cleaning Up
📚 Conclusion

🛠️ Prerequisites

Before starting this tutorial, ensure you have the following installed on your system:

Docker: Containerization platform.
Docker Compose: Tool for defining and running multi-container Docker applications.
Node.js: JavaScript runtime for building server-side applications.
npm: Node package manager (comes with Node.js).

📂 Project Structure

Organize your project directory as follows:

nodejs-redis-app/
├── docker-compose.yml
├── docker-compose.dev.yml
├── docker-compose.prod.yml
├── docker-compose.ref.yml
├── Dockerfile
├── Environment_Variables.md
├── .env
├── package.json
├── package-lock.json
├── src/
│   └── index.js
└── mongo-data/

📄 File Descriptions

File/Directory	Description
`docker-compose.yml`	Base Docker Compose configuration defining services.
`docker-compose.dev.yml`	Development-specific Docker Compose overrides.
`docker-compose.prod.yml`	Production-specific Docker Compose overrides.
`docker-compose.ref.yml`	Reference Docker Compose file for shared configurations.
`Dockerfile`	Defines how to build the Node.js Docker image.
`Environment_Variables.md`	Documentation of environment variables used in the project.
`.env`	Environment variables for Docker Compose and Node.js app.
`package.json`	Lists project dependencies and scripts.
`package-lock.json`	Locks project dependencies to specific versions.
`src/index.js`	Main Node.js application file.
`mongo-data/`	Persistent storage for MongoDB data.

📝 Setting Up Environment Variables

Managing environment variables is crucial for configuring your application across different environments (development, production). We'll use a .env file to store these variables and document them in Environment_Variables.md.

1. Create a `.env` File

Create a .env file in the root directory with the following content:

# .env

# MongoDB Configuration
MONGO_INITDB_ROOT_USERNAME=root
MONGO_INITDB_ROOT_PASSWORD=example
MONGO_INITDB_DATABASE=mydatabase
MONGO_HOST=mongo-dev
MONGO_PORT=27017

# Redis Configuration
REDIS_HOST=redis-dev
REDIS_PORT=6379

# Node.js Application
PORT=4002
NODE_ENV=development

2. Document Environment Variables

Create an Environment_Variables.md file to document each environment variable:

# Environment Variables

## MongoDB Configuration

- **MONGO_INITDB_ROOT_USERNAME**: The root username for MongoDB.
- **MONGO_INITDB_ROOT_PASSWORD**: The root password for MongoDB.
- **MONGO_INITDB_DATABASE**: The name of the default database.
- **MONGO_HOST**: The hostname for the MongoDB service.
- **MONGO_PORT**: The port on which MongoDB listens.

## Redis Configuration

- **REDIS_HOST**: The hostname for the Redis service.
- **REDIS_PORT**: The port on which Redis listens.

## Node.js Application

- **PORT**: The port on which the Node.js application runs.
- **NODE_ENV**: The environment in which the application is running (`development` or `production`).

🐳 Configuring Docker Compose

We'll use Docker Compose to orchestrate our multi-container application, which includes the Node.js app, MongoDB, and Redis.

📄 docker-compose.yml

This is the base Docker Compose file that defines the core services: Node.js app, MongoDB, and Redis.

# docker-compose.yml

version: "3.9"

services:
  node-app:
    container_name: node-app-dev
    build:
      context: .
      dockerfile: Dockerfile
      target: development
    ports:
      - "4002:4002"
    env_file:
      - .env
    volumes:
      - ./src:/app/src:ro
      - /app/node_modules
    depends_on:
      - mongo-dev
      - redis-dev

  mongo-dev:
    container_name: mongo-dev
    image: mongo:latest
    restart: always
    ports:
      - "27017:27017"
    environment:
      MONGO_INITDB_ROOT_USERNAME: ${MONGO_INITDB_ROOT_USERNAME}
      MONGO_INITDB_ROOT_PASSWORD: ${MONGO_INITDB_ROOT_PASSWORD}
      MONGO_INITDB_DATABASE: ${MONGO_INITDB_DATABASE}
    volumes:
      - mongo-db:/data/db

  redis-dev:
    container_name: redis-dev
    image: redis:latest
    restart: always
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data

volumes:
  mongo-db:
  redis-data:

📄 docker-compose.dev.yml

Development-specific overrides, such as mounting source code and using development environment variables.

# docker-compose.dev.yml

version: "3.9"

services:
  node-app:
    build:
      args:
        NODE_ENV: development
    environment:
      - NODE_ENV=development
    command: npm run start-dev

📄 docker-compose.prod.yml

Production-specific overrides, focusing on optimizing the image for production.

# docker-compose.prod.yml

version: "3.9"

services:
  node-app:
    build:
      args:
        NODE_ENV: production
    environment:
      - NODE_ENV=production
    ports:
      - "4002:4002"
    command: npm run start
    volumes:
      - /app/node_modules

📄 docker-compose.ref.yml

Reference file for shared configurations across different Compose files.

# docker-compose.ref.yml

version: "3.9"

services:
  node-app:
    environment:
      - MONGO_URI=mongodb://${MONGO_INITDB_ROOT_USERNAME}:${MONGO_INITDB_ROOT_PASSWORD}@${MONGO_HOST}:${MONGO_PORT}/${MONGO_INITDB_DATABASE}
      - REDIS_HOST=${REDIS_HOST}
      - REDIS_PORT=${REDIS_PORT}

🔄 Combining Docker Compose Files

To manage different environments, use the -f flag to specify multiple Docker Compose files. For example, to start in development:

docker-compose -f docker-compose.yml -f docker-compose.dev.yml up --build

For production:

docker-compose -f docker-compose.yml -f docker-compose.prod.yml up --build -d

🐋 Creating the Dockerfile

The Dockerfile defines how to build the Docker image for the Node.js application, supporting both development and production stages through multi-stage builds.

# Dockerfile

# Stage 1: Base Image
FROM node:16 AS base
WORKDIR /app
COPY package.json package-lock.json ./

# Stage 2: Development Image
FROM base AS development
ENV NODE_ENV=development
RUN npm install
COPY . .
CMD ["npm", "run", "start-dev"]

# Stage 3: Production Image
FROM base AS production
ENV NODE_ENV=production
RUN npm install --only=production
COPY . .
CMD ["npm", "run", "start"]

📝 Explanation

Stage 1: Base Image
Uses Node.js 16.
Sets the working directory to /app.
Copies package.json and package-lock.json to leverage Docker caching.
Stage 2: Development Image
Inherits from the base image.
Sets NODE_ENV to development.
Installs all dependencies.
Copies the entire project directory.
Runs the development script (start-dev).
Stage 3: Production Image
Inherits from the base image.
Sets NODE_ENV to production.
Installs only production dependencies.
Copies the entire project directory.
Runs the production script (start).

📦 Installing Dependencies

Ensure your package.json includes the necessary dependencies for Express, Mongoose, and Redis.

Example `package.json`

{
  "name": "nodejs-redis-app",
  "version": "1.0.0",
  "description": "A Node.js project with MongoDB and Redis caching",
  "main": "src/index.js",
  "scripts": {
    "start": "node src/index.js",
    "start-dev": "nodemon src/index.js"
  },
  "dependencies": {
    "express": "^4.18.2",
    "mongoose": "^7.3.1",
    "redis": "^4.6.7"
  },
  "devDependencies": {
    "nodemon": "^3.0.1"
  },
  "author": "Your Name",
  "license": "ISC"
}

Installing Packages

Run the following command to install dependencies:

npm install

For development dependencies like nodemon:

npm install --save-dev nodemon

🔗 Connecting Node.js to MongoDB and Redis

Update your src/index.js to connect to both MongoDB and Redis, and implement caching logic.

Updated `src/index.js`

const express = require("express");
const mongoose = require("mongoose");
const redis = require("redis");
const path = require("path");
const util = require("util");

// Load environment variables
require("dotenv").config();

// Initialize Express app
const app = express();
app.use(express.json()); // To parse JSON bodies
app.use(express.urlencoded({ extended: true })); // To parse URL-encoded bodies

// MongoDB connection details from environment variables
const BD_USER = process.env.MONGO_INITDB_ROOT_USERNAME;
const BD_PASSWORD = process.env.MONGO_INITDB_ROOT_PASSWORD;
const BD_PORT = process.env.MONGO_PORT || "27017";
const BD_NAME = process.env.MONGO_INITDB_DATABASE;
const BD_HOST = process.env.MONGO_HOST || "mongo-dev";
const URI = `mongodb://${BD_USER}:${BD_PASSWORD}@${BD_HOST}:${BD_PORT}/${BD_NAME}`;

// Connect to MongoDB
mongoose
  .connect(URI)
  .then(() => {
    console.log("Connected to MongoDB");
  })
  .catch((err) => {
    console.error("MongoDB connection error:", err);
  });

// Define the Todo schema
const todoSchema = new mongoose.Schema({
  taskName: { type: String, required: true },
  description: { type: String, required: true },
});

const Todo = mongoose.model("Todo", todoSchema);

// Redis client setup
const redisHost = process.env.REDIS_HOST || "redis-dev";
const redisPort = process.env.REDIS_PORT || 6379;

const client = redis.createClient({
  socket: {
    host: redisHost,
    port: redisPort,
  },
});

// Promisify Redis methods for async/await
client.get = util.promisify(client.get);
client.set = util.promisify(client.set);

client.on("error", (err) => {
  console.error("Redis connection error:", err);
});

client
  .connect()
  .then(() => {
    console.log("Connected to Redis");
  })
  .catch((err) => {
    console.error("Redis connection error:", err);
  });

// Serve the form directly from index.js
app.get("/", (req, res) => {
  res.send(`
    <html>
      <head>
        <title>Create Todo</title>
        <style>
          body { font-family: Arial, sans-serif; background-color: #f4f4f4; padding: 20px; }
          h1 { color: #333; }
          form { background: #fff; padding: 20px; border-radius: 5px; max-width: 400px; margin: auto; }
          label { display: block; margin-bottom: 10px; font-weight: bold; }
          input, textarea { width: 100%; padding: 10px; margin-bottom: 20px; border: 1px solid #ccc; border-radius: 4px; }
          button { background-color: #5cb85c; color: white; border: none; padding: 10px 15px; border-radius: 5px; cursor: pointer; font-size: 16px; }
          button:hover { background-color: #4cae4c; }
        </style>
      </head>
      <body>
        <h1>Create Todo Item</h1>
        <form id="todoForm">
          <label for="taskName">Task Name:</label>
          <input type="text" id="taskName" name="taskName" required>
          <br>
          <label for="description">Description:</label>
          <textarea id="description" name="description" rows="4" required></textarea>
          <br>
          <button type="submit">Create Todo</button>
        </form>
        <script>
          document.getElementById('todoForm').addEventListener('submit', async function(event) {
            event.preventDefault(); // Prevent default form submission
            const taskName = document.getElementById('taskName').value;
            const description = document.getElementById('description').value;

            try {
              const response = await fetch('/todoCreate', {
                method: 'POST',
                headers: {
                  'Content-Type': 'application/json'
                },
                body: JSON.stringify({ taskName, description })
              });

              const data = await response.json();
              if (response.ok) {
                alert('Todo created successfully!');
                document.getElementById('todoForm').reset();
              } else {
                alert('Error: ' + data.message);
              }
            } catch (error) {
              console.error('Error:', error);
              alert('Error connecting to server');
            }
          });
        </script>
      </body>
    </html>
  `);
});

// API endpoint to create a new Todo item
app.post("/todoCreate", async (req, res) => {
  const { taskName, description } = req.body;

  // Check Redis cache first
  try {
    const cacheKey = `todo:${taskName}`;
    const cachedTodo = await client.get(cacheKey);

    if (cachedTodo) {
      console.log("Serving from cache");
      return res.status(200).json(JSON.parse(cachedTodo));
    }
  } catch (err) {
    console.error("Redis GET error:", err);
  }

  // If not in cache, fetch from MongoDB
  const newTodo = new Todo({
    taskName,
    description,
  });

  try {
    const savedTodo = await newTodo.save();

    // Store in Redis cache
    try {
      const cacheKey = `todo:${savedTodo.taskName}`;
      await client.set(cacheKey, JSON.stringify(savedTodo), {
        EX: 3600, // Cache expires in 1 hour
      });
    } catch (err) {
      console.error("Redis SET error:", err);
    }

    res.status(201).json(savedTodo);
  } catch (err) {
    console.error("Error adding todo:", err);
    res.status(500).json({ message: "Error adding todo", error: err });
  }
});

// Run server
const PORT = process.env.PORT || 4002;
app.listen(PORT, () => {
  console.log(`Listening on port ${PORT}`);
});

📝 Explanation

Connecting to MongoDB:
Uses environment variables for configuration.
Establishes a connection using Mongoose.
Connecting to Redis:
Uses the redis package to create a Redis client.
Promisifies get and set methods for async/await usage.
Connects to Redis using environment variables.
Serving the Form:
The root route (/) serves an HTML form directly from index.js.
The form collects taskName and description.
Handling Form Submission:
On form submission, sends a POST request to /todoCreate with form data.
The server first checks if the todo item is in Redis cache.
- If found, serves from cache.
- If not, saves to MongoDB and caches the result in Redis.
Caching Logic:
Cache Key: Uses todo:<taskName> as the Redis key.
Cache Expiration: Sets the cache to expire in 1 hour (EX: 3600).

🗃️ Implementing Caching Logic with Redis

Implementing caching with Redis can significantly improve the performance of your application by reducing the number of database queries. Here's how the caching logic works in the provided index.js:

Check Cache Before Database:
When a user submits a new Todo, the server first checks if a Todo with the same taskName exists in Redis.
If it exists, it serves the Todo from Redis, bypassing MongoDB.
Cache Miss Handling:
If the Todo is not found in Redis, the server proceeds to save it in MongoDB.
After saving, it caches the new Todo in Redis for future requests.
Cache Invalidation:
The cached data has an expiration time (1 hour in this example). After expiration, the data will be removed from the cache, ensuring that stale data isn't served.

📈 Benefits

Reduced Latency: Serving data from Redis is faster than querying MongoDB.
Decreased Database Load: Fewer queries to MongoDB reduce the load on the database server.
Scalability: Efficient caching allows your application to handle more requests with the same resources.

🚀 Running the Application

1. Build and Start Docker Containers

Use Docker Compose to build and run your application along with MongoDB and Redis services.

Development Environment

sudo docker-compose -f docker-compose.yml -f docker-compose.dev.yml up --build

Flags:
-f docker-compose.yml -f docker-compose.dev.yml: Combines the base and development-specific configurations.
--build: Builds images before starting containers.

Production Environment

sudo docker-compose -f docker-compose.yml -f docker-compose.prod.yml up --build -d

Flags:
-d: Runs containers in detached mode.

2. Access the Application

Open your browser and navigate to http://localhost:4002/ to access the Todo creation form.

3. Create a Todo Item

Fill Out the Form:
Enter a Task Name and Description.
Submit:
Click the "Create Todo" button.
Confirmation:
An alert will notify you of successful creation.
The data is stored in MongoDB and cached in Redis.

🧪 Testing the Setup

1. Verify MongoDB Connection

Check the logs to ensure that the application has connected to MongoDB successfully.

sudo docker-compose -f docker-compose.yml -f docker-compose.dev.yml logs mongo-dev

You should see messages indicating successful authentication and database creation.

2. Verify Redis Connection

Similarly, check Redis logs to confirm a successful connection.

sudo docker-compose -f docker-compose.yml -f docker-compose.dev.yml logs redis-dev

3. Check Data in MongoDB

You can connect to MongoDB using the mongosh shell to verify that Todo items are being stored.

sudo docker exec -it mongo-dev mongosh -u root -p example --authenticationDatabase admin

Once connected, switch to your database and query the todos collection:

use mydatabase;
db.todos.find().pretty();

4. Check Data in Redis

To verify that Todo items are cached in Redis, access the Redis CLI:

sudo docker exec -it redis-dev redis-cli

Once in the Redis CLI, use the GET command with the appropriate key:

GET todo:<TaskName>

Replace <TaskName> with the actual task name you used when creating a Todo.

🧹 Cleaning Up

1. Stop and Remove Containers

To stop the running containers and remove them along with their networks:

sudo docker-compose -f docker-compose.yml -f docker-compose.dev.yml down

2. Remove Images

List all Docker images:

sudo docker images

Find the image names or IDs associated with your project and remove them:

sudo docker rmi <image-name-or-id>

Replace <image-name-or-id> with the actual image name or ID.

3. Remove Volumes (If Needed)

If you want to remove persistent data:

sudo docker volume ls
sudo docker volume rm <volume-name>

Replace <volume-name> with the actual volume name, e.g., mongo-db or redis-data.

📚 Conclusion

In this tutorial, you've learned how to integrate Redis as a caching layer in a Node.js project using Docker Compose. By leveraging Redis alongside MongoDB, you can significantly enhance your application's performance and scalability. Here's a quick recap of what you've accomplished:

Project Setup: Organized a Node.js project with Docker Compose configurations for development and production environments.
Environment Management: Utilized environment variables to manage configurations across different environments.
Docker Configuration: Created a multi-stage Dockerfile to optimize image builds for development and production.
Connecting Services: Established connections between Node.js, MongoDB, and Redis using Mongoose and the Redis client.
Implementing Caching: Integrated Redis caching logic to serve data faster and reduce database load.
Testing and Validation: Verified the setup by creating Todo items and ensuring data is stored in MongoDB and cached in Redis.

🚀 Comprehensive Tutorial: Integrating Redis Caching with a Node.js Project Using Docker Compose

🛠️ Prerequisites

📂 Project Structure

📄 File Descriptions

📝 Setting Up Environment Variables

1. Create a .env File

2. Document Environment Variables

🐳 Configuring Docker Compose

📄 docker-compose.yml

📄 docker-compose.dev.yml

📄 docker-compose.prod.yml

📄 docker-compose.ref.yml

🔄 Combining Docker Compose Files

🐋 Creating the Dockerfile

📝 Explanation

📦 Installing Dependencies

Example package.json

Installing Packages

🔗 Connecting Node.js to MongoDB and Redis

Updated src/index.js

📝 Explanation

🗃️ Implementing Caching Logic with Redis

📈 Benefits

🚀 Running the Application

1. Build and Start Docker Containers

Development Environment

Production Environment

2. Access the Application

3. Create a Todo Item

🧪 Testing the Setup

1. Verify MongoDB Connection

2. Verify Redis Connection

3. Check Data in MongoDB

4. Check Data in Redis

🧹 Cleaning Up

1. Stop and Remove Containers

2. Remove Images

3. Remove Volumes (If Needed)

📚 Conclusion

🔗 Additional Resources

1. Create a `.env` File

Example `package.json`

Updated `src/index.js`