API Pagination Guide

This document provides comprehensive guidance on implementing pagination in DeployStack Backend APIs. Pagination is essential for handling large datasets efficiently and providing a good user experience.

Overview

DeployStack uses offset-based pagination with standardized query parameters and response formats. This approach provides:

Consistent API Interface: All paginated endpoints use the same parameter names and response structure
Performance: Reduces memory usage and response times for large datasets
User Experience: Enables smooth navigation through large result sets
Scalability: Handles growing datasets without performance degradation

Standard Pagination Parameters

Query Parameters Schema

All paginated endpoints should accept these standardized query parameters:

const PAGINATION_QUERY_SCHEMA = {
  type: 'object',
  properties: {
    limit: {
      type: 'string',
      pattern: '^\\d+$',
      description: 'Maximum number of items to return (1-100, default: 20)'
    },
    offset: {
      type: 'string',
      pattern: '^\\d+$',
      description: 'Number of items to skip from the beginning (≥0, default: 0)'
    }
  },
  additionalProperties: false
} as const;

Parameter Details

limit (optional, default: 20)
- Type: String (converted to Number in handler)
- Range: 1-100
- Description: Maximum number of items to return
- Validation: Must be a positive integer between 1 and 100
offset (optional, default: 0)
- Type: String (converted to Number in handler)
- Range: ≥ 0
- Description: Number of items to skip from the beginning
- Validation: Must be a non-negative integer

Parameter Validation in Handlers

Query parameters are always strings in HTTP. Convert and validate them in your route handlers:

// Parse and validate pagination parameters
function validatePaginationParams(query: any): { limit: number; offset: number } {
  const limit = query.limit ? parseInt(query.limit, 10) : 20;
  const offset = query.offset ? parseInt(query.offset, 10) : 0;
  
  // Validate limit
  if (isNaN(limit) || limit < 1 || limit > 100) {
    throw new Error('Limit must be between 1 and 100');
  }
  
  // Validate offset
  if (isNaN(offset) || offset < 0) {
    throw new Error('Offset must be non-negative');
  }
  
  return { limit, offset };
}

Standard Response Format

Response Schema

All paginated endpoints should return responses in this format:

const PAGINATED_RESPONSE_SCHEMA = {
  type: 'object',
  properties: {
    success: { type: 'boolean' },
    data: {
      type: 'object',
      properties: {
        // Your actual data array (name varies by endpoint)
        items: { 
          type: 'array',
          items: { /* your item schema */ }
        },
        
        // Pagination metadata
        pagination: {
          type: 'object',
          properties: {
            total: { 
              type: 'number',
              description: 'Total number of items available'
            },
            limit: { 
              type: 'number',
              description: 'Items per page (as requested)'
            },
            offset: { 
              type: 'number',
              description: 'Current offset (as requested)'
            },
            has_more: { 
              type: 'boolean',
              description: 'Whether more items are available'
            }
          },
          required: ['total', 'limit', 'offset', 'has_more']
        }
      },
      required: ['items', 'pagination']
    }
  },
  required: ['success', 'data']
} as const;

Response Example

{
  "success": true,
  "data": {
    "servers": [
      {
        "id": "server-1",
        "name": "Example Server",
        // ... other server fields
      }
      // ... more servers
    ],
    "pagination": {
      "total": 150,
      "limit": 20,
      "offset": 40,
      "has_more": true
    }
  }
}

Pagination Metadata Fields

total: Total number of items available (across all pages)
limit: Number of items per page (echoes the request parameter)
offset: Current starting position (echoes the request parameter)
has_more: Boolean indicating if more items are available after this page

Implementation Pattern

1. Route Schema Definition

import { type FastifyInstance } from 'fastify';

// Query parameters schema (including pagination)
const QUERY_SCHEMA = {
  type: 'object',
  properties: {
    // Your filtering parameters
    category: { 
      type: 'string',
      description: 'Filter by category'
    },
    status: { 
      type: 'string',
      enum: ['active', 'inactive'],
      description: 'Filter by status'
    },
    
    // Standard pagination parameters
    limit: {
      type: 'string',
      pattern: '^\\d+$',
      description: 'Maximum number of items to return (1-100, default: 20)'
    },
    offset: {
      type: 'string',
      pattern: '^\\d+$',
      description: 'Number of items to skip (≥0, default: 0)'
    }
  },
  additionalProperties: false
} as const;

// Response schema
const RESPONSE_SCHEMA = {
  type: 'object',
  properties: {
    success: { type: 'boolean' },
    data: {
      type: 'object',
      properties: {
        items: {
          type: 'array',
          items: {
            // Your item schema here
            type: 'object',
            properties: {
              id: { type: 'string' },
              name: { type: 'string' },
              // ... other properties
            }
          }
        },
        pagination: {
          type: 'object',
          properties: {
            total: { type: 'number' },
            limit: { type: 'number' },
            offset: { type: 'number' },
            has_more: { type: 'boolean' }
          },
          required: ['total', 'limit', 'offset', 'has_more']
        }
      },
      required: ['items', 'pagination']
    }
  },
  required: ['success', 'data']
} as const;

// TypeScript interfaces
interface QueryParams {
  category?: string;
  status?: 'active' | 'inactive';
  limit?: string;
  offset?: string;
}

interface PaginatedResponse {
  success: boolean;
  data: {
    items: Item[];
    pagination: {
      total: number;
      limit: number;
      offset: number;
      has_more: boolean;
    };
  };
}

2. Route Handler Implementation

export default async function listItems(server: FastifyInstance) {
  server.get('/api/items', {
    schema: {
      tags: ['Items'],
      summary: 'List items with pagination',
      description: 'Retrieve items with pagination support. Supports filtering and sorting.',
      
      querystring: QUERY_SCHEMA,
      
      response: {
        200: RESPONSE_SCHEMA,
        400: {
          type: 'object',
          properties: {
            success: { type: 'boolean', default: false },
            error: { type: 'string' }
          }
        }
      }
    }
  }, async (request, reply) => {
    try {
      // Parse and validate query parameters
      const query = request.query as QueryParams;
      const { limit, offset } = validatePaginationParams(query);
      
      // Extract filter parameters
      const filters = {
        category: query.category,
        status: query.status
      };
      
      // Get all items (with filtering applied)
      const allItems = await yourService.getItems(filters);
      
      // Apply pagination
      const total = allItems.length;
      const paginatedItems = allItems.slice(offset, offset + limit);
      
      // Log pagination info
      server.log.info({
        operation: 'list_items',
        totalResults: total,
        returnedResults: paginatedItems.length,
        pagination: { limit, offset }
      }, 'Items list completed');
      
      // Return paginated response
      const response: PaginatedResponse = {
        success: true,
        data: {
          items: paginatedItems,
          pagination: {
            total,
            limit,
            offset,
            has_more: offset + limit < total
          }
        }
      };
      
      const jsonString = JSON.stringify(response);
      return reply.status(200).type('application/json').send(jsonString);
    } catch (error) {
      server.log.error({ error }, 'Failed to list items');
      
      const errorResponse = {
        success: false,
        error: error instanceof Error ? error.message : 'Failed to retrieve items'
      };
      const jsonString = JSON.stringify(errorResponse);
      return reply.status(400).type('application/json').send(jsonString);
    }
  });
}

Database-Level Pagination (Advanced)

For better performance with large datasets, implement pagination at the database level:

Using Drizzle ORM

import { desc, asc, sql, eq } from 'drizzle-orm';

async getItemsPaginated(
  filters: ItemFilters,
  limit: number,
  offset: number
): Promise<{ items: Item[], total: number }> {
  // Build base query with filters
  let query = this.db.select().from(items);
  
  // Apply filters
  if (filters.category) {
    query = query.where(eq(items.category, filters.category));
  }
  
  // Get total count (before pagination)
  const countQuery = this.db.select({ count: sql<number>`count(*)` }).from(items);
  // Apply same filters to count query
  if (filters.category) {
    countQuery = countQuery.where(eq(items.category, filters.category));
  }
  const [{ count: total }] = await countQuery;
  
  // Apply pagination and ordering
  const paginatedItems = await query
    .orderBy(desc(items.created_at))
    .limit(limit)
    .offset(offset);
  
  return {
    items: paginatedItems,
    total
  };
}

Updated Route Handler

// In your route handler
const { items, total } = await yourService.getItemsPaginated(filters, limit, offset);

const response: PaginatedResponse = {
  success: true,
  data: {
    items,
    pagination: {
      total,
      limit,
      offset,
      has_more: offset + limit < total
    }
  }
};

const jsonString = JSON.stringify(response);
return reply.status(200).type('application/json').send(jsonString);

Client-Side Usage Examples

JavaScript/TypeScript

interface PaginationParams {
  limit?: number;
  offset?: number;
}

interface PaginatedResponse<T> {
  success: boolean;
  data: {
    items: T[];
    pagination: {
      total: number;
      limit: number;
      offset: number;
      has_more: boolean;
    };
  };
}

async function fetchItems(params: PaginationParams = {}): Promise<PaginatedResponse<Item>> {
  const url = new URL('/api/items', baseUrl);
  
  if (params.limit) url.searchParams.set('limit', params.limit.toString());
  if (params.offset) url.searchParams.set('offset', params.offset.toString());
  
  const response = await fetch(url.toString(), {
    credentials: 'include',
    headers: { 'Accept': 'application/json' }
  });
  
  return await response.json();
}

// Usage examples
const firstPage = await fetchItems({ limit: 20, offset: 0 });
const secondPage = await fetchItems({ limit: 20, offset: 20 });
const customPage = await fetchItems({ limit: 50, offset: 100 });

Vue.js Composable

import { ref, computed } from 'vue';

export function usePagination<T>(
  fetchFunction: (limit: number, offset: number) => Promise<PaginatedResponse<T>>,
  initialLimit = 20
) {
  const items = ref<T[]>([]);
  const currentPage = ref(1);
  const limit = ref(initialLimit);
  const total = ref(0);
  const loading = ref(false);
  
  const totalPages = computed(() => Math.ceil(total.value / limit.value));
  const hasNextPage = computed(() => currentPage.value < totalPages.value);
  const hasPrevPage = computed(() => currentPage.value > 1);
  
  const offset = computed(() => (currentPage.value - 1) * limit.value);
  
  async function loadPage(page: number) {
    if (page < 1 || page > totalPages.value) return;
    
    loading.value = true;
    try {
      const response = await fetchFunction(limit.value, (page - 1) * limit.value);
      items.value = response.data.items;
      total.value = response.data.pagination.total;
      currentPage.value = page;
    } finally {
      loading.value = false;
    }
  }
  
  async function nextPage() {
    if (hasNextPage.value) {
      await loadPage(currentPage.value + 1);
    }
  }
  
  async function prevPage() {
    if (hasPrevPage.value) {
      await loadPage(currentPage.value - 1);
    }
  }
  
  return {
    items,
    currentPage,
    limit,
    total,
    totalPages,
    loading,
    hasNextPage,
    hasPrevPage,
    loadPage,
    nextPage,
    prevPage
  };
}

Best Practices

1. Consistent Parameter Validation

Always use the same validation rules across all endpoints:

// Create a reusable validation function
export function validatePaginationParams(query: any): { limit: number; offset: number } {
  const limit = query.limit ? parseInt(query.limit, 10) : 20;
  const offset = query.offset ? parseInt(query.offset, 10) : 0;
  
  if (isNaN(limit) || limit < 1 || limit > 100) {
    throw new Error('Limit must be between 1 and 100');
  }
  
  if (isNaN(offset) || offset < 0) {
    throw new Error('Offset must be non-negative');
  }
  
  return { limit, offset };
}

// Reusable schema constant
export const PAGINATION_QUERY_SCHEMA = {
  type: 'object',
  properties: {
    limit: {
      type: 'string',
      pattern: '^\\d+$',
      description: 'Maximum number of items to return (1-100, default: 20)'
    },
    offset: {
      type: 'string',
      pattern: '^\\d+$',
      description: 'Number of items to skip (≥0, default: 0)'
    }
  },
  additionalProperties: false
} as const;

// Use in your endpoint schemas
const QUERY_SCHEMA = {
  type: 'object',
  properties: {
    // Your specific filters
    category: { type: 'string' },
    status: { type: 'string', enum: ['active', 'inactive'] },
    
    // Include pagination
    ...PAGINATION_QUERY_SCHEMA.properties
  },
  additionalProperties: false
} as const;

2. Proper Error Handling

try {
  const { limit, offset } = validatePaginationParams(request.query);
  // ... rest of handler
} catch (error) {
  const errorResponse = {
    success: false,
    error: error instanceof Error ? error.message : 'Invalid query parameters'
  };
  const jsonString = JSON.stringify(errorResponse);
  return reply.status(400).type('application/json').send(jsonString);
}

3. Performance Considerations

Database Pagination: Use LIMIT and OFFSET at the database level for large datasets
Indexing: Ensure proper database indexes on columns used for sorting
Caching: Consider caching total counts for frequently accessed endpoints
Reasonable Limits: Enforce maximum page sizes (e.g., 100 items)

4. OpenAPI Documentation

Include clear pagination documentation in your API specs:

schema: {
  tags: ['Items'],
  summary: 'List items with pagination',
  description: `
    Retrieve items with pagination support. 
    
    **Pagination Parameters:**
    - \`limit\`: Items per page (1-100, default: 20)
    - \`offset\`: Items to skip (≥0, default: 0)
    
    **Response includes:**
    - \`data.items\`: Array of items for current page
    - \`data.pagination.total\`: Total items available
    - \`data.pagination.has_more\`: Whether more pages exist
  `,
  // ... rest of schema
}

Common Pitfalls and Solutions

1. Inconsistent Response Formats

❌ Wrong: Different endpoints use different response structures

// Endpoint A
{ data: items, total: 100, page: 1 }

// Endpoint B  
{ results: items, count: 100, offset: 20 }

✅ Correct: Use standardized response format

// All endpoints
{
  success: true,
  data: {
    items: [...],
    pagination: { total, limit, offset, has_more }
  }
}

2. Missing Validation

❌ Wrong: No parameter validation

const limit = parseInt(request.query.limit) || 20;
const offset = parseInt(request.query.offset) || 0;

✅ Correct: Proper validation function

const { limit, offset } = validatePaginationParams(request.query);

3. Performance Issues

❌ Wrong: Loading all data then slicing

const allItems = await db.select().from(items); // Loads everything!
const paginated = allItems.slice(offset, offset + limit);

✅ Correct: Database-level pagination

const items = await db.select().from(items)
  .limit(limit)
  .offset(offset);

4. Incorrect Total Count

❌ Wrong: Using paginated results length

const items = await getItemsPaginated(limit, offset);
const total = items.length; // Wrong! This is just current page

✅ Correct: Separate count query

const [items, total] = await Promise.all([
  getItemsPaginated(limit, offset),
  getItemsCount(filters)
]);

Real-World Examples

Example 1: MCP Servers List (Current Implementation)

// File: services/backend/src/routes/mcp/servers/list.ts
export default async function listServers(server: FastifyInstance) {
  server.get('/mcp/servers', {
    schema: {
      tags: ['MCP Servers'],
      summary: 'List MCP servers',
      description: 'Retrieve MCP servers with pagination support...',
      querystring: QUERY_SCHEMA,
      response: {
        200: LIST_SERVERS_RESPONSE_SCHEMA
      }
    }
  }, async (request, reply) => {
    const { limit, offset } = validatePaginationParams(request.query);
    const filters = extractFilters(request.query);
    
    const allServers = await catalogService.getServersForUser(
      userId, userRole, teamIds, filters
    );
    
    const total = allServers.length;
    const paginatedServers = allServers.slice(offset, offset + limit);
    
    const response = {
      success: true,
      data: {
        servers: paginatedServers,
        pagination: {
          total,
          limit,
          offset,
          has_more: offset + limit < total
        }
      }
    };
    
    const jsonString = JSON.stringify(response);
    return reply.status(200).type('application/json').send(jsonString);
  });
}

Example 2: Search Endpoint (Reference Implementation)

The search endpoint (/mcp/servers/search) demonstrates the complete pagination pattern and can serve as a reference for implementing pagination in other endpoints.

Testing Pagination

Unit Tests

describe('Pagination', () => {
  test('should return first page with default limit', async () => {
    const response = await request(app)
      .get('/api/items')
      .expect(200);
    
    expect(response.body.data.pagination).toEqual({
      total: expect.any(Number),
      limit: 20,
      offset: 0,
      has_more: expect.any(Boolean)
    });
  });
  
  test('should handle custom pagination parameters', async () => {
    const response = await request(app)
      .get('/api/items?limit=10&offset=20')
      .expect(200);
    
    expect(response.body.data.pagination.limit).toBe(10);
    expect(response.body.data.pagination.offset).toBe(20);
  });
  
  test('should validate pagination parameters', async () => {
    await request(app)
      .get('/api/items?limit=invalid')
      .expect(400);
      
    await request(app)
      .get('/api/items?limit=101') // Over maximum
      .expect(400);
  });
});

Integration Tests

test('should paginate through all results', async () => {
  const limit = 5;
  let offset = 0;
  let allItems = [];
  let hasMore = true;
  
  while (hasMore) {
    const response = await request(app)
      .get(`/api/items?limit=${limit}&offset=${offset}`)
      .expect(200);
    
    const { items, pagination } = response.body.data;
    allItems.push(...items);
    
    hasMore = pagination.has_more;
    offset += limit;
  }
  
  // Verify we got all items
  expect(allItems.length).toBe(totalExpectedItems);
});

Basics

Advanced

API

Database

Auth

Satellite

Overview

Query Parameters Schema

Parameter Details

Parameter Validation in Handlers

Standard Response Format

Response Schema

Response Example

Implementation Pattern

1. Route Schema Definition

2. Route Handler Implementation

Using Drizzle ORM

Updated Route Handler

Client-Side Usage Examples

JavaScript/TypeScript

Vue.js Composable

Best Practices

1. Consistent Parameter Validation

2. Proper Error Handling

3. Performance Considerations

4. OpenAPI Documentation

Common Pitfalls and Solutions

1. Inconsistent Response Formats

2. Missing Validation

3. Performance Issues

4. Incorrect Total Count

Real-World Examples

Example 1: MCP Servers List (Current Implementation)

Example 2: Search Endpoint (Reference Implementation)

Unit Tests

Integration Tests

Basics

Advanced

API

Database

Auth

Satellite

​Overview

​Standard Pagination Parameters

​Query Parameters Schema

​Parameter Details

​Parameter Validation in Handlers

​Standard Response Format

​Response Schema

​Response Example

​Pagination Metadata Fields

​Implementation Pattern

​1. Route Schema Definition

​2. Route Handler Implementation

​Database-Level Pagination (Advanced)

​Using Drizzle ORM

​Updated Route Handler

​Client-Side Usage Examples

​JavaScript/TypeScript

​Vue.js Composable

​Best Practices

​1. Consistent Parameter Validation

​2. Proper Error Handling

​3. Performance Considerations

​4. OpenAPI Documentation

​Common Pitfalls and Solutions

​1. Inconsistent Response Formats

​2. Missing Validation

​3. Performance Issues

​4. Incorrect Total Count

​Real-World Examples

​Example 1: MCP Servers List (Current Implementation)

​Example 2: Search Endpoint (Reference Implementation)

​Testing Pagination

​Unit Tests

​Integration Tests

Overview

Standard Pagination Parameters

Query Parameters Schema

Parameter Details

Parameter Validation in Handlers

Standard Response Format

Response Schema

Response Example

Pagination Metadata Fields

Implementation Pattern

1. Route Schema Definition

2. Route Handler Implementation

Database-Level Pagination (Advanced)

Using Drizzle ORM

Updated Route Handler

Client-Side Usage Examples

JavaScript/TypeScript

Vue.js Composable

Best Practices

1. Consistent Parameter Validation

2. Proper Error Handling

3. Performance Considerations

4. OpenAPI Documentation

Common Pitfalls and Solutions

1. Inconsistent Response Formats

2. Missing Validation

3. Performance Issues

4. Incorrect Total Count

Real-World Examples

Example 1: MCP Servers List (Current Implementation)

Example 2: Search Endpoint (Reference Implementation)

Testing Pagination

Unit Tests

Integration Tests