API Pagination Guide

This document provides comprehensive guidance on implementing pagination in DeployStack Backend APIs. Pagination is essential for handling large datasets efficiently and providing a good user experience.

Overview

DeployStack uses offset-based pagination with standardized query parameters and response formats. This approach provides:

Consistent API Interface: All paginated endpoints use the same parameter names and response structure
Performance: Reduces memory usage and response times for large datasets
User Experience: Enables smooth navigation through large result sets
Scalability: Handles growing datasets without performance degradation

Standard Pagination Parameters

Query Parameters Schema

All paginated endpoints should accept these standardized query parameters:

const PAGINATION_QUERY_SCHEMA = {
  type: 'object',
  properties: {
    limit: {
      type: 'string',
      pattern: '^\\d+$',
      description: 'Maximum number of items to return (1-100, default: 20)'
    },
    offset: {
      type: 'string',
      pattern: '^\\d+$',
      description: 'Number of items to skip from the beginning (≥0, default: 0)'
    }
  },
  additionalProperties: false
} as const;

Parameter Details

limit (optional, default: 20)
- Type: String (converted to Number in handler)
- Range: 1-100
- Description: Maximum number of items to return
- Validation: Must be a positive integer between 1 and 100
offset (optional, default: 0)
- Type: String (converted to Number in handler)
- Range: ≥ 0
- Description: Number of items to skip from the beginning
- Validation: Must be a non-negative integer

Parameter Validation in Handlers

Query parameters are always strings in HTTP. Convert and validate them in your route handlers:

// Parse and validate pagination parameters
function validatePaginationParams(query: any): { limit: number; offset: number } {
  const limit = query.limit ? parseInt(query.limit, 10) : 20;
  const offset = query.offset ? parseInt(query.offset, 10) : 0;
  
  // Validate limit
  if (isNaN(limit) || limit < 1 || limit > 100) {
    throw new Error('Limit must be between 1 and 100');
  }
  
  // Validate offset
  if (isNaN(offset) || offset < 0) {
    throw new Error('Offset must be non-negative');
  }
  
  return { limit, offset };
}

Standard Response Format

Response Schema

All paginated endpoints should return responses in this format:

const PAGINATED_RESPONSE_SCHEMA = {
  type: 'object',
  properties: {
    success: { type: 'boolean' },
    data: {
      type: 'object',
      properties: {
        // Your actual data array (name varies by endpoint)
        items: { 
          type: 'array',
          items: { /* your item schema */ }
        },
        
        // Pagination metadata
        pagination: {
          type: 'object',
          properties: {
            total: { 
              type: 'number',
              description: 'Total number of items available'
            },
            limit: { 
              type: 'number',
              description: 'Items per page (as requested)'
            },
            offset: { 
              type: 'number',
              description: 'Current offset (as requested)'
            },
            has_more: { 
              type: 'boolean',
              description: 'Whether more items are available'
            }
          },
          required: ['total', 'limit', 'offset', 'has_more']
        }
      },
      required: ['items', 'pagination']
    }
  },
  required: ['success', 'data']
} as const;

Response Example

{
  "success": true,
  "data": {
    "servers": [
      {
        "id": "server-1",
        "name": "Example Server",
        // ... other server fields
      }
      // ... more servers
    ],
    "pagination": {
      "total": 150,
      "limit": 20,
      "offset": 40,
      "has_more": true
    }
  }
}

Pagination Metadata Fields

total: Total number of items available (across all pages)
limit: Number of items per page (echoes the request parameter)
offset: Current starting position (echoes the request parameter)
has_more: Boolean indicating if more items are available after this page

Implementation Pattern

1. Route Schema Definition

import { type FastifyInstance } from 'fastify';

// Query parameters schema (including pagination)
const QUERY_SCHEMA = {
  type: 'object',
  properties: {
    // Your filtering parameters
    category: { 
      type: 'string',
      description: 'Filter by category'
    },
    status: { 
      type: 'string',
      enum: ['active', 'inactive'],
      description: 'Filter by status'
    },
    
    // Standard pagination parameters
    limit: {
      type: 'string',
      pattern: '^\\d+$',
      description: 'Maximum number of items to return (1-100, default: 20)'
    },
    offset: {
      type: 'string',
      pattern: '^\\d+$',
      description: 'Number of items to skip (≥0, default: 0)'
    }
  },
  additionalProperties: false
} as const;

// Response schema
const RESPONSE_SCHEMA = {
  type: 'object',
  properties: {
    success: { type: 'boolean' },
    data: {
      type: 'object',
      properties: {
        items: {
          type: 'array',
          items: {
            // Your item schema here
            type: 'object',
            properties: {
              id: { type: 'string' },
              name: { type: 'string' },
              // ... other properties
            }
          }
        },
        pagination: {
          type: 'object',
          properties: {
            total: { type: 'number' },
            limit: { type: 'number' },
            offset: { type: 'number' },
            has_more: { type: 'boolean' }
          },
          required: ['total', 'limit', 'offset', 'has_more']
        }
      },
      required: ['items', 'pagination']
    }
  },
  required: ['success', 'data']
} as const;

// TypeScript interfaces
interface QueryParams {
  category?: string;
  status?: 'active' | 'inactive';
  limit?: string;
  offset?: string;
}

interface PaginatedResponse {
  success: boolean;
  data: {
    items: Item[];
    pagination: {
      total: number;
      limit: number;
      offset: number;
      has_more: boolean;
    };
  };
}

2. Route Handler Implementation

export default async function listItems(server: FastifyInstance) {
  server.get('/api/items', {
    schema: {
      tags: ['Items'],
      summary: 'List items with pagination',
      description: 'Retrieve items with pagination support. Supports filtering and sorting.',
      
      querystring: QUERY_SCHEMA,
      
      response: {
        200: RESPONSE_SCHEMA,
        400: {
          type: 'object',
          properties: {
            success: { type: 'boolean', default: false },
            error: { type: 'string' }
          }
        }
      }
    }
  }, async (request, reply) => {
    try {
      // Parse and validate query parameters
      const query = request.query as QueryParams;
      const { limit, offset } = validatePaginationParams(query);
      
      // Extract filter parameters
      const filters = {
        category: query.category,
        status: query.status
      };
      
      // Get all items (with filtering applied)
      const allItems = await yourService.getItems(filters);
      
      // Apply pagination
      const total = allItems.length;
      const paginatedItems = allItems.slice(offset, offset + limit);
      
      // Log pagination info
      server.log.info({
        operation: 'list_items',
        totalResults: total,
        returnedResults: paginatedItems.length,
        pagination: { limit, offset }
      }, 'Items list completed');
      
      // Return paginated response
      const response: PaginatedResponse = {
        success: true,
        data: {
          items: paginatedItems,
          pagination: {
            total,
            limit,
            offset,
            has_more: offset + limit < total
          }
        }
      };
      
      const jsonString = JSON.stringify(response);
      return reply.status(200).type('application/json').send(jsonString);
    } catch (error) {
      server.log.error({ error }, 'Failed to list items');
      
      const errorResponse = {
        success: false,
        error: error instanceof Error ? error.message : 'Failed to retrieve items'
      };
      const jsonString = JSON.stringify(errorResponse);
      return reply.status(400).type('application/json').send(jsonString);
    }
  });
}

Database-Level Pagination (Advanced)

For better performance with large datasets, implement pagination at the database level:

Using Drizzle ORM

import { desc, asc, sql, eq } from 'drizzle-orm';

async getItemsPaginated(
  filters: ItemFilters,
  limit: number,
  offset: number
): Promise<{ items: Item[], total: number }> {
  // Build base query with filters
  let query = this.db.select().from(items);
  
  // Apply filters
  if (filters.category) {
    query = query.where(eq(items.category, filters.category));
  }
  
  // Get total count (before pagination)
  const countQuery = this.db.select({ count: sql<number>`count(*)` }).from(items);
  // Apply same filters to count query
  if (filters.category) {
    countQuery = countQuery.where(eq(items.category, filters.category));
  }
  const [{ count: total }] = await countQuery;
  
  // Apply pagination and ordering
  const paginatedItems = await query
    .orderBy(desc(items.created_at))
    .limit(limit)
    .offset(offset);
  
  return {
    items: paginatedItems,
    total
  };
}

Updated Route Handler

// In your route handler
const { items, total } = await yourService.getItemsPaginated(filters, limit, offset);

const response: PaginatedResponse = {
  success: true,
  data: {
    items,
    pagination: {
      total,
      limit,
      offset,
      has_more: offset + limit < total
    }
  }
};

const jsonString = JSON.stringify(response);
return reply.status(200).type('application/json').send(jsonString);

Client-Side Usage Examples

JavaScript/TypeScript

interface PaginationParams {
  limit?: number;
  offset?: number;
}

interface PaginatedResponse<T> {
  success: boolean;
  data: {
    items: T[];
    pagination: {
      total: number;
      limit: number;
      offset: number;
      has_more: boolean;
    };
  };
}

async function fetchItems(params: PaginationParams = {}): Promise<PaginatedResponse<Item>> {
  const url = new URL('/api/items', baseUrl);
  
  if (params.limit) url.searchParams.set('limit', params.limit.toString());
  if (params.offset) url.searchParams.set('offset', params.offset.toString());
  
  const response = await fetch(url.toString(), {
    credentials: 'include',
    headers: { 'Accept': 'application/json' }
  });
  
  return await response.json();
}

// Usage examples
const firstPage = await fetchItems({ limit: 20, offset: 0 });
const secondPage = await fetchItems({ limit: 20, offset: 20 });
const customPage = await fetchItems({ limit: 50, offset: 100 });

Vue.js Composable

import { ref, computed } from 'vue';

export function usePagination<T>(
  fetchFunction: (limit: number, offset: number) => Promise<PaginatedResponse<T>>,
  initialLimit = 20
) {
  const items = ref<T[]>([]);
  const currentPage = ref(1);
  const limit = ref(initialLimit);
  const total = ref(0);
  const loading = ref(false);
  
  const totalPages = computed(() => Math.ceil(total.value / limit.value));
  const hasNextPage = computed(() => currentPage.value < totalPages.value);
  const hasPrevPage = computed(() => currentPage.value > 1);
  
  const offset = computed(() => (currentPage.value - 1) * limit.value);
  
  async function loadPage(page: number) {
    if (page < 1 || page > totalPages.value) return;
    
    loading.value = true;
    try {
      const response = await fetchFunction(limit.value, (page - 1) * limit.value);
      items.value = response.data.items;
      total.value = response.data.pagination.total;
      currentPage.value = page;
    } finally {
      loading.value = false;
    }
  }
  
  async function nextPage() {
    if (hasNextPage.value) {
      await loadPage(currentPage.value + 1);
    }
  }
  
  async function prevPage() {
    if (hasPrevPage.value) {
      await loadPage(currentPage.value - 1);
    }
  }
  
  return {
    items,
    currentPage,
    limit,
    total,
    totalPages,
    loading,
    hasNextPage,
    hasPrevPage,
    loadPage,
    nextPage,
    prevPage
  };
}

Best Practices

1. Consistent Parameter Validation

Always use the same validation rules across all endpoints:

// Create a reusable validation function
export function validatePaginationParams(query: any): { limit: number; offset: number } {
  const limit = query.limit ? parseInt(query.limit, 10) : 20;
  const offset = query.offset ? parseInt(query.offset, 10) : 0;
  
  if (isNaN(limit) || limit < 1 || limit > 100) {
    throw new Error('Limit must be between 1 and 100');
  }
  
  if (isNaN(offset) || offset < 0) {
    throw new Error('Offset must be non-negative');
  }
  
  return { limit, offset };
}

// Reusable schema constant
export const PAGINATION_QUERY_SCHEMA = {
  type: 'object',
  properties: {
    limit: {
      type: 'string',
      pattern: '^\\d+$',
      description: 'Maximum number of items to return (1-100, default: 20)'
    },
    offset: {
      type: 'string',
      pattern: '^\\d+$',
      description: 'Number of items to skip (≥0, default: 0)'
    }
  },
  additionalProperties: false
} as const;

// Use in your endpoint schemas
const QUERY_SCHEMA = {
  type: 'object',
  properties: {
    // Your specific filters
    category: { type: 'string' },
    status: { type: 'string', enum: ['active', 'inactive'] },
    
    // Include pagination
    ...PAGINATION_QUERY_SCHEMA.properties
  },
  additionalProperties: false
} as const;

2. Proper Error Handling

try {
  const { limit, offset } = validatePaginationParams(request.query);
  // ... rest of handler
} catch (error) {
  const errorResponse = {
    success: false,
    error: error instanceof Error ? error.message : 'Invalid query parameters'
  };
  const jsonString = JSON.stringify(errorResponse);
  return reply.status(400).type('application/json').send(jsonString);
}

3. Performance Considerations

Database Pagination: Use LIMIT and OFFSET at the database level for large datasets
Indexing: Ensure proper database indexes on columns used for sorting
Caching: Consider caching total counts for frequently accessed endpoints
Reasonable Limits: Enforce maximum page sizes (e.g., 100 items)

4. OpenAPI Documentation

Include clear pagination documentation in your API specs:

schema: {
  tags: ['Items'],
  summary: 'List items with pagination',
  description: `
    Retrieve items with pagination support. 
    
    **Pagination Parameters:**
    - \`limit\`: Items per page (1-100, default: 20)
    - \`offset\`: Items to skip (≥0, default: 0)
    
    **Response includes:**
    - \`data.items\`: Array of items for current page
    - \`data.pagination.total\`: Total items available
    - \`data.pagination.has_more\`: Whether more pages exist
  `,
  // ... rest of schema
}

Common Pitfalls and Solutions

1. Inconsistent Response Formats

❌ Wrong: Different endpoints use different response structures

// Endpoint A
{ data: items, total: 100, page: 1 }

// Endpoint B  
{ results: items, count: 100, offset: 20 }

✅ Correct: Use standardized response format

// All endpoints
{
  success: true,
  data: {
    items: [...],
    pagination: { total, limit, offset, has_more }
  }
}

2. Missing Validation

❌ Wrong: No parameter validation

const limit = parseInt(request.query.limit) || 20;
const offset = parseInt(request.query.offset) || 0;

✅ Correct: Proper validation function

const { limit, offset } = validatePaginationParams(request.query);

3. Performance Issues

❌ Wrong: Loading all data then slicing

const allItems = await db.select().from(items); // Loads everything!
const paginated = allItems.slice(offset, offset + limit);

✅ Correct: Database-level pagination

const items = await db.select().from(items)
  .limit(limit)
  .offset(offset);

4. Incorrect Total Count

❌ Wrong: Using paginated results length

const items = await getItemsPaginated(limit, offset);
const total = items.length; // Wrong! This is just current page

✅ Correct: Separate count query

const [items, total] = await Promise.all([
  getItemsPaginated(limit, offset),
  getItemsCount(filters)
]);

Real-World Examples

Example 1: MCP Servers List (Current Implementation)

// File: services/backend/src/routes/mcp/servers/list.ts
export default async function listServers(server: FastifyInstance) {
  server.get('/mcp/servers', {
    schema: {
      tags: ['MCP Servers'],
      summary: 'List MCP servers',
      description: 'Retrieve MCP servers with pagination support...',
      querystring: QUERY_SCHEMA,
      response: {
        200: LIST_SERVERS_RESPONSE_SCHEMA
      }
    }
  }, async (request, reply) => {
    const { limit, offset } = validatePaginationParams(request.query);
    const filters = extractFilters(request.query);
    
    const allServers = await catalogService.getServersForUser(
      userId, userRole, teamIds, filters
    );
    
    const total = allServers.length;
    const paginatedServers = allServers.slice(offset, offset + limit);
    
    const response = {
      success: true,
      data: {
        servers: paginatedServers,
        pagination: {
          total,
          limit,
          offset,
          has_more: offset + limit < total
        }
      }
    };
    
    const jsonString = JSON.stringify(response);
    return reply.status(200).type('application/json').send(jsonString);
  });
}

Example 2: Search Endpoint (Reference Implementation)

The search endpoint (/mcp/servers/search) demonstrates the complete pagination pattern and can serve as a reference for implementing pagination in other endpoints.

Testing Pagination

Unit Tests

describe('Pagination', () => {
  test('should return first page with default limit', async () => {
    const response = await request(app)
      .get('/api/items')
      .expect(200);
    
    expect(response.body.data.pagination).toEqual({
      total: expect.any(Number),
      limit: 20,
      offset: 0,
      has_more: expect.any(Boolean)
    });
  });
  
  test('should handle custom pagination parameters', async () => {
    const response = await request(app)
      .get('/api/items?limit=10&offset=20')
      .expect(200);
    
    expect(response.body.data.pagination.limit).toBe(10);
    expect(response.body.data.pagination.offset).toBe(20);
  });
  
  test('should validate pagination parameters', async () => {
    await request(app)
      .get('/api/items?limit=invalid')
      .expect(400);
      
    await request(app)
      .get('/api/items?limit=101') // Over maximum
      .expect(400);
  });
});

Integration Tests

test('should paginate through all results', async () => {
  const limit = 5;
  let offset = 0;
  let allItems = [];
  let hasMore = true;
  
  while (hasMore) {
    const response = await request(app)
      .get(`/api/items?limit=${limit}&offset=${offset}`)
      .expect(200);
    
    const { items, pagination } = response.body.data;
    allItems.push(...items);
    
    hasMore = pagination.has_more;
    offset += limit;
  }
  
  // Verify we got all items
  expect(allItems.length).toBe(totalExpectedItems);
});

On this page