> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ticktock.bet/llms.txt
> Use this file to discover all available pages before exploring further.

# Recovery

> UOF-style stateful recovery — republishes active markets + recent settlements to your AMQP vhost, terminated by snapshot_complete

## When to call it

Use this endpoint whenever your AMQP consumer falls behind and you need to resync without polling REST for every match individually. Typical triggers:

* AMQP connection dropped and your durable queue's TTL elapsed.
* Service restart with no persisted last-message timestamp.
* The `<alive>` message arrived with `subscribed="0"` (producer downtime) — UOF convention is to immediately initiate recovery.

Before starting monitoring or recovery, fetch
`GET /v1/descriptions/producers`. Recovery runs per discovered producer; the
catalog's `producer.id` matches the `product` attribute in feed messages, and
its `scope` can be used as the UOF-compatible recovery path alias.

## How it works

You make a single REST call. The server schedules a background replay onto **your tenant's AMQP vhost** (the same exchange the live feed uses, `ttfeed`). For every booked match the replay publishes:

1. One `<odds_change>` listing the current active B2B market offers with up-to-date odds.
2. One `<bet_settlement>` per match for offers settled between `after` and now.
3. One `<bet_cancel>` per match for offers voided between `after` and now.
4. A terminating `<snapshot_complete request_id="…">` so you know the replay is finished.

Every recovery XML envelope carries the same client-provided `request_id` as
a top-level attribute. Classify replay messages by that positive integer
attribute; the `recovery=1` AMQP header remains available but is not required
for correlation. Recovery routing keys use priority `lo` (vs live `hi`) and
echo `node_id` in their final section.

## Required scope

* **Minimum:** `cs2:recovery:initiate`

This scope is included by default in the `cs2.odds.basic` bundle, so existing odds-feed customers don't need a new key.

## Parameters

| Parameter    | Required | Description                                                                                                                                                                                                                                                                                                       |
| ------------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `after`      | no       | Positive Unix timestamp in **milliseconds** for incremental recovery. The replay includes settlements and voids whose latest transition happened **after** this moment; active markets are always republished. Omit it or pass `0` for a full active-market snapshot with no historical settlement/cancel replay. |
| `request_id` | yes      | Client-chosen positive integer (digits only, first digit `1-9`, ≤ 64 chars). Echoed verbatim in the root XML attribute of every recovery `odds_change`, `bet_settlement`, `bet_cancel`, rollback, and `snapshot_complete` envelope.                                                                               |
| `node_id`    | no       | Optional positive integer (same format, ≤ 64 chars). When supplied, it is echoed verbatim in the last section of every recovery routing key.                                                                                                                                                                      |

### Full snapshot versus incremental recovery

* **Full snapshot:** omit `after` or send `after=0`. Ticktock republishes every
  current active market for booked matches, followed by
  `<snapshot_complete>`. It does not query historical settlements/cancels and
  does not apply the stateful recovery window.
* **Incremental recovery:** send a positive Unix-millisecond `after`. Current
  active markets are republished together with settlement/cancel transitions
  after that timestamp.

The incremental window defaults to `4320` minutes (3 days) and is advertised
by `GET /v1/descriptions/producers`. It is configurable per deployment, not a
hard-coded global maximum. A positive `after` older than the configured window
or in the future returns HTTP `422`.

## Errors

| Status | Reason                                                                                                                                                                                                                                                                                     |
| ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `409`  | Another recovery for the same `(tenant, node_id)` is already in flight. Different `node_id` values from the same tenant can recover in parallel — see [Concurrency](#concurrency) below. Wait for `<snapshot_complete>` from the previous request before retrying with the same `node_id`. |
| `409`  | The API key has `cs2:recovery:initiate` but the tenant has no provisioned AMQP vhost yet. Contact support.                                                                                                                                                                                 |
| `422`  | `request_id` or supplied `node_id` is not a positive integer; or a positive `after` is in the future or older than the configured stateful-recovery window.                                                                                                                                |

## Concurrency

The in-flight lock is keyed on `(tenant, node_id)`. That means:

* **Different `node_id` values from the same tenant can recover at the same time.** For example, nodes `3001` and `3002` can run independent replays and bind queues to their respective numeric routing-key suffixes.
* **Same `node_id` (including omitting it — empty maps to the implicit slot `"global"`) is serialized.** A second request before the first one's `<snapshot_complete>` arrives gets HTTP 409. Wait for that envelope, then retry if you still need to.

The lock has a server-side TTL (10 minutes) that acts as the safety net if a replay worker crashes before releasing it, so a hung recovery cannot lock you out indefinitely.

## Example

```bash theme={null}
# Replay everything since five minutes ago for client node 3003
curl -X POST -H "X-API-Key: $TT_KEY" \
  "https://ticktock.bet/cs2/v1/recovery/initiate_request?after=$(($(date +%s%3N) - 300000))&request_id=4567&node_id=3003"
```

```json theme={null}
{
  "data": {
    "request_id": "4567",
    "after": 1778586502057,
    "node_id": "3003",
    "scheduled_at": 1778586802057
  }
}
```

## Client recipe

<Steps>
  <Step title="Detect the loss">
    Your AMQP consumer disconnects, falls behind, or you see an `<alive subscribed="0">` message. **Stop processing live messages** until recovery completes — either pause your consumer or buffer the incoming envelopes and apply them after `<snapshot_complete>`.
  </Step>

  <Step title="Initiate the replay">
    `POST /cs2/v1/recovery/initiate_request` with `after = <unix_ms of last processed message>`. The server returns `202 Accepted` immediately; the replay is scheduled in the background.
  </Step>

  <Step title="Apply recovery envelopes">
    Resume reading from your AMQP queue. Treat an envelope as recovery when its root XML `request_id` equals the positive integer you sent. The `recovery=1` header and priority `lo` remain supplementary metadata.
  </Step>

  <Step title="Switch back to live">
    When you receive `<snapshot_complete request_id="..."/>` whose `request_id` matches the one you sent, your state is in sync. Apply any buffered live messages, then resume normal processing.
  </Step>
</Steps>

<Tip>
  If you get a 409 "already running" for **your** `node_id`, don't retry — just wait for the in-flight `<snapshot_complete>` to arrive; a single recovery per node delivers everything you need. If you got the 409 from a *different* node sharing your tenant, that's expected and harmless — your own `node_id`'s slot is still free, just send your request with your tag.
</Tip>

## Routing keys produced

```
lo.-.live.odds_change.2.od:match.{match_num}.{node_id}
lo.-.live.bet_settlement.2.od:match.{match_num}.{node_id}
lo.-.live.bet_cancel.2.od:match.{match_num}.{node_id}
-.-.-.snapshot_complete.-.-.-.{node_id}
```

When `node_id` is omitted, the last section is `-`.

## See also

* [AMQP messaging feed](/odds-feed/amqp) — the channel your replay arrives on
* [UOF REST compatibility](/api-reference/uof-rest-compatibility) — producer discovery and per-producer recovery paths
* [Market descriptions](/api-reference/markets/descriptions) — fetch the template + code catalog once at integration time


## OpenAPI

````yaml openapi/cs2-api.json POST /cs2/v1/recovery/initiate_request
openapi: 3.1.0
info:
  title: TickTock B2B API
  description: >-
    Unified CS2 API (``/cs2/v1/*``) plus sport-agnostic ``/v1/*`` meta
    endpoints. Tenant-authenticated via ``X-API-Key`` or ``x-access-token``
    header (REST), or ``api_key`` query parameter (WebSocket). Field visibility
    is gated by per-key scopes — see ``docs/CS2_API_REFERENCE.md`` for the
    catalog and migration.
  version: 1.0.0
servers:
  - url: https://ticktock.bet
    description: Production (same-origin proxy)
  - url: https://feed.ticktock.bet
    description: Production (B2B subdomain)
security: []
paths:
  /cs2/v1/recovery/initiate_request:
    post:
      tags:
        - CS2
      summary: Initiate UOF-style stateful recovery
      description: >-
        Asks the server to republish the current state of the B2B feed to **this
        tenant's** AMQP vhost so the client can resync after disconnection
        without polling REST. Mirrors the UOF
        ``/v1/{product}/recovery/initiate_request`` endpoint.


        On success returns ``202 Accepted`` immediately; the actual replay
        happens in the background and is delivered as ordinary AMQP messages on
        the same exchange the live feed uses, with priority ``lo`` (vs live
        ``hi``) and a ``recovery=1`` header. Every replay envelope carries the
        client-provided positive integer ``request_id`` as a root XML attribute.
        The replay ends with a matching ``<snapshot_complete request_id=…>``
        message — the client knows it is caught up once it sees that.


        What gets replayed:
          * one ``<odds_change>`` per booked match listing the current active B2B offers;
          * with a positive ``after``, one ``<bet_settlement>`` per match for offers settled since that timestamp;
          * with a positive ``after``, one ``<bet_cancel>`` per match for offers voided since that timestamp;
          * a final ``<snapshot_complete>``.

        Omitting ``after`` or passing ``after=0`` requests a full current-state
        snapshot. It republishes active markets, skips historical
        settlement/cancel messages, and is not constrained by the
        stateful-recovery window.


        Errors: ``409`` if a recovery for this tenant is already in flight;
        ``422`` if a positive ``after`` is older than the configured
        stateful-recovery window (default 3 days) or in the future.
      operationId: initiate_recovery_cs2_v1_recovery_initiate_request_post
      parameters:
        - name: after
          in: query
          required: false
          schema:
            anyOf:
              - type: integer
                minimum: 0
              - type: 'null'
            description: >-
              Unix timestamp in milliseconds for incremental recovery.
              Settlements / voids strictly after this moment are replayed. Omit
              or pass 0 for a full active-market snapshot without historical
              settlement/cancel replay.
            title: After
          description: >-
            Unix timestamp in milliseconds for incremental recovery. Settlements
            / voids strictly after this moment are replayed. Omit or pass 0 for
            a full active-market snapshot without historical settlement/cancel
            replay.
        - name: request_id
          in: query
          required: true
          schema:
            type: string
            maxLength: 64
            pattern: ^[1-9][0-9]*$
            description: >-
              Client-chosen positive integer echoed verbatim in the root
              ``request_id`` attribute of every recovery XML envelope.
            title: Request Id
          description: >-
            Client-chosen positive integer echoed verbatim in the root
            ``request_id`` attribute of every recovery XML envelope.
        - name: node_id
          in: query
          required: false
          schema:
            anyOf:
              - type: string
                maxLength: 64
                pattern: ^[1-9][0-9]*$
              - type: 'null'
            description: >-
              Optional positive integer client-node identifier. When supplied,
              it is echoed verbatim in the last section of every recovery AMQP
              routing key.
            title: Node Id
          description: >-
            Optional positive integer client-node identifier. When supplied, it
            is echoed verbatim in the last section of every recovery AMQP
            routing key.
      responses:
        '202':
          description: Successful Response
          content:
            application/json:
              schema:
                type: object
                additionalProperties: true
                title: >-
                  Response Initiate Recovery Cs2 V1 Recovery Initiate Request
                  Post
            application/xml: {}
        '401':
          description: >-
            Missing API key. Pass either `X-API-Key: <your-key>` or
            `$x$access-token: <your-key>` on REST calls (or `?api_key=` on
            WebSocket handshakes).
          content:
            application/json:
              schema:
                type: object
                properties:
                  detail:
                    type: string
                    description: Human-readable error message.
                required:
                  - detail
              example:
                detail: Missing X-API-Key or $x$access-token header
        '403':
          description: >-
            API key authenticated but lacks the scope required for this
            endpoint, or the path's sport is not in the key's `sport_allowlist`.
            Use `GET /v1/whoami` to inspect the granted scope set.
          content:
            application/json:
              schema:
                type: object
                properties:
                  detail:
                    type: string
                    description: Human-readable error message.
                required:
                  - detail
              example:
                detail: >-
                  Required scope 'cs2:matches:detail' is not granted to this API
                  key. Contact your account manager to extend the key's scopes.
        '422':
          description: >-
            Validation error — a query/path parameter failed validation. The
            body lists the offending fields.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
              example:
                detail:
                  - type: enum
                    loc:
                      - query
                      - time_filter
                    msg: 'Input should be one of: 3m, 6m, 1y, all'
                    input: 2w
        '429':
          description: >-
            Tenant rate limit exceeded. Default is 600 req/min per tenant;
            configurable per contract. WebSocket connections are not
            rate-limited at the request layer.
          content:
            application/json:
              schema:
                type: object
                properties:
                  detail:
                    type: string
                    description: Human-readable error message.
                required:
                  - detail
              example:
                detail: Rate limit exceeded (600 req/min)
          headers:
            Retry-After:
              description: Seconds to wait before retrying.
              schema:
                type: integer
                example: 30
      security:
        - XAPIKeyHeader: []
        - XAccessTokenHeader: []
        - UOFAccessTokenHeader: []
components:
  schemas:
    HTTPValidationError:
      properties:
        detail:
          items:
            $ref: '#/components/schemas/ValidationError'
          type: array
          title: Detail
      type: object
      title: HTTPValidationError
    ValidationError:
      properties:
        loc:
          items:
            anyOf:
              - type: string
              - type: integer
          type: array
          title: Location
        msg:
          type: string
          title: Message
        type:
          type: string
          title: Error Type
        input:
          title: Input
        ctx:
          type: object
          title: Context
      type: object
      required:
        - loc
        - msg
        - type
      title: ValidationError
  securitySchemes:
    XAPIKeyHeader:
      type: apiKey
      description: Tenant API key issued during onboarding
      in: header
      name: X-API-Key
    XAccessTokenHeader:
      type: apiKey
      description: UOF-standard alias for the tenant API key
      in: header
      name: x-access-token
    UOFAccessTokenHeader:
      type: apiKey
      description: Deprecated alias for x-access-token (backward compatibility)
      in: header
      name: $x$access-token

````