Kafka consumers

Create a new Kafka topic

1. Add the topic to the KAFKA_TOPIC_TO_CLUSTER in src/sentry/conf/server.py:

• e.g. subscription-results-eap-items

2. Add the topic to Topic in src/sentry/conf/types/kafka_definition.py

Define or re-use a processing strategy

In most cases a Streaming Factory is what you want to when defining a consumer (see next section). You can find examples of it in Sentry's code base.

Define a new Kafka consumer

1. Add a new entry in the KAFKA_CONSUMERS key in src/sentry/consumers/init.py:

KAFKA_CONSUMERS = {
    "<your_topic_str_here>": {
        "topic": Topic.YOUR_TOPIC,
        "strategy_factory": "sentry_package_defining_your_strategy_factory_class",
    }
}

2. You may need optional properties (e.g. click_options, you will need to research them by looking at ConsumerDefinition's code.

3. Make sure you can run it: sentry run consumer <your_topic>

4. You may need to add some devserver options here.

5. Add tests for your consumer

Deployment

Visit the Ops repo and search for shared_config/kafka/README.md for a full, in-depth step-by-step guide.