📬 How to Create a New Pub/Sub Topic on GCP

🌟 What is Google Cloud Pub/Sub?

Google Cloud Pub/Sub is a fully-managed messaging service that facilitates asynchronous communication between applications and services. It enables low-latency, durable messaging for workflows like event notifications, real-time data streaming, and asynchronous task execution.

Pub/Sub operates on a **publisher-subscriber model**:

• Publisher: Sends messages to a Topic.
• Subscriber: Receives messages from the Topic.

👉 Learn more about Google Cloud Pub/Sub on the official documentation.

Key Features of GCP Pub/Sub

1. Publisher-Subscriber Model:
   • Publisher: Sends messages to a topic.
   • Subscriber: Subscribes to a topic and receives messages via a subscription.
2. Asynchronous Messaging:
   • Enables systems to operate independently without waiting for immediate responses.
3. Durable and Reliable:
   • Messages are stored until they are delivered to subscribers or their retention duration expires.
4. Global Scalability:
   • Handles millions of messages per second with low latency.
   • Scales automatically to accommodate varying workloads.
5. Message Ordering:
   • Ensures that messages published to a topic with message ordering are delivered in the same order.
6. Integration:
   • Works seamlessly with other Google Cloud services like Dataflow, BigQuery, and Cloud Functions.
7. Schema Support:
   • Ensures that published messages conform to a predefined schema, reducing errors.
8. Dead Letter Queue (DLQ):
   • Unacknowledged messages can be forwarded to a dead letter topic for debugging or reprocessing.
9. Security:
   • Integrated with IAM (Identity and Access Management) for role-based access control.
   • Supports encryption for data at rest and in transit.

Core Components of GCP Pub/Sub

1. Topics:
   • A topic is a named resource to which publishers send messages.
   • It acts as the entry point for messages.
2. Messages:
   • Messages are data objects sent by publishers.
   • Each message can contain a payload (data) and attributes (key-value metadata).
3. Subscriptions:
   • A subscription is the mechanism that delivers messages from a topic to a subscriber.
   • Types of subscriptions:
     • Pull Subscription: Subscribers pull messages from Pub/Sub.
     • Push Subscription: Pub/Sub pushes messages to an endpoint specified by the subscriber.
4. Dead Letter Topics:
   • A destination for messages that cannot be delivered successfully.
5. Acknowledgment:
   • Subscribers must acknowledge (ack) receipt of a message to indicate successful processing.

How GCP Pub/Sub Works

1. Publish Messages:
   • A publisher sends a message to a topic.
   • Messages are retained in Pub/Sub until they are acknowledged by all subscriptions or the retention period expires.
2. Deliver Messages:
   • Pub/Sub delivers messages to all active subscriptions for a topic.
   • Messages can be delivered via pull or push methods.
3. Acknowledge Messages:
   • Subscribers process the messages and send an acknowledgment to Pub/Sub.
   • Unacknowledged messages are retried until acknowledged or until they expire.

Use Cases of GCP Pub/Sub

1. Real-Time Event Streaming:
   • Stream and process data in real-time, such as sensor data, logs, or clickstream data.
2. Asynchronous Workflows:
   • Decouple services in a microservices architecture to allow asynchronous communication.
3. Event Notifications:
   • Send notifications for events like user sign-ups, payment processing, or system alerts.
4. Data Ingestion:
   • Ingest large volumes of data into data lakes or streaming pipelines.
5. IoT Applications:
   • Connect IoT devices to cloud applications for data streaming and processing.
6. Distributed Systems:
   • Use Pub/Sub to synchronize events between distributed applications.

Types of Subscriptions in Pub/Sub

1. Pull Subscription:
   • Subscribers actively pull messages from the subscription.
   • Suitable for applications that control when they receive messages.
2. Push Subscription:
   • Pub/Sub pushes messages to a subscriber’s endpoint (HTTP/HTTPS).
   • Ideal for applications that can process messages in real-time.
3. Streaming Subscription:
   • Stream messages using Pub/Sub client libraries for high-performance message processing.

🔧 Step-by-Step Guide to Create a Pub/Sub Topic

Follow these steps to create a Pub/Sub Topic in Google Cloud:

1. Log in to Google Cloud Console

1. Go to the Google Cloud Console and sign in with your credentials.
2. Ensure you have the required permissions (e.g., Pub/Sub Admin role) to create a Topic.

2. Navigate to the Pub/Sub Section

1. Open the Pub/Sub Topics Page.
2. Click the Create Topic button.

3. Create a New Topic

In the “Create a Topic” pop-up:

1. Enter a Topic ID (e.g., `my-topic`). This will be the unique identifier for your Topic.
2. Optional: Configure additional settings such as:
   • Default Subscription: Automatically creates a subscription for this Topic.
   • Use a Schema: Enforce a specific message format.
   • Message Retention Duration: Retain messages for up to 31 days (a paid feature).
3. Click Create Topic.

✨ Additional Options When Creating a Pub/Sub Topic

1. Add a Default Subscription

By selecting this option, a subscription will be automatically created with the following default settings:

• Delivery type: Pull
• Message retention: 7 days
• Expiration: 31 days of inactivity
• Acknowledgment deadline: 10 seconds
• Retry policy: Immediate

If you want a customized subscription, follow these steps:

1. Navigate to the Subscriptions Page.
2. Click Create Subscription.
3. Enter a Subscription Name and select the Topic ID.
4. Configure additional settings like Dead-Lettering, message expiration, or retry policies.
5. Click Create.

2. Use a Schema

A Schema enforces a specific format for the messages published to a Topic. You can either:

• Create a new schema for the Topic.
• Assign an existing schema to the Topic.

Select the schema type (e.g., Protocol Buffers or Avro) and define the encoding format.

3. Set Message Retention Duration

By default, Pub/Sub Topics retain messages for 7 days. Enabling **Message Retention Duration** lets you retain messages for up to 31 days. This feature is useful for data recovery or reprocessing but incurs additional costs.

📋 Use Cases for Google Cloud Pub/Sub

• 🚀 Event Notifications: Deliver real-time notifications across distributed applications.
• 📊 Streaming Data: Process real-time data from IoT devices or logs.
• 🔄 Asynchronous Workflows: Decouple microservices by implementing asynchronous communication.
• 📦 Queueing Systems: Use Pub/Sub to create durable and scalable message queues.