In the world of distributed systems and microservices, message queues play a vital role in enabling reliable communication between components. Oracle Cloud Infrastructure (OCI) offers a robust and scalable message queuing service that simplifies the process of building event-driven architectures. The OCI Queue Service introduces a simple concept: publish messages to a persistent store and enable consumers to retrieve them from the store in the order they were received, following the First-In-First-Out (FIFO) principle.
OCI Queue is a fully managed, highly available and durable messaging service provided by Oracle Cloud. It allows developers to decouple the sending and receiving of messages, facilitating asynchronous communication between various system components. By leveraging OCI Queue we can build scalable, fault-tolerant and most importantly event-driven architectures
The OCI Queue allows multiple producers to send messages to a shared queue. Several consumers can inquire the queue to fetch messages and contend for their processing, yet only one consumer will ultimately receive each message (without any dissemination). Once a message is successfully consumed by a consumer, it is considered processed and will not be delivered to other consumers. However, if a consumer fails to report success, the message will become available again for another consumer to process. It is important to note that OCI Queue does not include the concept of subscriptions for message delivery.
OCI Queue can be accessed through REST API calls or the STOMP protocol (Simple Text Oriented Messaging Protocol), allowing seamless interaction with the service. STOMP clients can directly connect to OCI Queue for native communication. Additionally, the OCI SDKs for popular languages like Java, Python, and Go, as well as the OCI CLI. In this blog, we will focus on the Python SDK.
Create a new Queue
Access the Queue service in your OCI account. This can easily be done by searching for “Queue” in the search bar
In the Queue overview page click on Create Queue.
Type the name of the queue and select the compartment where the queue should be created - We used hello_world and compartment RnD
Select Custom Configuration and let's go through these configuration parameters
Visibility timeout: refers to the total time a message is hidden from message consumers after it has been collected by a consumer.
Maximum retention period: the amount of time a message can remain in the queue if no successful consumption happens. In other words the amount of time the queue can retain a message until it is deleted.
Dead letter queue settings
Messages that are not successfully processed within the maximum number of delivery attempts are moved to a Dead Letter Queue and will be kept there until either the retention time ends or a form of “manual” processing handles the message.
Maximum delivery attempts: The maximum number of retries for a message consumption before it is sent to a dead letter queue (DLQ).
Press Create and the queue overview page is shown. The status of hello_world is shown as Creating
and when it is ready to be used the state changes to Active
Now the queue is created let's gather some attributes that will later be needed in the Python application.
Click in the queue name - here hello_world, and in the queue information area gather the following attributes: OCID, DQL OCID, Messages Endpoint
When working with ODI SDKs, SSH keys must be created in other to access OCI resources.
Access your OCI account and open the Profile menu, click on API Keys from the bottom left corner and click on Add API Key
Download the Private Key and Public key as you will need it late and click Add
A new window will be shown with the sample of a configuration file needed later (save these values as they will be needed during the python configuration).
Python environment configuration
pipenv to wrap the application development under its own environment and as a package manager.
This will create a virtual environment where all the code and packages will be installed.
As part of our development principles, we create Python applications with a specific structure that makes code easier to read and organise. If you use Visual Studio Code, you could have a structure similar to this:
Pipfile file and add
oci SDK to the package dependencies
In the terminal navigate to the development environment and run
pipenv install - here
If your app structure follows the suggested structure below, under the conf directory create a conf.ini file and add the following information:
Running both files under the virtual environment, we will see messages being produced and consumed.