Fluvio supports multiple programming languages for developing your own custom connectors. In you’d like to submit your connector to the InfinyOn Connector catalog, send us a note on Discord.
In chapter, we’ll give an step-by-step example on how to develop a Python Connector:
- Build and run your client locally
- Package your Client into Docker Image
- Start Connector
- Verify data in Fluvio topic
You’ll need the following tools installed
- Python 3.x
- pip
- python-fluvio
- python-requests
- Docker
- Rust development environment
- This may be needed to install the
fluviopython package
- This may be needed to install the
Let’s start with building a simple client.
This client is written in Python, but we have client libraries for Rust, Javascript, Java and Go (with community support).
#!/usr/bin/env python3
#
# get-cat-facts.py
# An example Python-based Fluvio connector
from fluvio import Fluvio
import requests
import time
WAIT_SECONDS = 10
CAT_FACTS_API = 'https://catfact.ninja/fact'
CAT_FACTS_TOPIC = 'cat-facts-random'
if __name__ == '__main__':
# Before entering event loop
# Connect to cluster and create a producer before we enter loop
fluvio = Fluvio.connect()
producer = fluvio.topic_producer(CAT_FACTS_TOPIC)
# Event loop
while True:
# Get random cat fact
catfact = requests.get(CAT_FACTS_API)
# Save fact
producer.send_string(catfact.text)
# Print fact to container logs
print(catfact.text)
# Be polite and control the rate we send requests to external API
time.sleep(WAIT_SECONDS)
Before we run this code, we need to create the fluvio topic that our client produces data to
$ fluvio topic create cat-facts-random
In order to test our connector application, we’ll need to install libraries in our development environment.
Install the fluvio and requests python packages using pip (or pip3):
$ pip install fluvio requests
Running the Python code prints out a new cat fact every 10 seconds
$ python3 ./get-cat-facts.py
{"fact":"Cats bury their feces to cover their trails from predators.","length":59}
{"fact":"Cats step with both left legs, then both right legs when they walk or run.","length":74}
And we verify that these records have made it to the Fluvio cluster by consuming from the topic.
$ fluvio consume cat-facts-random -B
Consuming records from the beginning of topic 'cat-facts-random'
{"fact":"Cats bury their feces to cover their trails from predators.","length":59}
{"fact":"Cats step with both left legs, then both right legs when they walk or run.","length":74}
This section is optional, but is recommended if you want to share your connectors with others. Packaging with Docker will allow you to bundle in any libraries or tools.
The result will enable another user to start the connector locally without additional setup (assuming they have Docker and Fluvio on their machine).
We use the python base image to keep things simple.
Then we create a new user fluvio with a home directory (in /home/fluvio).
This is required for all connectors. The Fluvio cluster shares information with the fluvio user on startup.
# Dockerfile for Python cat-facts connector
FROM python
# Copy our python script into the connector image
COPY get-cat-facts.py /usr/local/sbin/get-cat-facts.py
RUN chmod +x /usr/local/sbin/get-cat-facts.py
# This is required to connect to a cluster
# Connectors run as the `fluvio` user
ENV USER=fluvio
RUN useradd --create-home "$USER"
USER $USER
# Install Rust development environment
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y
ENV PATH="/home/fluvio/.cargo/bin:${PATH}"
RUN pip install --upgrade pip
# Install dependencies
RUN pip3 install fluvio requests
# Start script on start
ENTRYPOINT get-cat-facts.py
You can build the Docker image with this command.
$ docker build -t cat-facts-connector .
The image should have been created
% docker image list cat-facts-connector
REPOSITORY TAG IMAGE ID CREATED SIZE
cat-facts-connector latest 9a48639f8bac 17 minutes ago 2.66GB
In needed, create your topic before starting your container
$ fluvio topic create cat-facts-random
Start a the container with this docker command
% docker run --init -it --rm -v $HOME/.fluvio:/home/fluvio/.fluvio --network host cat-facts-connector
{"fact":"The first cat show was organized in 1871 in London. Cat shows later became a worldwide craze.","length":93}
...
<CTRL>-C
You can check out docker run --help if you want a description of what these do, so I’ll describe why you want to use them instead.
| Docker option | Why you want it |
|---|---|
--init |
This will ensure that your container will be able to support signals to exit |
-it |
Without both -i and -t, you can’t send ctrl+c to exit signals your container |
--rm |
Prevents accumulating Docker-related mess on your host while you are testing your connector |
-v $HOME/.fluvio:/home/fluvio/.fluvio |
Share existing Fluvio config - This is effectively what fluvio connector create does |
--network host |
The default docker network can’t reach the host’s Fluvio cluster |
We can look at topic, and verify our records.
% fluvio consume cat-facts-random -B
Consuming records from the beginning of topic 'cat-facts-random'
{"fact":"The first cat show was organized in 1871 in London. Cat shows later became a worldwide craze.","length":93}
{"fact":"A cat's cerebral cortex contains about twice as many neurons as that of dogs. Cats have 300 million neurons, whereas dogs have about 160 million. See, cats rule, dogs drool!","length":173}
We just walked through how to develop a Fluvio connector from scratch, using the Fluvio python library.
Also we demonstrated how to package your connector with Docker, how to start the Docker-based connector, and verified data in our topic.
You are now prepared to build your own connectors. This one was for Python, but the pattern should be the same for any of our other language bindings or CLI.
