This Week in Fluvio #50

Mar 31, 2023

Fluvio is a distributed, programmable streaming platform written in Rust.

This Week in Fluvio is our weekly newsletter for development updates to Fluvio open source.

Welcome to the 50th edition of this week in Fluvio.

It’s been 18 weeks since we last published our weekly newsletter. It was our first Fluvio Newsletter winter!

We are rebooting the newsletter and there is a lot more to come…

New release

https://github.com/infinyon/fluvio/releases/tag/v0.10.6

New features

Connector Development Kit Launch: The goal of the Connector Development Kit (CDK) is to enable developers to write connectors to read data from data sources across different services using fluvio and Rust. We have been internally building our connectors using the CDK and are now ready to share it with the open source community.

To get started, install cdk with the fluvio CLI

$ fluvio install cdk

Generating a project

$ cdk generate
🤷   Project Name: my-connector
🔧   Destination: ~/my-connector ...
🔧   project-name: my-connector ...
🔧   Generating template ...
✔ 🤷   Will your Connector be public? · false
🤷   Please set a group name: aj
✔ 🤷   Which type of Connector would you like [source/sink]? · source
Ignoring: /var/folders/r8/4x6_d2rn283946frzd1gc1pr0000gn/T/.tmptToFV3/cargo-generate.toml
[1/6]   Done: Cargo.toml             
[2/6]   Done: Connector.toml
[3/6]   Done: sample-config.yaml
[4/6]   Done: src/config.rs
[5/6]   Done: src/main.rs
[6/6]   Done: src
🔧   Moving generated files into: `~/my-connector`...
💡   Initializing a fresh Git repository
✨   Done! New project created ~/my-connector

This is what the generated rust code for the connector looks like

mod config;
use config::CustomConfig;


use fluvio::{RecordKey, TopicProducer};
use fluvio_connector_common::{
    connector,
    Result
};

#[connector(source)]
async fn start(config: CustomConfig, producer: TopicProducer) -> Result<()> {
    println!("Starting my-connector source connector with {config:?}");
    for i in 1..1000 {
        let value = format!("Hello, Fluvio - {i}");
        producer.send(RecordKey::NULL, value).await?;
        producer.flush().await?;
        std::thread::sleep(std::time::Duration::from_millis(1000));
    }
    Ok(())
}

If you’re interested in learning more, here are the docs to CDK

We are excited to see what you are going to build with the CDK!

We are currently working on a way to certify connectors built by the community and would love your inputs on our Discord channel.

Besides the Connector Development Kit, we have made other improvements to the packaging of Fluvio components.

Full changelog available here: https://github.com/infinyon/fluvio/blob/master/CHANGELOG.md

Upcoming features

We are aware of the frustrations of getting started with fluvio due to the clunky installation process. We are considering a clean binary install, as well as installing using a package manager. However, we have been busy with the CDK, SMDK and other Cloud Features.

We would love to know what do you recommend we build next to make your development flow easier. Please comment in our Discord channel and let us know. Tag Deb (DRC) on your feedback and feature requests.

Bug fixes

Enabled –file and –key-separator to be used together, fix –key handling when producing lines (#3092)

Community update

Carson Rajcan presented to our team a really cool contribution to the Fluvio OSS. In the project Carson developed the functionality to apply Smart Module transformation for Producer - PR #3014

As Winter is now over… We are planning community events to champion community contributors, as well as planning hackathons to build som cool stuff with the community using CDK, SMDK etc.

Let us know what events you are interested in our Discord channel.

InfinyOn Cloud updates

We are putting the finishing touches on connector secrets for you to use the InfinyOn certified connectors built using CDK to interact with InfinyOn Cloud while useing passwords, API keys.

We are also shaping work for materialized views of time based aggregates from topics.

We are also exploring a developer user experience for our cloud platform. More updates on this in a couple of months.