SmartModule ArrayMaps are used to break apart Records into smaller pieces. This can be very useful for working with your data at a fine granularity. Often, each record in a Topic may actually represent many data points, but we’d like to be able to analyze and manipulate those data points independently. ArrayMap allows us to dig in and break apart these composite records into the smaller units of data that we want to work with.
Let’s take a look at an example ArrayMap and walk through how it works and what some sample input and output data might look like.
Prerequisites
This section assumes that SMDK is installed.
A common use case is to transform JSON arrays and produce a stream of the records of those arrays. For example, suppose that each element we receive in Fluvio is a JSON array, though we want to interact with the elements of these arrays rather than the arrays themselves. Then, using an ArrayMap, we can transform composite arrays to records that look like this (where this line is a single record):
["a", "b", "c"]
But, we want those elements as a distinct record, like this (where each line is a distinct record):
"a"
"b"
"c"
If you’d like to see a practical example of ArrayMap in action, check out our blog on using ArrayMap to break apart paginated API requests.
Let’s dive in and see how to this up in Fluvio.
Run smdk generate
with the name of the filter and choose the “filter” options:
$ smdk generate array-map
Generating new SmartModule project: array-map
project-group => 'john'
fluvio-smartmodule-cargo-dependency => '"0.3.0"'
๐ง Destination: ~/smdk/array-map ...
๐ง Generating template ...
โ ๐คท Will your SmartModule use init parameters? ยท false
โ ๐คท Which type of SmartModule would you like? ยท array-map
โ ๐คท Will your SmartModule be public? ยท false
Ignoring: /var/folders/5q/jwc86771549058kmbkbqjcdc0000gn/T/.tmp4imt4g/cargo-generate.toml
[1/5] Done: Cargo.toml
[2/5] Done: README.md
[3/5] Done: SmartModule.toml
[4/5] Done: src/lib.rs
[5/5] Done: src
๐ง Moving generated files into: `~/smdk/array-map`...
๐ก Initializing a fresh Git repository
โจ Done! New project created ~/smdk/array-map
The code in this generated project takes JSON arrays as input records and returns the elements of those arrays as output records. Let’s take a look at the full source, then we’ll cover it piece by piece. Let’s look at src/lib.rs
:
$ cd array-map && cat src/lib.rs
use fluvio_smartmodule::{smartmodule, SmartModuleRecord, RecordData, Result};
#[smartmodule(array_map)]
pub fn array_map(record: &SmartModuleRecord) -> Result<Vec<(Option<RecordData>, RecordData)>> {
// Deserialize a JSON array with any kind of values inside
let array: Vec<serde_json::Value> = serde_json::from_slice(record.value.as_ref())?;
// Convert each JSON value from the array back into a JSON string
let strings: Vec<String> = array
.into_iter()
.map(|value| serde_json::to_string(&value))
.collect::<core::result::Result<_, _>>()?;
// Create one record from each JSON string to send
let records: Vec<(Option<RecordData>, RecordData)> = strings
.into_iter()
.map(|s| (None, RecordData::from(s)))
.collect();
Ok(records)
}
This ArrayMap essentially has three steps it takes:
- Deserialize a JSON array as input and store it in a
Vec<Value>
- Converts each
Value
back into a JSON string - Converts each JSON string into a distinct output Record
Let’s take this for a test drive and see it in action.
Let’s make sure our code compiles. If eveything works as expected, there is a .wasm
file generated in the target directory.
$ smdk build
...
Compiling array-map v0.1.0 (~/smdk/array-map)
Finished release-lto [optimized] target(s) in 11.31s
Your SmartModule WASM binary is now ready for use.
Let’s test our work using the command line test
facility.
$ smdk test --text='["a", "b", "c"]'
loading module at: ~/smdk/array-map/target/wasm32-unknown-unknown/release-lto/array_map.wasm
3 records outputed
"a"
"b"
"c"
Great, everything works as expected. Let’s test on cluster.
Let’s create a new Fluvio topic to produce the sample records we want to consume with our SmartModule:
$ fluvio topic create array-map
topic "array-map" created
Now we can produce the sample data to our topic.
$ fluvio produce array-map
> ["a", "b", "c"]
Ok!
> ["d", "e", "f"]
Ok!
> ^C
Let’s double check it’s all there.
$ fluvio consume array-map -dB
Consuming records from the beginning of topic 'array-map'
["a", "b", "c"]
["d", "e", "f"]
The SmartModule can be loaded to local Fluvio Cluster or InfinyOn Cloud, as determined by the current profile
. In this example, the profile points to InfinyOn Cloud.
$ smdk load
Found SmartModule package: array-map
loading module at: ~/smdk/array-map/target/wasm32-unknown-unknown/release-lto/array_map.wasm
Trying connection to fluvio router.infinyon.cloud:9003
Creating SmartModule: array-map
Rust fluvio smartmodule list
to ensure your SmartModule has been uploaded:
$ fluvio smartmodule list
SMARTMODULE SIZE
john/array-map@0.1.0 157.8 KB
SmartModules that have been uploaded on the cluster can be used by other areas of the system (consumers, producers, connectors, etc):
$ fluvio consume array-map -dB --smartmodule=john/array-map@0.1.0
Consuming records from the beginning of topic 'array-map'
"a"
"b"
"c"
"d"
"e"
"f"
Congratulations! ๐ Eveything worked as expected!
It turns out this SmartModule was requested by other data streaming teams in the organization, so we’ve decided to publish it on SmartModule Hub.
$ smdk publish
Creating package john/array-map@0.1.0
.. fill out info in hub/package-meta.yaml
Package hub/array-map-0.1.0.ipkg created
Package uploaded!
Let’s double check that the SmartModule is available for download:
$ fluvio hub list
SMARTMODULE Visibility
john/array-map@0.1.0 private
...
Congratulations! ๐ Your SmartModule is now available for download in the SmartModule Hub.