Loading...
Loading...
NebulaStream is a general purpose, end-to-end data management system for the IoT.
Coding for 7 years.
NebulaStream is an end-to-end data-management system for cloud-edge-sensor deployments. The platform combines ease of use, extensibility, and efficiency to let teams focus on business logic while the engine optimizes the data path.
Core Pillars
NebulaStream is a joint research project at BIFOLD with contributors from the DIMA Group at TU Berlin and the DFKI IAM Group. It aims to execute thousands of queries over millions of heterogeneous sources in massively distributed environments. We advance this vision through five core technologies:
The system architecture spans sensor to cloud:
Install Nix with flakes enabled and generate build links:
nix run .#clion-setup
Configure, build, and test through the Nix-wrapped helpers:
./.nix/nix-cmake.sh \
-DCMAKE_BUILD_TYPE=Debug \
-G Ninja \
-S . -B cmake-build-debug
./.nix/nix-cmake.sh --build cmake-build-debug
./.nix/ctest --test-dir cmake-build-debug -j
Build or reuse the local development image (installs the current user inside the container to avoid permission issues):
./scripts/install-local-docker-environment.sh
Configure, build, and test by mounting the repository into the container:
docker run \
--workdir $(pwd) \
-v $(pwd):$(pwd) \
nebulastream/nes-development:local \
cmake -B cmake-build-debug
docker run \
--workdir $(pwd) \
-v $(pwd):$(pwd) \
nebulastream/nes-development:local \
cmake --build cmake-build-debug -j
docker run \
--workdir $(pwd) \
-v $(pwd):$(pwd) \
nebulastream/nes-development:local \
ctest --test-dir cmake-build-debug -j
Build the systest executable in your chosen build directory and execute a specific query. The example below targets the first query in the arithmetic function suite and stores artifacts in cmake-build-debug/systest-run:
cmake --build cmake-build-debug -j --target systest
cmake-build-debug/systest/systest -t nes-systests/function/arithmetical/FunctionAdd.test:1
Append worker configuration overrides after -- (for example -- --worker.default_query_execution.execution_mode=INTERPRETER). See the systest guide for authoring custom .test files and the complete CLI reference.
Build the client and worker binaries, start the worker in one terminal, and submit a short-lived query from another terminal using nebuli:
cmake --build cmake-build-debug -j --target nes-single-node-worker nes-nebuli
# Terminal 1: start the worker (listens on localhost:8080 by default)
cmake-build-debug/nes-single-node-worker/nes-single-node-worker
# Terminal 2: prepare a tiny CSV and submit a query via nebuli
printf '1\n2\n3\n' > demo-input.csv
cat > demo.sql <<'EOF'
CREATE LOGICAL SOURCE demo(value UINT64);
CREATE PHYSICAL SOURCE FOR demo TYPE File SET('./demo-input.csv' AS `SOURCE`.FILE_PATH, 'CSV' AS PARSER.`TYPE`, '\n' AS PARSER.TUPLE_DELIMITER, ',' AS PARSER.FIELD_DELIMITER);
CREATE SINK result(demo.value UINT64) TYPE File SET('./demo-output.csv' AS `SINK`.FILE_PATH, 'CSV' AS `SINK`.INPUT_FORMAT);
SELECT value FROM demo INTO result;
EOF
cmake-build-debug/nes-nebuli/nes-nebuli -s localhost:8080 -w < demo.sql
The generated CSV appears at demo-output.csv. Inspect or retire the query with additional nebuli subcommands such as show, stop, or unregister as needed.
clang-format, clang-tidy, license checks, and pragma guards; run the format target before submitting patches.