{ "cells": [ { "cell_type": "markdown", "id": "3e4eb758", "metadata": {}, "source": [ "## Compute Sample" ] }, { "cell_type": "markdown", "id": "2460ecfc", "metadata": {}, "source": [ "In this demo notebook we will showcase running a compute to data:\n", "\n", "- we will create a table in a private data collection that cannot be accessed remotely\n", "- we'll populate it with data\n", "- run a compute to data task to train a model without having access to the data\n", "- verify the task lineage\n", "- generate a ZK proof for the data\n", " \n", "The default Weavechain node installation is preconfigured to support this scenario (by connecting to a public weave, having a *private* data collection defined and mapped to a in-process SQLite instance and read rights for that collection already given)." ] }, { "cell_type": "markdown", "id": "8e0dbccf", "metadata": {}, "source": [ "### 1. Create an API session" ] }, { "cell_type": "code", "execution_count": 31, "id": "d96cfcaf", "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\"res\":\"ok\",\"data\":\"pong 1674735153640\"}\n" ] } ], "source": [ "import pandas as pd\n", "\n", "from weaveapi.records import *\n", "from weaveapi.options import *\n", "from weaveapi.filter import *\n", "from weaveapi.weaveh import *\n", "\n", "WEAVE_CONFIG = \"config/demo_client_local.config\"\n", "nodeApi, session = connect_weave_api(WEAVE_CONFIG)" ] }, { "cell_type": "markdown", "id": "bb1eacb9", "metadata": {}, "source": [ "### 2. Read data from the prepared file" ] }, { "cell_type": "markdown", "id": "b272d4cf", "metadata": {}, "source": [ "- go to the folder where the local node was installed\n", "- download [sample.csv](https://public.weavechain.com/file/sample.csv) and place it under storage/file/private_files folder or run\n", "```\n", "mkdir -p storage/files/private_files\n", "cd storage/files/private_files\n", "curl -O https://public.weavechain.com/file/sample.csv\n", "```\n", "- theoretically we could have used the file from the jupyter server, using this step to show how to connect a local storage to the node\n", "- the *private_files* storage is already configured in the node at install time, the following config section marking it as non-replicated and as storing raw files (many formats are supported, from CSV to feather or ORC, for those each file being treated as a table)\n", "```\n", " 'private_files': {\n", " 'connectionAdapterType': 'file',\n", " 'replication': {\n", " 'type': 'none',\n", " },\n", " 'fileConfig': {\n", " 'rootFolder': 'weavestorage/files',\n", " 'format': 'file'\n", " }\n", " }\n", "```" ] }, { "cell_type": "code", "execution_count": 2, "id": "568e52c8", "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
\n", " | id | \n", "name | \n", "age | \n", "gender | \n", "air_pollution | \n", "alcohol_use | \n", "dust_allergy | \n", "occupational_hazards | \n", "genetic_risk | \n", "chronic_lung_disease | \n", "... | \n", "fatigue | \n", "weight_loss | \n", "shortness_of_breath | \n", "wheezing | \n", "swallowing_difficulty | \n", "clubbing_of_fingernails | \n", "frequent_cold | \n", "dry_cough | \n", "snoring | \n", "level | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | \n", "1 | \n", "Lorenzo Rasmussen | \n", "33 | \n", "1 | \n", "2 | \n", "4 | \n", "5 | \n", "4 | \n", "3 | \n", "2 | \n", "... | \n", "3 | \n", "4 | \n", "2 | \n", "2 | \n", "3 | \n", "1 | \n", "2 | \n", "3 | \n", "4 | \n", "1 | \n", "
1 | \n", "2 | \n", "Zechariah Gallegos | \n", "17 | \n", "1 | \n", "3 | \n", "1 | \n", "5 | \n", "3 | \n", "4 | \n", "2 | \n", "... | \n", "1 | \n", "3 | \n", "7 | \n", "8 | \n", "6 | \n", "2 | \n", "1 | \n", "7 | \n", "2 | \n", "2 | \n", "
2 | \n", "3 | \n", "Lukas Jenkins | \n", "35 | \n", "1 | \n", "4 | \n", "5 | \n", "6 | \n", "5 | \n", "5 | \n", "4 | \n", "... | \n", "8 | \n", "7 | \n", "9 | \n", "2 | \n", "1 | \n", "4 | \n", "6 | \n", "7 | \n", "2 | \n", "3 | \n", "
3 | \n", "4 | \n", "Trey Holden | \n", "37 | \n", "1 | \n", "7 | \n", "7 | \n", "7 | \n", "7 | \n", "6 | \n", "7 | \n", "... | \n", "4 | \n", "2 | \n", "3 | \n", "1 | \n", "4 | \n", "5 | \n", "6 | \n", "7 | \n", "5 | \n", "3 | \n", "
4 | \n", "5 | \n", "Branson Rivera | \n", "46 | \n", "1 | \n", "6 | \n", "8 | \n", "7 | \n", "7 | \n", "7 | \n", "6 | \n", "... | \n", "3 | \n", "2 | \n", "4 | \n", "1 | \n", "4 | \n", "2 | \n", "4 | \n", "2 | \n", "3 | \n", "3 | \n", "
5 rows × 26 columns
\n", "\n", " | id | \n", "name | \n", "age | \n", "gender | \n", "air_pollution | \n", "alcohol_use | \n", "dust_allergy | \n", "occupational_hazards | \n", "genetic_risk | \n", "chronic_lung_disease | \n", "... | \n", "fatigue | \n", "weight_loss | \n", "shortness_of_breath | \n", "wheezing | \n", "swallowing_difficulty | \n", "clubbing_of_fingernails | \n", "frequent_cold | \n", "dry_cough | \n", "snoring | \n", "level | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | \n", "1 | \n", "Lorenzo Rasmussen | \n", "33 | \n", "1 | \n", "2 | \n", "4 | \n", "5 | \n", "4 | \n", "3 | \n", "2 | \n", "... | \n", "3 | \n", "4 | \n", "2 | \n", "2 | \n", "3 | \n", "1 | \n", "2 | \n", "3 | \n", "4 | \n", "1 | \n", "
1 | \n", "2 | \n", "Zechariah Gallegos | \n", "17 | \n", "1 | \n", "3 | \n", "1 | \n", "5 | \n", "3 | \n", "4 | \n", "2 | \n", "... | \n", "1 | \n", "3 | \n", "7 | \n", "8 | \n", "6 | \n", "2 | \n", "1 | \n", "7 | \n", "2 | \n", "2 | \n", "
2 | \n", "3 | \n", "Lukas Jenkins | \n", "35 | \n", "1 | \n", "4 | \n", "5 | \n", "6 | \n", "5 | \n", "5 | \n", "4 | \n", "... | \n", "8 | \n", "7 | \n", "9 | \n", "2 | \n", "1 | \n", "4 | \n", "6 | \n", "7 | \n", "2 | \n", "3 | \n", "
3 | \n", "4 | \n", "Trey Holden | \n", "37 | \n", "1 | \n", "7 | \n", "7 | \n", "7 | \n", "7 | \n", "6 | \n", "7 | \n", "... | \n", "4 | \n", "2 | \n", "3 | \n", "1 | \n", "4 | \n", "5 | \n", "6 | \n", "7 | \n", "5 | \n", "3 | \n", "
4 | \n", "5 | \n", "Branson Rivera | \n", "46 | \n", "1 | \n", "6 | \n", "8 | \n", "7 | \n", "7 | \n", "7 | \n", "6 | \n", "... | \n", "3 | \n", "2 | \n", "4 | \n", "1 | \n", "4 | \n", "2 | \n", "4 | \n", "2 | \n", "3 | \n", "3 | \n", "
5 rows × 26 columns
\n", "