Upload measurements¶

Now you will post your measurement data and analysis to the database via the API.

You will need to authenticate to the database with your username and password. To make this easy, you can create a file called .env in this folder and complete it with your organization's URL and authentication information as follows:

GDSFACTORY_HUB_API_URL="https://{org}.gdsfactoryhub.com"
GDSFACTORY_HUB_QUERY_URL="https://query.{org}.gdsfactoryhub.com"
GDSFACTORY_HUB_KEY="<your-gdsfactoryplus-api-key>"

import getpass
from pathlib import Path

import pandas as pd
from tqdm.auto import tqdm

import gdsfactoryhub as gfh

New project¶

Let's now create a project.

In normal circumstances, everyone will be sharing and contributing to a project. In this demo, however, we want to keep your project separate from other users for clarity, so we will append your username to the project name. This way you can also safely delete and recreate projects without creating issues for others. If you prefer though, you can change the PROJECT_ID to anything you like. Just be sure to update it in the subsequent notebooks of this tutorial as well.

project_id = f"resistance-{getpass.getuser()}"
client = gfh.create_client_from_env(project_id=project_id)
api = client.api()
query = client.query()

Lets delete the project if it already exists so that you can start fresh.

api.delete_project()

project = api.create_project(
    eda_layout_file="test_chip.gds",
    extraction_rules=[
        gfh.ExtractionRule(cell_name="resistance", include_patterns=["resistance_sheet.*"]),
    ],
)

Upload Design Manifest¶

The design manifest is a CSV file that includes all the cell names, the cell settings, a list of analysis to trigger, and a list of settings for each analysis.

dm = pd.read_csv("design_manifest.csv")
dm

dm = dm.drop(columns=["analysis", "analysis_parameters"])
dm

dm.to_csv("design_manifest_without_analysis.csv", index=False)

api.upload_design_manifest(
    design_attributes_file="design_manifest_without_analysis.csv",
)

api.download_design_manifest(file_path="design_manifest_downloaded.csv");

Upload Wafer Definitions¶

The wafer definition is a JSON file where you can define the wafer names and die names and location for each wafer.

api.upload_wafer_definitions(wafer_definitions_file="wafer_definitions.json")

Upload data¶

You can get all paths which have measurement data within the test path.

data_files = list(Path("wafers").resolve().glob("**/data.parquet"))

You can now upload measurement data.

This is a bare bones example, in a production setting, you can also add validation, logging, and error handling to ensure a smooth operation.

Every measurement you upload will trigger all the analysis that you defined in the design manifest.

for path in (pb := tqdm(data_files)):
    device_id = path.parts[-2]
    wafer_id = path.parts[-4]
    die_x, die_y = path.parts[-3].split("_")
    pb.set_postfix(device_id=device_id, wafer_id=wafer_id, die=f"({die_x},{die_y})")
    with gfh.suppress_api_error():  # useful if you want to keep going if one of the uploads fails.
        device_data = api.add_measurement(
            device_id=device_id,
            data_file=path,
            wafer_id=wafer_id,
            die_x=die_x,
            die_y=die_y,
            plot_spec=gfh.MatplotlibPlotSpec(
                x_name="v",
                y_name="i",
                x_col="v",
                y_col=["i"],
            ),
        )

Querying Device Data¶

Now that the device data is safely on the server, we can query it.

For example, to get all the data belonging to the project:

all_data = query.device_data().eq("die.wafer.project.project_id", project_id).execute().data

Where we included the equality clause:

    eq("die.wafer.project.project_id", project_id)

to only get the device data belonging to the project. This becomes clear when you look at the database diagram:

flowchart LR
    A["project"]
    A --> B["cell"]
    B --> C["device"]
    C --> D["device_data"]
    D --> E["analysis"]
    A --> F["wafer"]
    F --> G["die"]
    G --> D
    H["function"] --> E

Here you see that we worked our way backwards from device data, which includes a reference to the die it belongs to, which in turn includes a reference to the wafer it belongs to and which finally includes a reference to the project it belongs to.

In the case of device data, we could just as well gone the other route:

all_data = query.device_data().eq("device.cell.project.project_id", project_id).execute().data

To get only the data of a device belonging to a certain die, we can query as follows:

first_die = query.dies().eq("wafer.project.project_id", project_id).limit(1).execute().data[0]
data_belonging_to_die = query.device_data().eq("die.pk", first_die["pk"]).execute().data
len(data_belonging_to_die)

JMP table¶

We recommend doing all analysis in python but we also support exporting the data to a flat table for JMP or Excel.

For example, to get a table of all the data belonging to a device:

device_id = query.devices().execute().data[-1]["device_id"]
device_id

infos = query.device_data().eq("device.device_id", device_id).execute().data

dfs = []
for i in tqdm(infos):
    df = api.download_df(i["data_file"]["path"])
    df["device_id"] = i["device"]["device_id"]
    df["cell_id"] = i["device"]["cell"]["cell_id"]
    df["die_x"] = i["die"]["x"]
    df["die_y"] = i["die"]["y"]
    df["wafer_id"] = i["die"]["wafer"]["wafer_id"]
    df["project_id"] = i["die"]["wafer"]["project"]["project_id"]
    dfs.append(df)
df = pd.concat(dfs, axis=0)
df