This page outlines the overarching product system, materials needed, build and test procedures, and possible future next steps for the colony counter. The automated colony counter was built out of a need to minimize manual time in counting colonies on petri dishes and uses machine learning to achieve this automated computation. With a finetuned instance of DETR, the team realized a mAP of 0.038 with some overfitting. The physical hardware is relatively simple and consists of a Raspberry Pi to hold the camera and the machine learning model and an Arduino microcontroller to provide human/machine interfacing functionality.

In the team's experience, counting colonies on a petri dish can take up to forty minutes per plate. Colony counting is an essential task for the growth assays the wet lab team requires. In particular, quantification of origin of replication efficacy demands quantifying growth assays through either repeated OD600 or colony count measurements, ideally as a time series. To achieve high throughput in wet lab operations, this bottleneck in measurement must be expedited as much as possible.

Automated colony counting has been made possible through machine learning, as demonstrated by [1]. This task is a specific case of the object detection machine learning problem, made famous through datasets such as COCO [2]. The aforementioned resource used a ResNet-50 based object detection model [1]. Our team plans on improving upon AGAR's work by using the same dataset but a higher capacity model in the form of the DETR vision transformer [ref, DETR]. In keeping with the team's focus on internet of things architectures, any product system the team creates will be modular and compatible with extensions towards wireless/remote operation as well as integration with other digital devices, enabling creators to quickly combine this system with other components and upscale.

The combined product system consists of a Raspberry Pi 5 single board computer with an Arduino microcontroller.

Process diagram

Any full Raspberry Pi board above Pi 3 will suffice, with the caveat that it possess enough computing power for machine learning inference tasks. For demonstrative purposes, a Raspberry Pi 5 will be used in this literature. The Raspberry Pi 5 houses a fine-tuned DETR model, based on [3]. All inference is to be done on the Raspberry Pi 5. The Raspberry Pi 5 possesses a RAM of 8GB. The camera used is the High Quality Camera with a Wide Angle Lens from the original equipment manufacturer. The Arduino microcontroller is connected from the Raspberry Pi with a USB wire. The serial communication protocol is used. The Arduino microcontroller is powered by the Raspberry Pi this way.

Any Arduino with the capability to communicate with the I2C communication protocol will suffice. For demonstrative purposes, an Arduino Uno Wifi Rev2 will be used in this discussion. The Arduino will house all human/machine interaction components so as to shield the Raspberry Pi from electrical disturbances. The Arduino will be equipped with two buttons and an LCD display to provide human/machine interaction capability.

The machine learning model used is DETR . Code from the MedSAM project from the Bowang lab at University of Toronto Mississauga was adapted by Chloe Nguyen Chau, a dry lab team member assisting the hardware team.

For further information on developing the components, circuitry, and machine learning model, in the sidebar, refer to:

• Model for the machine learning model.
• CAD and circuitry for the CAD work and circuitry;

This section mainly deals with integrating the CAD work, circuitry, and the machine learning model, and assumes that all the prerequisites in the aforementioned individual pages are met. The scripts supplied in the aforementioned two pages already enable serial communication between the Raspberry Pi and the Arduino.

When initialized, the device will be in a state ready to accept a Petri dish with colonies.

Unfortunately, we were not able to perform integrative testing with the hardware components and the machine learning model in an actual in-lab setting. Testing was done component-wise, as follows:

Our team obtained a mAP of 0.038 and a validation loss of 1.704 with the latest model generated. 50 epochs of training were used on ~5% of the entire AGAR [1]; dataset to speed up training given limited resources. Qualitatively, there are situations where the model is extremely successful:

There are also situations where the model is extremely underperforming:

And, in the average case, the model has an average performance, producing a colony count that is directionally correct but more often than not undercounting:

This suggests to the team that in the limited time the team had to develop the model, the model is overfitting on the training dataset. In any future step, the team shall attempt to:

• Increase the size of the training and validation datasets;
• Increase the number of epochs for training;
• Increase the degree of variability in the training dataset, such as by rotating images;
• Identify specific pathological cases (such as in the second image in this sequence), determine the root cause behind those cases, and mitigate their impact on training.

The physical hardware component does not need substantial testing as the product system is limited to a single device. Functionality of the physical hardware component, with the finite state machine framework, was trivially shown in a typical testing/debugging test scenario.

Integrative testing was unfortunately not finished by the team, though the plan was to perform testing in two stages:

This testing aims to ensure the model's functionality with the physical hardware component. The model would be loaded onto the script in the Raspberry Pi and the whole product system would be tested. Emphasis is laid on where the physical hardware interacts with the model:

• The transmission of the photograph taken by the camera to the model;
• The transmission of a resulting colony count to the Arduino microcontroller.

This testing aims to ensure the product system's compatibility with the wet lab team's practices. The whole product system would be provided to the wet lab team and any feedback collected. Emphasis would be laid on any points that the hardware team could not provide:

• Performance in a typical laboratory environment
• Ease of use
• Additional features relevant to improving wet lab's workflow, tying into principles discussed in our process engineering improvements.

Modularity is a key feature of our designs and for this specific project, damaged or customizable components may be swapped in or out without any loss to functionality provided they meet all the prerequisites above. The following restrictions apply:

• Most Arduino boards are 5V boards (only the small ones are 3.3V), while all Raspberry Pi boards are 3.3V. Note: Do not connect the Arduino to the Raspberry Pi directly as this will damage the board. Use a USB connector or a logic level converter.
• To integrate more sensors with this product system, integrate them with the Arduino instead of the Raspberry Pi. If you need more I2C pins, use a multiplexer coming out of the SDA/SCK pins. Alternatively, use a wireless connection.
• To link this system to a GUI (graphical user interface), use the Raspberry Pi connect service and run the GUI on the Raspberry Pi computer itself. If the GUI needs to be run on a laptop computer, use a USB connection to the Raspberry Pi or use MQTT. Libraries such as tkinter or PyQT provide an easily accessible means of creating GUI windows on Python or C++.

The colony counter project utilizes a machine learning model, specifically a fine-tuned version of the Detection Transformer (DETR) model published by Nicolas Carion et al. [4], to automate the counting of bacterial or fungal colonies on petri dishes. The team utilized the Annotated Germs for Automated Recognition (AGAR) dataset that contains more than 18000 photos of different bacteria with labels of whether the petri dish was countable, the location of each of the bacteria, and the total numbers of them [1]. The goal of this machine learning model is to eliminate the need for manual counting, streamlining laboratory workflows and improving measurement efficiency, which includes:

1. Using the photo taken to accurately locate the location of each colony like objects
2. Counting the number of such objects
3. Returning to the user the number of colonies available.

The model is currently under construction as the team is still in progress of finalizing the training, and the next step of assessing the model accuracy is to acquire photos from Wet Lab to test it on actual data collected the team.

Manual colony counting on petri dishes is time-consuming and error-prone. The colony counter project seeks to automate this process using machine learning, specifically by applying object detection techniques to colony identification. Leveraging the DETR model, the system is designed to detect bacterial or fungal colonies in petri dish images and provide an accurate count, streamlining laboratory operations and reducing human error.

The machine learning system requires several components:

• Dataset: The AGAR dataset for training, which contains annotated petri dish images. [1]
• Software: The DETR model implemented using PyTorch, as well as standard machine learning libraries for training and inference. The following Python libraries are required:
• torch and torchvision: For building and running the DETR model/code
• matplotlib: For visualizing results
• numpy: For numerical computations
• pandas: For data manipulation
• scikit-learn: For evaluation metrics like mAP
• cython: For compiling C extensions for faster computation
• albumentations: For data augmentation
• transformers: For using transformer models
• timm: PyTorch Image Models for working with pre-trained models
• monai: For medical imaging deep learning, which can be used for image processing and transformations
• imantics: For handling COCO-style annotations
• pycocotools: For working with the COCO dataset format and evaluation metrics

The core component of this project is a fine-tuned DETR model, which is trained to detect and count bacterial colonies on petri dishes. Consult the full scripts here: <link to github script>

• DETR Architecture: The Detection Transformer (DETR) is a transformer-based model for object detection, introduced by Carion et al . DETR consists of two main components: a CNN backbone and a transformer-based encoder-decoder structure. The CNN backbone (ResNet-50) extracts feature maps from the input image, which are then passed through the transformer. The transformer encoder has 6 multiple layers of self-attention and feed-forward layers, allowing it to model global dependencies between image features. The decoder part has a similar structure, using learned object queries to predict bounding boxes and class labels for the detected objects. For this colony counter project, the CNN backbone was frozen during fine-tuning, meaning its weights were not updated to save on computational resources and avoid overfitting on limited data. Only the transformer layers and the final detection heads were fine-tuned to learn the specific task of bacterial colony detection and counting.
• Training Dataset: The model is trained on the AGAR dataset, which contains over 18,000 images of petri dishes, annotated with colony locations and counts. This large dataset provides a robust foundation for training the model to detect colonies in a variety of conditions.

• Current Status: The model is still undergoing training, with early results showing that it can detect colonies with moderate accuracy (for up to 300 colonies per image). Training has been limited to 50 epochs on approximately 5% of the AGAR dataset due to resource constraints.
• Challenges: The model is currently exhibiting signs of overfitting, meaning it performs well on the training set but struggles to generalize to new data. Future improvements will focus on expanding the training dataset and increasing variability to improve generalization.

Once trained, the DETR model will be used to perform inference on new petri dish images. The process involves:

1. Input Image: A high-resolution image of the petri dish is provided to the model.
2. Detection: The model identifies bacterial or fungal colonies in the image, treating each colony as an object.
3. Counting: The total number of detected colonies is counted, and the result is returned to the user or the script at large.

The DETR model is fine-tuned on the AGAR dataset using supervised learning techniques. The dataset provides labeled images with colony locations and counts, allowing the model to learn to detect and count colonies effectively.

• Data Preprocessing: Images from the AGAR dataset are first preprocessed, including resizing to a uniform scale. The preprocessing pipeline includes converting images to a format suitable for the transformer-based DETR architecture, where a fixed-size input is essential for consistent object detection results.
• Data Augmentation: To improve model generalization, data augmentation techniques such as image rotation and scaling are applied to the training set that was taken from the portion of the AGAR dataset utilized. Augmentation ensures that the model can robustly detect colonies in diverse settings and angles, as colony appearance may vary slightly due to experimental setup variations.
• Training Configuration: The fine-tuning of the DETR model is performed with a limited portion of the AGAR dataset. The training process involves optimizing both the detection and counting tasks by leveraging a combination of loss functions:
  • Bounding Box Regression Loss: This helps the model learn the exact location and size of the colonies on the petri dishes.
  • Classification Loss: This component ensures that the model correctly identifies colonies as distinct objects from the background.
• Batching & Gradient Accumulation: During training, batch sizes are configured to fit within the memory limits of the available GPU using ComputeCanada. Gradient accumulation is applied over several batches when the batch size is limited by memory, allowing the model to effectively train on larger datasets despite hardware limitations.
• Loss Function: The model uses a combination of bounding box regression and classification loss to learn the location of the colonies.
• Validation: A portion of the AGAR dataset is set aside for validation to track the model's performance on unseen data during training. Validation metrics like mean Average Precision (mAP) and validation loss are used to evaluate the accuracy of the model in detecting and counting colonies.

• Validation Set: A portion of the AGAR dataset is held out for validation, allowing the team to monitor the model's performance during training. The primary evaluation metric is mAP, which measures the accuracy of the model's object detection capabilities.
• Challenges: Current validation results show a mAP of 0.038 and a validation loss of 1.704, indicating room for improvement in both accuracy and generalization. Increasing the dataset size and improving data variability are key future steps.

The model is currently in the final stages of training. Once training is complete, the following steps will be taken:

• Real-World Testing: The model will be tested on actual petri dish images from our wet lab team to assess its performance on data that is not processed in AGAR but actual data that it could be facing in daily operations.
• Refinement: Based on the test results, the model will be fine-tuned further, with a focus on improving generalization and reducing overfitting.
• Future Improvements: Potential improvements include increasing the training dataset size, applying more advanced augmentation techniques, and addressing specific cases where the model underperforms.

A colony counter stand was built to accommodate the colony counter by supporting the electronics assemblies and provide a consistent environment for pictures of colonies to be taken with. The colony counter stand was fabricated with similar methods to the rack lab manager, including similar 3D printing settings and fastenings. The Arduino and the Raspberry Pi used in the assembly operate as a finite state machine to ease usage.

Introduction

Naturally, where there is a colony counter, there is a need to obtain pictures for the counter. Although this is fulfilled by our use of the camera, there is still a need to take high quality pictures consistently under even lighting, as well as to provide a sufficiently secure support for any electronics. We designed a stand for the colony counter in a similar vein to the <link: lab management rack> to allow secure placement of devices.

• CAD software, prior CAD knowlege, and a 3D printer.
• An Arduino board, and wiring knowledge.
• Some Python knowledge, including use of the pyserial library.

Figure 1. Wiring diagram for the colony counter Arduino.

The user interaction workflow can be summed up with the diagram above. Besides slotting the petri dish into the baseplate, the only user interaction is with the two buttons on the attached breadboard. The first button controls when to activate the DETR model and if to save the model output while the second button controls if the output should be discarded.

CAD, design considerations

To ensure modularity, the stands of this colony counter stand are identical to the lab management solution rack, and use similar fastenings. Refer to our lab management page for details, under CAD and circuitry.

CAD, final drawings

The top plate was specially designed to accommodate multiple pieces of electronics, including the camera and all single board computers. Here is a visualisation. Please consult our lab management page, under CAD and circuitry, for visualisation of the vertical supports used.

As the colony counter stand and the lab management rack were fabricated with similar conditions, consult our lab management page, under CAD and circuitry, for a full description.

This page outlines the overarching product system, materials needed, build and test procedures, and possible future next steps for the rack lab manager. To achieve high throughput, the team needs to be able manage samples as efficiently as possible. Current sample management solutions are inadequate as they lead to missing, misplaced, and mislabeled samples. We designed an internet-of-things augmented rack that allows wireless stock keeping over multiple racks at multiple locations. The product system consists of the racks themselves, RFID readers, graphical user interface (GUI) endpoints, and servers used for communication and synchronous data storage. How such a system would improve wet lab workflows from a process engineering perspective was investigated.

“…samples from 2019 in the 4th floor fridge…” “Not having the date written on tubes” “Taking too long to find the correct 1.5ml tubes”   — From the wet lab team
To increase throughput, the wet lab team must be able to focus on synthesizing and evaluating plasmids, not locating mislabeled samples. Current stock control methods, usually by labeling on the tubes and recording the location of said tube in a public spreadsheet, are inadequate. They differ from person to person introducing variability, omit important details such as time which are not conveniently acquired, and sometimes are outright forgotten. A more accurate form of stock control is required. Required key functions include ability to associate a sample with a potentially long description, automatically timestamp each sample, identify in which building or fridge a sample is, and enable a stock accounting system for samples.
The hardware team has used industry as an inspiration and designed a rack for storing samples on tubes. Said racks will have electronics for identifying the presence of a sample in rack wells, as well as other functionality to enable stock accounting. This is in line with industry solutions, where industrial racking is supplied with barcodes to enable stocks of goods to be tracked on a database. To address the need for a system that is functional across multiple locations, internet of things techniques were used. This is featured in the use of wireless microcontrollers, databases, and communication servers. Ergonomics and modularity were also considerations addressed in this product system’s architecture.

The combined product system consists of a rack with IR (infrared) proximity sensors, LEDs, and a wireless-enabled microcontroller (Arduino Uno Wifi Rev2), a RFID reader assembly with a wireless-enabled microcontroller (Arduino MKR Wifi 1010), an MQTT broker, a MongoDB database server, and a GUI endpoint. This architecture is adapted from [1], which was originally intended for use in self-driving labs.
Overall product system

For further information on developing the components, circuitry, microcontroller scripts, and GUI endpoint, in the sidebar refer to:

• CAD and circuitry for the CAD work, including stl file visualisations and circuit diagrams;
• Microcontroller Programming for the microcontroller scripts, including circuit diagrams;
• Graphical User Interface Design for the GUI endpoint.

This section mainly deals with integrating the CAD, microcontroller circuitry, and GUI endpoint, and assumes that all the prerequisites in the aforementioned individual pages are met. The scripts supplied in the aforementioned three pages already enable wireless communication between each other through the MQTT broker. However, ensure that:

• All devices are connected to WiFi, and for the microcontrollers, use the correct connection type (WiFi.connect or WiFi.connectEnterprise);
• All devices are connected to the same MQTT broker and the same MongoDB database;
• All RFID chips are numbered uniquely. To do this, write a unique number to the RFID chip at block 0, sector 2, 0th byte.

When initialized, the device will be in a state ready to have samples keyed in.

Unfortunately, we were not able to perform complete integrative testing with the entire product system and in an actual in-lab setting. Testing was done component-wise, and some integrative testing was completed, as follows. End user testing was never completed but planned.
Component-wise testing
Component-wise testing was done via a black box model. Each component was tested for compliance with the protocols. Other components were simulated mainly through their MQTT and database interactions. A workflow of testing is as follows:

1. Identify the device’s specific role and task in the protocol, what communication (usually MQTT) input triggers the task, and the expected outputs of the task.
2. Simulate the MQTT input via a Python script.
3. Eavesdrop on all outgoing MQTT messages via a Python script.
4. Depending on whether the component meets the intended behaviour, correct any scripts or circuitry involved, add traces in any scripts, and repeat.

Two relevant scripts for component-wise testing are the mqtt_publisher.py and the mqtt_listener.py, which allow arbitrary MQTT messages to be sent and eavesdropped on.
Component-wise testing was done over multiple levels, starting with the microcontroller only and gradually adding additional peripheral hardware such as sensors. Gradually, in our integrative testing stage, simulated inputs were replaced with other actual components.

This testing aims to ensure the product system’s compatibility with the wet lab team’s practices. The whole product system would be provided to the wet lab team and any feedback collected. Emphasis would be laid on any points that the hardware team could not provide:

• Performance in a typical laboratory environment
• Ease of use
• Additional features relevant to improving wet lab’s workflow, tying into principles discussed in our process engineering improvements.

As this is a process engineering project, process engineering metrics will be used to measure the efficacy of the system, and identify actionable items moving forward:

• Time needed to learn the system
• Frequency of deviations made from intended workflows
• Time saved or spent using the system relative to the previous state, where sample details were marked on the tube and potentially lost.

Modularity is a key feature of our designs and for this specific project, damaged or customizable components may be swapped in or out without any loss to functionality provided they meet all the prerequisites above. The following restrictions apply:

• All Arduino Unos have 5V logic. All Arduino Nanos and MKRs have 3.3V logic. The Arduino MKR was used because the RFID reader uses 3.3V logic. Do not supply 5V to a component that uses 3.3V logic.
• Each IR sensor will require an analog pin, and there are usually only six on an Arduino microcontroller. If more are required, use a multiplexer. In addition, digital pins can be used to receive analog inputs, if an analog-to-digital converter is used in between the inputs and the pin.
• Although the team used a third-party database and MQTT broker, it is possible to implement these on a local network. A Raspberry Pi can be used to serve a database with an API as well as an MQTT broker.

The goal of this Lab Management project is to streamline lab operations by creating a test tube rack that makes it easier to locate and identify vials. A main component of this project is the rack which hold the vials. It consists of four components: a top rack, bottom rack, and two vertical supports, all designed using SOLIDWORKS and 3D printed. Key design features include proximity sensors integrated into the rack to detect tube placement and an LED system to visually assist in vial identification. The design is made of lightweight and durable PLA material, as well as screw-based assembly to ensure structural stability. A significant innovation in this design is the RFID sleeve which is slipped onto the vial. This is to enhance the longevity of RFID chips used for tracking the content of the Eppendorf tubes. The sleeve, fabricated using crocheting, ensures the chip can be reused across multiple tubes, thereby reducing waste. Throughout the design process, the team utilized rapid prototyping, balancing precision and durability with time and cost constraints. This resulted in a rack integrated with electronics which not only improves lab efficiency but also incorporates modular design choices.

The primary hardware design solution to optimize lab workflow was to make a test tube rack that locates and identifies where specific vials. The physical test tube rack had various design considerations to effectively reduce the time taken to search for test tubes in the fridge. Here, we discuss key design decisions we took while designing the rack. The rack has 4 components, the top rack, the bottom rack, and 2 vertical supports. The majority of the rack was designed using SOLIDWORKS (CAD software).

Furthermore, an RFID sleeve is required to increase the lifespan of the RFID Chip. Its primary purpose is to ensure that the RFID chip is always attached to the Eppendorf tube. The sleeve design is also used so that the chips can be reused for multiple tubes. This was fabricated using crocheting.

• CAD software, prior CAD knowlege, and a 3D printer.
• A WiFi Arduino board, and wiring knowledge.
• Yarn for the RFID sleeve, basic crocheting skills, and a crocheting hook

Top rack

The top rack is where all of the Eppendorf tubes will be placed. It has a thickness of 5 mm to save space and reduce printing costs. We have decided to make the slots as holes that go straight through the rack rather than an indent where the tube is rested on a solid platform. This decision was made so that the tube could dangle directly over the top of the proximity sensors on the bottom rack, within its detection range. This way, the proximity sensor can detect if the slot is filled or not.

Taking into consideration the threshold distance, we had to design the plate so that the Eppendorf tube would dangle directly on top of the proximity sensor to communicate with the Arduino and the GUI that the slot is filled so that the user knows where each specific tube is located. We have designed the rack, using the dimensions of a 1.5 mL Eppendorf tube. Out of the 40 mm of the length of the tube, around 15 mm of the tube will be dangling past the top rack. This places the tube within the detection threshold of the rack, allowing it to be detected by the sensor.

Here is a visualisation of our top plate.

Individual well design

Figure 1. Note the portion of tube dangling beneath the top plate (purple line).

We were able to get 15 mm of the tube dangling through the design of the slot itself. The slot is made of 2 holes, one hole with a constant diameter of 11 mm directly on top of a tapered hole that reduces the diameter from 11 mm to 9.94 mm. This shape for the slot mimics the shape of the Eppendorf tube so that it would stop it from falling through the rack. The ideal diameters were determined through testing, where before, the diameter at the bottom of the tapered hole was too small, which led to less of the tube dangling, and thus, it not reaching the threshold distance of the proximity sensor. After 3D printing our original design, we decided to make the diameters slightly larger, so that the tube is well within the threshold.

Furthermore, beside each slot on the top rack, there is a pocket for an LED to be placed. We decided to incorporate LEDs into the rack to make it easier to identify which slot the user should be placing their tubes into or which slot a specific tube is placed in for them to retrieve it. We integrated LEDs onto the top rack by having a circular pocket with a diameter of 5 mm and 2 small holes at the bottom with a diameter of 1 mm for the pins of the LEDs. There are grooves that run on the bottom of the top rack from the LED pocket to the side of the rack to organize and hold the wires in place. This keeps the wiring clean and avoids having loose wires, as that could cause safety hazards.

Vertical supports

Figure 2. Note the nut and screw assembly in the pink rectangle. The screwhead fits flush into the blue plate, and the voids in the green rectangles, which are grooves meant for housing wires.

The rack incorporates two side supports which provide structural integrity. Side supports are designed to be modular, allowing reuse between different projects.

The wires of the LED run to the side of the top rack, where the vertical supports are attached. The vertical support connects the top rack to the bottom rack. The vertical allowed there to be a gap large enough between the top and the bottom plate so that the proximity sensor does not detect the top rack as discussed before. There are also grooves that run from the top to the bottom of the vertical support for the LED wires. This is so that the LEDs can be connected to the breadboard on the bottom rack, and to avoid loose wires. This helped us easily integrate electronics onto the top rack.

The vertical support held together the top and the bottom racks using M4 and M2.5 screws. These screws were used since they were easily accessible and fit the best within the dimensions of the rack to avoid taking up unnecessary space. Attaching the components using screws and nuts is more reliable and stable as the 3D printing that we have access to is not very accurate. As a result, we didn't use less secure methods such as dovetails to attach the components.

Here is a visualisation of our vertical support plates.

Securing sensors

Figure 3. Note how the IR proximity sensors sit on top of earring attachments in the green rectangles, and are fastened by the stud in the red rectangle.

To secure the proximity sensors to the bottom rack, plastic earrings are used instead of screws. These earrings can be glued onto the rack, with the post sticking up, since they are also made of plastic, ensuring adhesive compatibility. The earrings hold the sensors securely by pacing the screw holes of the proximity sensor onto the earring posts and pressing the backing against the rack with the proximity sensor being sandwiched in between, which creates a stable connection. This method is more economical and practical compared to using screws, particularly due to the small size of the proximity sensor screw holes, which would make using screws more difficult to find, more expensive and harder to set into.

The rack is made from polylactic acid (PLA) which is a thermoplastic monomer derived from renewable, organic sources [6]. It is lightweight, durable, easy to clean as well as easily accessible as it is wildley used in 3D printing . When properly printed it is not porous and it can be sanitized. This makes it suitable for a lab, where durability and hygiene are critical.

Here is a visualisation of our bottom plate, where the plastic earrings and the sensors are mounted on.

RFID sleeve

In addition to the aforementioned rack, we also developed an RFID sleeve to attach the RFID to the Eppendorf tube. After multiple design iterations, we decided to settle on a sleeve design where users can easily insert and remove the RFID from a tube as it is being used. This way, we can increase the lifecycle of the RFID chips.

Figure 4. Above is a finished crocheted RFID sleeve.

Fabrication

3D printing, instead of wood/metal material fabrication, was used to minimize turnaround time. By considering the operating environment, PETG would be used for the final design since it is less brittle than other polymers that were commonly used in 3D printing, rendering it more secure in extremely low temperatures. However, PLA was used for prototypes since printing time with PLA is significantly faster than PETG. The percentage in-fill for 3D printing was 15%, which has the balance between optimal print time and the prints' strength, considering that the test tube rack would not experience high loads.

Despite that the precise dimensions of the test tubes were provided for the top rack, the team considered that the accuracy of 3D-printed parts would lead to a tight fit for test tubes. +0.5 mm of clearance was added to the CAD model which turned out to be a great fit. At the prototyping stage, only one hole was 3D printed to minimize the printing time. The bottom rack has multiple holes to secure the sensors in place with plastic earrings and super glue. The clearance of the holes was also +0.5mm of the earring's diameter for a tight fit.

To assemble the top and bottom racks together with the vertical members, the team decided to use M5 screws and nuts to secure the components in place. For the top rack, the screws were inserted from the top for easier assembly experience. The bottom rack would have the screws inserted from below for the same reason. Since the test tube rack had to stand by itself, a counterbore was included in the design to hide the screwhead to avoid it being wobbly. Washers were used for assembly as well to ensure the screws do not go through the holes.

Figure 5. Above is a detail of the counterbore, in which the screwhead and washer fit in.

Wiring

Wires are assembled as shown in the picture. The rack is not depicted; the proximity sensors go into the rack via the earring slot mechanism described in the Product System section.

Figure 6. Sample wiring diagram for the rack. The wires for the proximity sensor already come in the colour red, black, and white which are for the power (5V), ground, and signal respectively. Each of these wires is soldered to one with a male end to be able to easily connect it to either the breadboard or the Arduino. The power and the ground is distributed to all the sensors with the use of the breadboard. The grey and orange wires are male to male wires to and they connect the 5V and ground from the Arduino to the board, while the white signal wires from the sensors are instead directly connected to the Arduino board.

RFID Sleeve

To fabricate the sleeve, the following crocheting steps were done:

• Chain 6 and close it off into a circle using a slip knot. This is to make the ring that will go around the Eppendorf tube.
• Chain 6 and close it off. This is just a straight line of crochet. This will be used to attach the rind around the tube to the actual sleeve.
• Make a magic circle using 6 chains. Use a combination of half-double crochet and double crochet to make a circle with a diameter slightly larger than the RFID chip (25 mm). Make 2 circles.
• Attach the 2 circles to make the sleeve. When attaching the circle, make sure that the RFID chip is in the middle.
• Attach the sleeve to the straight line of the crochet.
• Attach the straight line of crochet to the ring.

In the context of the lab management system, microcontrollers are important because they detect critical information and act as communication intermediary between user interfaces and cloud databases. For our solution, two microcontrollers were set up, one to monitor the rack on which samples were placed, and the other for identifying and updating RFID information that each sample is associated with. Communication between the two microcontrollers and a user interface is done through MQTT (Message Queuing Telemetry Transport) in publisher-subscriber relationships through string or JSON (JavaScript Object Notation) format. Communication between microcontrollers and cloud databases is achieved via HTTP (Hypertext Transfer Protocol Secure).

As our first step, all possible user interactions with the management system are broken down into five processes, namely, startup, registration, checking-in, checking-out, and deregistration. Each process has its own operations, payload, and MQTT topic, all of which are agreed upon between microcontroller and UI developers via a contract. Process workflows, operations, and fields of required data are detailed in the protocol document. Next, the hardware team set up the MongoDB cloud database, translated protocols into microcontroller code, and established wireless communication between various parties of the management system. Main expertise required from the hardware team includes Arduino coding, SQL (Structured Query Language), cURL commands, MQTT and HTTP communication protocols.

The lab management system contains several key functionalities such as fridge monitoring, recording of user interactions, and data storage, which necessitates the introduction of several components such as microcontrollers and online databases. All these components must have wireless communication enabled for an integrated system, and it is based on this that the hardware team selected the components. We used Arduino Rev 2 as the microcontroller, MongoDB as the cloud database, and MQTT as the communication protocol. Another issue the hardware team discovered was the importance of attaching an identifier to each sample, which is why RFID is selected to differentiate the samples. Therefore, the solution we devised for the fridge management system would be to have user commands interacting with microcontrollers and other components (proximity sensors and RFID reader), who then sends information to the cloud database using wireless communication.

• Coding experience in C++
• 2x Arduino microcontroller with wifi capabilities (Arduino uno rev2)
• Libraries: Arduino, WiFiNINA, ArduinoHttpClient, ArduinoJson, Wire, RTClib, ArduinoMqttClient, MFRC522

The product system can be divided up into the hardware and software components.

 Hardware component

• Rack Arduino: the microcontroller connected to proximity sensors on the rack responsible for tracking rack occupancy after user gives in check-in or check-out commands. Communicate with the cloud database and update occupancy field (binary variable: true/ false)
• RFID Arduino: the microcontroller responsible for the identification of RFID and by extension the sample information. User interactions prime the RFID reader to read. Communicate with the cloud database and update information about RFID tap time (date objects).
• Proximity Sensors: sensors that detect occupancy on the rack according to a calibrated formula. When there is a change in the distance detected, occupancy information is updated
• RFID Reader: reads RFID number of each sample. In our product system, each RFID gets designated with specific numbers, and the designated RFID number is sent to the rack arduino or back to user interface.

 Software component

• MongoDB database: a MongoDB atlas database was created, and there were two clusters in the database named “Rack Well” and “RFID Chips”. The fields for each of the two clusters are presented in a table form down below.

RFID Chips

• MQTT broker: using HiveMQ, a cluster was created to hold messages across different topics. This included login credentials for the ability to publish and subscribe to different topics.
• Arduino programming: the code for arduino programming was structured such that each user interaction such as registration, checking-in, RFID tap, proximity sensor detection are coded in individual functions. Depending on how each process update the cloud database, specific payloads for different actions were set up inside those individual functions. MQTT is used as the communication protocol between Arduinos and cloud database, so specific topics were created for publication and subscription. In the callback functions, the individual functions written for different process were executed depending on the payload. For instance, in the arduino/rfid topic, if the python user payload is “REGISTRATION”, the function regcheckin_putsample is ran in the arduino code. Reference the contract established for more details.

1. Set up the Arduino’s connections to the wifi network of your choice.

First, establish the login details to the wifi network.

        // name of the wifi network
        const char *ssid = "";
        // the username is not always required, but if the network asks for one,
        // include it here
        const char *username = "";
        const char *password = "";
      

Second, depending on the nature of the wifi network, connect to the using your login credentials like Wifi.begin(ssid, password) or Wifi.beginEnterprise(ssid, username, password). If the Arduino connects successfully, it will print “You're connected to the network”. Otherwise, the program will continue printing “.”

        void setup(){
          Serial.begin(9600);
          
          // messages to the user 
          Serial.print("Attempting to connect to WPA SSID: ");
          Serial.println(ssid);
          
          // begin process of connecting to wifi
          while(WiFi.begin(ssid, password) != WL_CONNECTED){
          //while(WiFi.beginEnterprise(ssid, username, password) != WL_CONNECTED){
            delay(5000)
            Serial.print(".");
          }
          Serial.println("You're connected to the network");
        }
      

2. Set up the MQTT server and MongoDB database with the proper credentials.

For the MQTT server:

Note the cluster url and port that is generated with your cluster. Also write down the username and password that you chose for your MQTT server. You will be using these credentials to connect to the MQTT server with both of the Arduino microcontrollers to both publish to and subscribe to topics.

For the MongoDB database:

Set up the MongoDB Atlas database project, create database (igemhardware) and collections within the database (RFID Chips & Rack Well). Set up credentials such as data API key name and private key, user access, network access, and URL endpoint. Set up the collections (i.e creating fields with empty values) via a Python script HTTPS communication.

3. Establishing basic pub/sub functionality with the Arduino microcontrollers.

This is where the Arduinos can receive specific commands from the GUI endpoint or send over the data to be generated with the sensors + readers connected to them.

Again, remember to initialise the variables below so that we can call the relevant mqtt client functions, connecting the arduino to the mqtt server.

// initialise values
      // enter the cluster url + port you noted down earlier here
      const char* mqtt_server = "";
      int port = 443;
      
      // enter the username and password credentials you generated earlier
      const char *mqtt_username = "";
      const char *mqtt_password = "";
      

In the setup function, use mqttClient.setUsernamePassword to send your login credentials to the mqtt server and mqttClient.connect to connect to the mqtt server. mqttClient.subscribe allows the arduino to receive the topics that were called with this function. Arduinos will not receive messages from topics outside of these subscriptions.

void setup(){
        mqttClient.setUsernamePassword(mqtt_username, mqtt_password);
        
        // keep trying to connect to mqtt server upon failure
        if (!mqttClient.connect(mqtt_server, 8883)){
          Serial.print("MQTT connection failed! Error code = ");
          Serial.println(mqttClient.connectError());
      
          while (1);
        }
        
        // where mqttClient.subscribe(topic name, QoS)
        mqttClient.subscribe("arduino/rack/registration", 1);
        mqttClient.subscribe("arduino/rack/checkin", 1);
        mqttClient.subscribe("arduino/checkout", 1);
      }

However, in order to process the payloads that are sent through each topic, we need to create a function called “read_mqtt” to print out the received message in the serial monitor and process it in the later to be explained callback function (step 6) as a string.

String read_mqtt() {
        Serial.print("Message arrived on topic: ");
        // retrieve topic of message that arduino was subscribed to
        Serial.print(mqttClient.messageTopic());
        String payload = "";
      
        // because the payload is received as an array, we need to convert 
        // each character to a String and append it to payload
        Serial.print(". Message: ");
        while (mqttClient.available()){
          payload += String((char)mqttClient.read());
        }
        Serial.print(payload);
        return payload;
      }

Now that we’ve established functionality for receiving and reading messages from the mqtt server, we need to establish the sending functionality. We can do this through making a function called mqttSend using the following code.

void mqttSend(String topic, String payload){
        mqttClient.beginMessage(topic);
        mqttClient.print(payload);
        mqttClient.endMessage();
      }

4. Interfacing Arduino connections to proximity sensors, the RTC clock, and RFID reader.

Proximity sensors:

Code from [7] was adapted for use in reading IR sensor distance readouts, and ultimately, the presence of a vial. This function updates the array “distance” every time it runs.

const int sensorPin[] = {A0, A1, A2, A3, A4, A5};
      float distance[6];
      float old_distance[6];
      const int AVERAGE_OF =50;
      const float MCU_VOLTAGE = 5.0;
      
      int readDistance(int sensor){
        float voltage_temp_average=0;
            
        for(int i=0; i < AVERAGE_OF; i++)
        {
          int sensorValue = analogRead(sensorPin[sensor]);
          delay(1);      
          voltage_temp_average +=sensorValue * (MCU_VOLTAGE / 1023.0);
          }
          voltage_temp_average /= AVERAGE_OF;
      
        // eqution of the fitting curve
        ////33.9 + -69.5x + 62.3x^2 + -25.4x^3 + 3.83x^4
        distance[sensor] = 33.9 + -69.5*(voltage_temp_average) + 62.3*pow(voltage_temp_average,2) + -25.4*pow(voltage_temp_average,3) + 3.83*pow(voltage_temp_average,4);
      

// find threshold for distance changes..
        if (abs(old_distance[sensor] - distance[sensor]) > 2){
          // Currently the threshold for movement detection is 2cm
          // send value to rack arduino code to indicate change
            Serial.println("CHANGE DETECTED");
            old_distance[sensor] = distance[sensor];
            return 0;
        }
        old_distance[sensor] = distance[sensor];
        return 1;
      }

RTC clock:

An RTC clock is set up with the RFID Arduino such that every time the user taps RFID, date information is recorded. The RTC clock used in this project is the MKR WiFi 1010. When setting up the Arduino, calibrate the datetime object. The code for the RTC clock can be found below.

void setup() {
        Serial.begin(9600);
      
        delay(5000);
      
        rtc.begin();
        dateparser(cheese);
        rtc.setDate(cheese[2], cheese[1], cheese[0]);
        rtc.setTime(cheese[5], cheese[4], cheese[3]);
      }

RFID reader: The MFRC522 RFID reader is used in our product system. When the reader receives the instruction, it reads the RFID. The code for RFID can be found below.

int readRFID (byte arrayAddress[]) {
        /*
        Reads the identification number (as we determine it) of the RFID.
        All identification numbers are stored in block 2, zeroth member.
        */
        const int block_number = 2;
        const int trailer_block = 3;
      
        while (!mfrc522.PICC_IsNewCardPresent()){}
        while (!mfrc522.PICC_ReadCardSerial()){}
      
        byte status = mfrc522.PCD_Authenticate(MFRC522::PICC_CMD_MF_AUTH_KEY_A, 3, &key, &(mfrc522.uid));
        if (status != MFRC522::STATUS_OK){
          return -1;
        }
      
        byte buffer_size = 20;
        status = mfrc522.MIFARE_Read(2, arrayAddress, &buffer_size);
        if (status != MFRC522::STATUS_OK){
          return -1;
        }
      
        mfrc522.PICC_HaltA();
        mfrc522.PCD_StopCrypto1();
      
        return (int) arrayAddress[0];
      }

5. Programming the registration, check-in, check-out, and de-registration functionalities.

Based on the workflows developed for fridge management, the payloads generated for different processes are different. The payload are parsed into the ejson format in the Arduino and sent to the Python user interface. In creation of these payloads, the cURL command is used. In a curl command, information such as content type, api-key are specified as headings, and the actual text is written in a dictionary before the whole payload gets converted to JSONText.

String registration_RFIDtap(int RFID){
        int milliseconds = 0;
        char curTime[30];
      
        snprintf(curTime, sizeof(curTime), "%04d-%02d-%02dT%02d:%02d:%02d.%03d+00:00",
         rtc.getYear(), rtc.getMonth(), rtc.getDay(), rtc.getHours(),
         rtc.getMinutes(), rtc.getSeconds(), milliseconds);
      
        DynamicJsonDocument payload_db(1024);
        payload_db["filter"]["RFID no"] = RFID;
        payload_db["update"]["$set"]["In Use?"] = true;
        payload_db["update"]["$set"]["Last tapped time"]["$date"] = curTime;
        payload_db["update"]["$set"]["Registration time"]["$date"] = curTime;
      
        String JSONText;
        size_t JSONlength = serializeJson(payload_db, JSONText);
        Serial.println("[" + JSONText + "]");
      
        mqtt_parseSend("python/dbupdate/RFID Chips", "[" + JSONText + "]");
        return JSONText;
      }

The workflow specifies what fields in the clusters would change based on the user actions. For instance, whenever an RFID is tapped, time information across the database is updated. Occupancy information only changes in the check-in and check-out process in conjunction with proximity sensor detection.

These functions containing payloads are called in the callback functions. To differentiate between different user actions, the payload from the user interface to Arduino differs depending on the subscribed topic and payload. The conditionals constructed in the payback ensure that the appropriate database payload is sent back to the user with each action.

6. Hardware Assembly

The wiring connections, including the connections between the microcontrollers, RFID reader, and RTC clock, are shown in the wiring diagram under CAD and circuitry (See sidebar).

7. Testing Procedures

After constructing the hardware and software components of the lab management system, the hardware team underwent testing for the system for the following steps. Since the system is very integrated, testing is challenging to perform without the entirety of the system constructed. Therefore, the hardware team approached testing through isolating key components and functionalities, such as MQTT communication, database queries, etc.

1. MQTT Communication

The MQTT communication for both the rack and RFID arduinos are done by creating two python scripts named publisher and listener. The publisher publishes messages which get shown on the terminal for listener. Depending on the specific topic and payload, the rack arduino and RFID arduinos execute actions and send the payload back over to the Python user interface. If the listener prints out the payload sent by arduino, the MQTT communication is functional. Otherwise, tracing is performed to figure out the step at which the program fails its execution.

2. RFID Reader

The main functionality to be tested in the RFID reader is its ability to read RFID values after being instructed to read. The RFID values are printed on the serial terminal of Arduino, and when Arduino serial provides no response, the RFID reader is not working properly.

3. RTC Clock

The two main functionalities tested with the RTC reader is its correct recording of time information and whether the database recognizes the date information as a proper date object. The first functionality is achieved by calibrating the RTC clock upon setup or whenever the clock loses power. As for date object recognition, the payload to the database is stored as an ejson file in contrast to json file normally. Additionally, during the testing process, the hardware team transitioned from using the DS3231 external RTC clock to Arduino Uno R3. During the testing process, we made adjustments to the Arduino code to format the year of the date object as four digit string instead of two since we realized in the process that the RTC clock only records “24” in contrast to “2024”.

The graphics user interface allows the researcher to access and update inventory data stored in a cloud Mongo database. It also serves to communicate with associated arduino devices, and compile information received over MQTT. The GUI is structured by classes for each window, which each have methods for registration, deregistration, check-in, and check-out operations. A callback function accesses the necessary classes and makes updates upon receiving communications from MongoDB or MQTT.

Scientists working in biological and chemical laboratories often work with hundreds of small vials of samples. When storing them for later use, it is critical to keep track of their identities. If there is a long time between successive uses of the vials, relying on memory and penned notes is unreliable. Our inventory management software creates a streamlined process for researchers to record notes and history for their samples. Accessing and searching inventory data on a computer interface can boost researcher productivity and streamline workflows, by removing uncertainty and confusion about the samples that are in use.

The graphics user interface (GUI) for our inventory management software is designed with reliability and convenience as the highest priorities. The inventory management system will only work if its protocols are followed; therefore, the system has been designed to be minimally disruptive to the researcher’s workflow. The result is a system that saves time and boosts collaboration between researchers, democratizing information about samples and safeguarding against mistakes in picking samples.

Competence with programming with a GUI library, use of MQTT protocols, and Pymongo HTTPS communication is required for this system. Understanding of multithreading and avoiding race conditions is essential for MQTT communication with other devices.

The GUI is programmed in Python using the Tk library. The GUI maintains a local database that is updated based on communication with the cloud database and Arduino devices, as well as through the user interface itself. The local database is used to identify discrepancies between the vial that the user indicates they want to select, and the vial that they scan with the RFID reader. The codebase for the GUI is structured by classes for each window, and objects being created and deleted as vials are registered and deregistered.

A MongoDB database is updated periodically by the GUI, and the GUI periodically refreshes its own local database with information from the cloud. The current architecture assumes that a single GUI endpoint is used in combination with a set of racks; conflicting database updates from multiple GUI endpoints should not occur.

The GUI communicates with the other arduino devices on the network (RFID scanner, vial rack) with the MQTT protocol and uses multithreading to handle communications across the network and with the cloud database. Issues with race conditions in multithreading are avoided by using a queue based system, with queues from MQTT and database communications being handled by callback functions for each.

A storyboard for the GUI in addition to the desired functionality was planned at the beginning. Based on the requirements and functionality, pseudocode and a skeleton of the architecture was written.

1. Overall Code Architecture

The GUI is organized hierarchically; each window is a Toplevel that is a child to a parent Toplevel, with the main Tk() object being the root. Termination of the main Tk() object exits the program. The inventory window and the popup windows associated with each of the registration, deregistration, check-in, and check-out processes are classes that are invoked by button presses. The constructor for each window class draws all elements within the window; if elements need to be drawn or added after the window is constructed, the callback can change the configuration of existing widgets, delete and redraw existing widgets, or some other combination. This program was coded in a manner that avoids deleting and redrawing widgets, as this can result in significant added computational overhead.

The multithreading occurs within a Threaded Client class; the client class constructs the visual elements of the GUI as well as starts the threads for communicating on MQTT and with MongoDB. The messages received from those communication threads are passed into a queue and handled by callback functions in the threads as well as in the main Tk code.

class ThreadedClient:
        def __init__(self, master: Tk):
            self.master = master
            self.db_queue = queue.Queue()
            self.mqtt_db_queue = queue.Queue()
            
            self.mongo_client = MongoClient(DB_URL, server_api=ServerApi('1'))
            self.db = self.mongo_client["Wetlab-Fridge-Management"]
            
            # Open to the Inventory Screen, pass db and mqtt threads to Tk screen
            self.gui = HomeScreen(master, self.db_queue, self.mqtt_db_queue, self.pub_client, self.mongo_client, self)
            
            self.db_thread = th.Thread(target=self.db_worker_thread)
            self.mqtt_thread = th.Thread(target=self.mqtt_worker_thread)
            self.db_thread.start()
            self.mqtt_thread.start()

The above code is a snippet which shows the threaded client initializing the various queue objects, the main GUI screen, and the multithreading components.

2. Networking

The classes for each GUI window were tested in scratch files before being added to the main file; communication between different GUI windows was managed by passing references to other windows as required.

A contract for interdevice communication (especially MQTT) was written to coordinate development of codebases for the GUI and the Arduino scripts. Strong communication across the team about how the contract would work was critical; all aspects must interoperate smoothly, including the formatting and structure of messages sent between devices.

MQTT communication occurs by passing strings between devices. The GUI is written to parse the data according to the contract parameters. Based on the topic assigned to the MQTT communication, the GUI will initiate a callback from the MQTT thread. This callback then initiates a callback within the main Tk loop, which updates the screens as necessary; the current state is managed by an activeWindow attribute, which will modify how the callback function runs. The active window depends on whether registration, deregistration, check-in, and check-out

3. Functionality

The GUI verifies whether the scanned vial and location is a valid pair for registration, deregistration, checkin and checkout. It will prompt the user based on whether the operation can proceed. The GUI takes the event time, the researcher (user) using the GUI, as well as other operation information (e.g., user entered description about the vial) and compile into an event history which is published to the MongoDDB database. Other functionality includes search and lookup of vial histories, and the ability to toggle between different fridges and racks.

4. Testing

Extensive testing of networking capabilities and database update operations took place while the GUI was being built. This was important, as writing the GUI code was an extensive process and it was important that progress was made on other functionalities while the GUI was being built. MQTT multi-device communication was simulated for testing using python scripts run on the same device, as well as physical testing in-person with the arduinos. MongoDB testing was done on a test database cluster; the code was transitioned to update the true database as we concluded testing and moved onto implementation in the lab.