Smart Shopping Cart







Introduction

Payment methods typically used in retail environments require a lot of work and effort. New progress in deep learning technology has led to the creation of unmanned stores that provide a unique kind of shopping experience. A good example of an unmanned store is Amazon Go. In Amazon Go, there is no need to wait in line to perform a calculation. Because they provide a user experience that significantly reduces computing time, unmanned storage is essential. As a result, interest in this is increasing both nationally and internationally.

Hundreds of cameras, microphones, and pressure sensors follow consumers in real time at "Amazon Go," which is portrayed as an example of an unmanned store, to determine when they come into touch with goods. A consumer enters a store and scans his smartphone on a device placed at the entrance; the system then displays the customer as a 3D target. However, due to issues with AI reading, Amazon Go has a significant flaw that prevents it from accommodating more than 100 customers. A lot of data is gathered because Amazon Go also monitors and analyzes every user's movement patterns. It is therefore challenging to apply it to extensive distribution networks.

Design requirements for smart shopping cart

When designing smart shopping carts, the following requirements were considered.

  • For convenience of shopping, it should be operated with a wireless battery.
  • Only minimal power should be consumed.
  • It must be recognizable even if several products are simultaneously in or out of the cart.
  • It should be separated whether a product is added or removed.
  • Need a way to add items like sugar, flour etc.
  • It must detect all the unusual activities of users.
  • In each add or remove it must ensure only expected items placed inside the cart.

The order of operation of smart shopping cart

Shopping begins when the user connects the shopping cart with his smart-phone. User can login to his account then scan the QR code attached with the cart and connect to a cart.

After the connection is complete,

If the user wants to add an item: 

  • Users can scan the barcode in the item and then they can add that item to the cart.
  • The ultrasonic sensor in the shopping cart recognizes the product when the user puts or pulls out the product and the camera attached to the cart in the Raspberry Pi will capture images.
  • Then, it sends images to the python server over the http network. The python server reads the images from the raspberry pi and then uses deep learning to determine (the type, quantity, and entry or exit of the product) and passes this information to the raspberry pi.
  • Also pi gets item details from the database by the barcode id and it compares the weight change with the expected weight.
  • If weight change is equal to expected weight change  and expected items list received from the python server the app adds product data in the virtual shopping cart.

If the user wants to remove an item:

  • Rpi will detect an interrupt in ultrasonic sensor and if weight is decreased and python server detects that an item is removed then rpi will lead that user to scan the barcode of removed item and if that is similar item removed then app removes product data in virtual shopping cart.

If the user wants to add an item that does not have a fixed shape:

  • Supermarket workers can generate a barcode based on item id and quantity. This will be stored in a temporary collection in the database. Users can scan this barcode and add this item to the cart.
  • System will check weight change  whether it is similar to expected weight change or not and send images to the python server to detect any products added without scanning.
  • If all conditions are satisfied then items will be added to the virtual shopping cart.

To detect unusual activities of users:

Activities we considered:

  • Adding an item without scanning
  • Scan one item (Anchor)  and add another item in the cart (Ratthi).
  • Scan one product and add more than one of it.
  • Scanned item not added to the cart.
  • Hiding an item under another item.
  • Add items that are not selled in the supermarket (Ex: Water bottle, umbrella).

To detect these:

Each time an ultrasonic sensor senses an interrupt,

  • The pi cam will capture images, and this video will be processed by the python server to detect if any item is removed or added without scanning.
  • Controller will check the weight change and compare it with the expected weight.
  • If an item is added without scanning, the cart will be freezed and the buzzer will be turned on.
  • We used two cameras(placed in 90 degrees) to capture images in all angles and cameras will start to capture when it finds an interrupt.
  • If they add items that are not selled in the supermarket system will give an instruction to remove that item.

Based on these requirements we developed a controller (RPI) workflow.

Detailed implementation of smart shopping cart systems

Hardware

Components

Functionalities

Model

Raspberry Pi

Controller

Raspberry Pi 3b+

Ultrasonic sensor

Detects that the product is being put into the cart

HC-SR04

 PiCamera

 Recognizes the product, a raspberry pie that controls the entire system and performs network communication

PiCamera v2

Weight sensor

Measures the weight of the products placed in the shopping cart

Loadcell HX711

Barcode scanner

To detect

Battery

  • This system attaches an ultrasonic sensor to the top of the cart and uses it as a trigger for image streaming. In other words, the image streaming was implemented as an asynchronous transmission that would require the python server to be notified only when the products were put in or taken out of the cart without continuing to be transmitted during shopping hours.
  • Since ultrasonic sensors have a narrow detection range compared to the area of shopping carts, the prototype uses two sensors to compensate for this point to increase accuracy.
  • We used two cameras to
  • Weight sensors were used as a way to determine whether a product enters or goes out of a shopping cart.The weight sensor is installed on the bottom of the shopping cart and the current weight compared to the previous weight to determine whether to increase or decrease.
  • Calibration? Small weight?
  • Circuit diagram

Mobile Application

  • Developed for customers

Technologies  

Frontend

react-native

Backend

nodeJS

Database

MongoDB

Main functionalities:

  • Users must create an account.
  • To connect with the cart they can scan the QR code attached to a cart and start shopping.
  • Shows a list of products that is added to the cart.
  • Make payment

Additional functionalities:

  • Users can make to buy list
  • Offer details will be shown

App is connected to the database via http. Raspberry pi will directly update the items added to the cart in the database.

Frontend                 Backend                 Database                RPI

Security :

  • Hashed passwords will be stored in the database.
  • Access token and Refresh token is used.
  • Email verification system

Web application

  • Can be used by supermarket administrators.

Technologies:

Frontend

reactJS

Backend

nodeJS

Database

MongoDB

Main features:

  • Employees can create an account and login to the application.
  • Add and update  product details.
  • Generate barcode for products like sugar, flour, etc.
  • Insert and update cart details.

Security :

  • Hashed passwords will be stored in the database.
  • Access token and Refresh token is used.
  • Email verification system
  • Admin must verify the account to let employees login to the application.

Python server

In a smart shopping cart, it is a very important task to accurately recognize the products added to the cart. After receiving the image through the camera, the shopping cart should detect the object in the image. The smart cart system  uses a YOLO library (You Only Look Once), which provides grid-based object detection, one of the deep learning technologies.

Python server will receive images from the RPI and it will perform a pre-trained object detection algorithm and send back the detected products to RPI.

Image processing :

YOLO is an open source library that implements not only a classifier but also location algorithms, enabling multiple objects to be detected within one image YOLO is an open-source library that provides better performance than previous object detection neural networks, and is a powerful real-time object detection system that supports real-time detection of videos via webcam as well as file format video data. Because YOLO is based on CNN, it provides a convolution-based architecture that gradually down samples the size of the input image, such as going through the convolutional layer and subsampling

We captured and collected some products and we labeled all the images. Then we trained our model using YOLO library.

Possible scenarios:

  • Item is added without scanning

Ex: Anchor, umbrella

  • One item is scanned different item is added

Ex: Scanned - anchor & added - ratthi

  • One item is added with that another item is added

Ex: Scanned - anchor & added - anchor + ratthi

  • Small items that cannot be detected by weight sensor

Ex: pen, mixture packet

  • Scanned and item is not added to the cart
  • Items like : sugar , with this also all above scenarios possible
  • If put item inside their bag
  • One item is behind another
  • Need to add more than 2 items

Need to be completed

  • Rpi
  •  insert 10 items in the database
  • Generate barcodes
  • Collect images
  • Image processing
  • Ui
  • Document
  • User manual
  • Sensor calibration
  • PCB

User manual :

If user scanned a item they should add that item to the cart before scanning or removing another item to cart

Should not add any unnecessary items : Umbrella, baby

They can connect two carts

Power failure