Voice Shopping on Non-Amazon E-commerce Sites Using Alexa Shopping Lists

Introduction

Hello, I'm Namabeer, currently studying machine learning as a hobby. This is my first post on Zenn, written as a personal memorandum.
I'm quite lazy and hate doing chores that don't motivate me. If I have time for chores, I'd rather spend it drinking beer.

Motivation

Getting married prompted me to start streamlining housework to increase our free time. As a first step, I introduced voice-controlled devices like Alexa and Switchbot. This made things a bit easier, but after our child was born, I decided to try further reducing housework time.
For example, if I think "We're out of takoyaki sauce" and add it to a shopping list "on the spot", and then it's automatically added to an online shopping cart, it saves the effort of checking the list and manually tapping through an app. It should also reduce the seeds of marital arguments like "Hey, you forgot to buy that!"

Background

While you can shop via voice through Alexa within Amazon, it would be great to do the same on other EC sites, like online supermarkets.
In urban areas supported by Amazon Fresh, you can probably buy daily necessities and groceries via voice (maybe, I don't really know), but where I live in the countryside, Amazon's grocery service isn't available. You might argue that for daily items, I could just use Amazon Subscribe & Save, but I even find it tedious to break down the cardboard boxes. Plus, my house isn't very big, so I don't want to buy in bulk.
Regarding the voice shopping implementation for other EC sites, I considered using IFTTT with Google Tasks (ToDo) as a trigger to run Cloud BOT, but unfortunately, the online supermarket I want to use seemed to be blocking access from Cloud BOT. Consequently, I ended up implementing it locally.

Verification Environment

• OS: macOS Ventura 13.5.2
• Python: 3.11.5
• Selenium: 4.12.0
• chromedriver 117.0.5938.62

Implementation Overview

1. Alexa Settings

"Alexa, add it to my shopping list." This adds the item to the standard Alexa shopping list. If you've done the initial setup, there's nothing specific you need to do.

2. Integration with Google Tasks

Note: Added on 2023/11/20 due to IFTTT specification changes
Using IFTTT, the Alexa shopping list is integrated with Todoist, and then IFTTT is used to link Todoist with Google Tasks (ToDo). This way, I share the shopping list with my wife. While sharing the list on the Alexa side is an option, I find it easier to manage everything within Google, including calendar sharing.
Note: I previously linked the Alexa shopping list and Google Tasks via IFTTT, but Alexa-related triggers were discontinued in November 2023. I found the following information and it was very helpful.

https://x.com/takeaship/status/1723935167407120560?s=20

3. Obtaining Google Tasks API

To call the Google Tasks API, create a GCP project and download the OAuth 2.0 credentials JSON file.

4. Automated Operation with Selenium

I retrieve the Google Tasks shopping list based on the downloaded OAuth credentials file -> use Selenium to log in to the online supermarket and add products that partially match the strings in the retrieved shopping list to the cart. I recorded the Selenium operations using the Chrome extension Selenium IDE and exported the Python code. For detailed logic, I had ChatGPT generate the code to create the .py file.

Please note that code output by Selenium IDE may require optimization or customization.

5. Periodic Execution with Automator × Calendar

I created an app to run the above Python file via shell script execution using the standard Mac app Automator, and scheduled it to run at a specific time every week via the Calendar app.
While I could have used the schedule library in Python, I chose this method because I thought keeping a script running constantly would be a waste of resources.

Implementation Details

Omitting the details of steps 1, 2, 3, and 5 above, I will provide the Python code for step 4.

import pickle
import os.path
from google_auth_oauthlib.flow import InstalledAppFlow
from googleapiclient.discovery import build
from google.auth.transport.requests import Request
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
from dotenv import load_dotenv

# Load environment variables
load_dotenv() # Assumes .env file is in the same directory

# Chrome options settings
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument('window-size=1400,600')
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
chrome_options.add_experimental_option('useAutomationExtension', False)
chrome_options.add_experimental_option('prefs', {
    'credentials_enable_service': False,
    'profile': {'password_manager_enabled': False}
})

# Get authentication info from environment variables
CREDENTIALS_PATH = os.environ['CREDENTIALS_PATH']# Path to the credentials JSON file for Google Tasks API.
TOKEN＿PATH = os.environ['TOKEN_PATH']# File path to cache authentication info
USER_ID = os.environ['USER_ID']# EC site ID
USER_PASS = os.environ['USER_PASS']# EC site password

# Function to get the authenticated Google Tasks service
def get_service():
    SCOPES = ['https://www.googleapis.com/auth/tasks.readonly']
    creds = None
    # Check if saved credentials already exist
    if os.path.exists(TOKEN_PATH):
        with open(TOKEN_PATH, 'rb') as token:
            creds = pickle.load(token)

    # If no valid credentials, perform new authentication
    if not creds or not creds.valid:
        if creds and creds.expired and creds.refresh_token:
            creds.refresh(Request())
        else:
            flow = google_auth_oauthlib.flow.InstalledAppFlow.from_client_secrets_file(CREDENTIALS_PATH, SCOPES)
            creds = flow.run_local_server(port=0)
            # Save new credentials
            with open(TOKEN_PATH, 'wb') as token:
                pickle.dump(creds, token)
    service = build('tasks', 'v1', credentials=creds)
    return service

# Function to get all task titles from Google Tasks
def get_all_task_titles(service):
    tasklists = service.tasklists().list().execute()
    tasklist_id = tasklists['items'][6]['id'] # Specify the Google Tasks ID used for the shopping list
    results = service.tasks().list(tasklist=tasklist_id, maxResults=20).execute()
    tasks = results.get('items', [])
    return [task['title'] for task in tasks] if tasks else []

# Main processing
def main():
    # Get service and task titles
    service = get_service()
    task_titles = get_all_task_titles(service)
    
    # Start browser operation
    driver = webdriver.Chrome(options=chrome_options)
    ### Customize from here depending on the EC site you want to use
    driver.get("url/of/the/ec/site/you/want/to/use")
    WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".c-login-body-ecsite .c-input__input"))).click()
    driver.find_element(By.CSS_SELECTOR, ".c-login-body-ecsite .c-input__input").send_keys(USER_ID)
    driver.find_element(By.CSS_SELECTOR, ".c-card:nth-child(1) .c-pwd-input__input:nth-child(1)").send_keys(USER_PASS)
    driver.find_element(By.CSS_SELECTOR, ".c-login-body-ecsite .c-button").click()
    WebDriverWait(driver, 30).until(EC.element_to_be_clickable((By.LINK_TEXT, "お気に入り"))).click()
    WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.LINK_TEXT, "よく買うリスト"))).click()

    # Add items to the cart based on the shopping list
    for title in task_titles:
        try:
            product_element = driver.find_element(By.XPATH, f"//a[contains(@title, '{title}')]")
            add_to_cart_button = product_element.find_element(By.XPATH, ".//following::button[@title='カゴ追加']")
            add_to_cart_button.click()
            WebDriverWait(driver, 10).until(EC.staleness_of(add_to_cart_button))
        except Exception as e:
            print(f"Error with {title}: {e}")
            continue

    # Log out and close the browser
    driver.find_element(By.LINK_TEXT, "ログアウト").click()
    driver.quit()

if __name__ == '__main__':
    main()

Conclusion

I’ve implemented a system where I can add items to my shopping list the moment I realize they’re needed, and then have the online supermarket handle the shopping somewhat automatically. Since I shop at the online supermarket once a week, I have it set to run on a weekly basis.
Currently, it only adds items to the cart if there's a partial string match, so I'm considering using semantic search for matching in the future.
I’ve had this idea for a while, but never actually got around to implementing it. However, with the advent of ChatGPT, I finally got moving. I'm truly grateful for ChatGPT. Cheers! 🍺