Back to blog

Semantic Search over iMessage and SMS

A short demo of how to index your personal iMessage with Operand!

Demo

Even though we're only in early beta, it's already possible to build some pretty incredible stuff with Operand. This is the first post of many outlining a mini-project which we've done to help showcase some of our features, and what's possible. More specifically, our goal with these posts are the following:

  • Inspire people to build and make cool stuff using Operand
  • Show off some of the features we've built
  • ... and fix some bugs / dogfood our own features

Background

There's a lot going on these days, and frankly, it gets pretty overwhelming to keep track of all the different messages and emails flying around. One of the reasons why we started Operand in the first place was to help deal and manage this pure "onslaught" of information, and iMessage / SMS seems like a good place to start.

Once configured, this demo will allow you to:

  • Index your iMessage and SMS message history
  • Search over that history, ask it questions, and pull answers from it
  • Quickly find information and content in images, and PDF documents, sent within iMessage conversations

Setup

For full setup instructions, please follow the instructions in the GitHub repository. Specifically, you'll need to:

  • Configure Go (download link)
  • Clone ABCS (used for sending and receiving iMessages via API) as well as the imessage-demo repository (which indexes incoming messages).
  • Setup the required environment variables, specifically:
    • OPERAND_ENDPOINT: Should be https://prod.operand.ai.
    • OPERAND_API_KEY: Your Operand API key, viewable in Dashboard (Settings -> API Keys).
    • OPERAND_PARENT_ID: The ID of the collection you want to put your messages in. Although this is optional, it's recommended to keep your Operand index organized into folders (for example, imessage/).
    • S3_{ENDPOINT,REGION,BUCKET,KEY,SECRET}: A total of 5 environment variables, which allows you to configure which bucket attachments are stored in. If any of these variables aren't specified, then attachments (images, pdfs, etc.) won't be indexed.
  • Run the two applications (in seperate terminal windows):
./abcs -listen=127.0.0.1:11106 -endpoint=http://127.0.0.1:8080
PORT=8080 go run imd.go

Note: It's important that the iMessage demo is running on the same port as specified in the endpoint flag to ABCS. In this case, ABCS will send incoming messages to 127.0.0.1:8080 which will be received by the iMessage demo.

Usage

If everything was done successfully, you should see logs in the terminal whenever you receive a new message (iMessage or SMS). These messages will be automatically uploaded to Operand and indexed, usually in a second or two (depending on the content itself). On the Dashboard, you'll be able to see the data as it's being indexed.

The Operand Dashboard & Object Browser

There are however a few limitations right now:

  • Only indexes incoming messages, not outgoing.
  • Can only index a few content types, namely PNG, JPEG, and PDF.
    • HEICs aren't supported yet, but we're working on supporting them since they're becoming ubiquitous across the Apple ecosystem.
  • Doesn't index the name of the sender, or any other metadata.

We'd happily accept PRs, suggestions and feedback on how to improve this demo, or our systems as a whole :)

As always, feel free to reach out to us via email if you have any questions, or have an itch to build something cool with Operand. We'll support you as much as we can!