10 October 2021

Bike components global shortage and a hacking session

For the last year, I've been into bikes quite heavily. I usually enjoy road routes but prefer MTB stuff much more (name it cross-country), so I've decided to replace my entry-level aluminum hardtail with a full-suspension carbon-frame one. Sadly, it's not a good time to buy a new bike. The bike market is one of the most impacted sectors by the global supply chain crisis. Almost any kind of bike you think of will have some Asian factory-produced component, and due to the high demand and rising shipping costs, it's virtually impossible to get a new bike from a local store in less than a few months. The same happens for e-commerce that, by the way, are the fewer options.

After comparing lots of different brands and models, I did my pick: a 2022 model with updated geometry and mid-range quality components from a company that only sells online (not naming it, but easy to guess, not many options, as I said). The bike is currently out of stock, of course. However, they have a notification system on their website to send you an email whenever the bike is re-stocked, and I've been subscribed since they released this new model early this year. On one occasion, I checked the product page to compare the specs with another bike I thought could be a good alternative. To my surprise, not having received any emails from them, there was stock, so I tried, but I wasn't fast enough to complete the buy. Disclosure: their notification system didn't work.

At that point, I thought that I could do some quick hacking to not miss re-stockings again: schedule a Web scrapping script to check stock availability several times a day and get actual notifications somehow. So this is how I did it:

Web scrapping

This is the easiest part. Something I've done many times in the past, now it's just a few lines of code thanks to Puppeteer:

const URL = ''; // product's page URL
const TARGET = ''; // "buy button" CSS selector
const browser = await puppeteer.launch({headless: true})
const page = await browser.newPage()

await page.goto(URL)
await page.waitForSelector(TARGET)
await page.evaluate({selector} => {

  // if selector found; send a notification...

}, {selector: TARGET})

As simple as loading the product page looking for a "buy button" using a CSS selector.

Notifications

Since I didn't want anything involving emails, getting notifications was the challenging part. My first choice was to look for a service that would send SMSs via an HTTP API. Twillio seemed the right choice, but I didn't want to signup just for this, and its API looks complex enough to be more time-consuming than what I was willing to spend on this hacky experiment. So finally, I went for Telegram.

Telegram bots are powerful and easy to set up. Basically, all you have to do is start a conversation with @BotFather and send it the /newbot command, enter a name and username for your bot, and then you'll get a token that you can use to send requests to the Telegram Bot API.

There's one drawback, though: a bot can't start a conversation by itself. So, given that the goal is to get a notification when the scrapping script detects a re-stocking, it'd be nice to achieve it without any interaction on my side as a user. The solution is to create a channel, add the bot as admin, and myself as a channel member. Then, allow notifications from that channel.

Once you have an API token and a channel ID, using the Node.js Telegram Bot API, sending messages to the channel is straightforward:

const TelegramBot = require('node-telegram-bot-api')
const bot = new TelegramBot('token')
bot.sendMessage('channelID', 'hello world!')

Automation

Initial option: use one of the AWS Lightsail instances running Wordpress Bitnami images I have. I'd deploy the script and add a crontab entry to schedule the execution every hour. It worked, but after a couple hours, the CPU utilization spiked, and the instance stopped. It's the smallest Lightsail instance you can choose: 512 MB RAM and 1 vCPU; it runs a Wordpress blog with 1k daily visits (approx) smoothly, and while I know Puppeteer is somewhat resource-intense, I thought it shouldn't have any problems running it. Since I wanted this done fast, I moved on and ended up running it on my local machine (MacBook Pro 2017) using launchd. Here is the basic job definition template I used:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>

    <key>Label</key>
    <!-- 
      Replace [project]; handy commands:
      - `launchctl load ~/Library/LaunchAgents/com.[project].daemon.plist`
      - `launchctl unload ~/Library/LaunchAgents/com.[project].daemon.plist`  
    -->
    <string>com.[project].daemon.plist</string>

    <key>RunAtLoad</key>
    <true/>

    <key>StartInterval</key>
    <!-- value in seconds, so this will run every hour -->
    <integer>3600</integer>

    <key>StandardErrorPath</key>
    <string>/path/to/project/stderr.log</string>

    <key>StandardOutPath</key>
    <string>/path/to/project/stdout.log</string>

    <key>EnvironmentVariables</key>
    <dict>
      <key>PATH</key>
      <string><![CDATA[/usr/local/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin]]></string>
    </dict>

    <key>WorkingDirectory</key>
    <string>/path/to/project</string>

    <key>ProgramArguments</key>
    <array>
      <string>/usr/local/bin/node</string>
      <string>index.js</string>
    </array>

  </dict>
</plist>

Final result

After just 3 days running this experiment, I got notified on a Monday morning at 6:00 AM of a re-stock, and I managed to complete the purchase. A couple hours later, the bike went out of stock again.