Kadence:
The one about the scraper

Kadence International is a boutique market research firm with global reach. They offer a full suite of market research services to companies ranging from startups to Fortune 500 brands. The Kadence Boston office approached Setfive with a unique project with a unique set of constraints.

The Kadence team was interested in building a tool to automate extracting data from a Japanese weather website based on the locations that had been indicated in completed Kadence surveys.

  • Because this was a short term project, and Kadence Boston didn't have any existing cloud infrastructure the solution needed to run on the laptops that Kadence analysts were using.
  • The Japanese weather website was using a mix of JavaScript and Adobe Flash to load data. Because of this we needed an approach that could execute Flash to make the data available to us.
  • The Kadence team would need to configure certain attributes while using the tool and because they're non-technical we would need some sort of GUI.

Drawing on our past experience, we suggested building an Electron app that Kadence analysts would be able to use on their laptops to scrape data as they needed it.

  • Electron is a cross platform framework, so the app was able to run on any analysts' laptop regardless if they were running Windows or OSX. Additionally, Kadence didn't have to set up or pay for any cloud resources.
  • Under the hood Electron uses Chrome, so the app was able to fully execute both the JavaScript and the Flash applet on the website. This functionality was key to being able to scrape the data from the site.
  • Because Electron loads HTML/CSS similar to a normal Chrome browser, we were able to build a UI for the app using the same web technologies that we're familiar with.

The app was a big hit with the Kadence analysts. It saved them hundreds of hours of tedious work over the course of the project and allowed them to focus on analyzing and understanding the data.

Overall we think Electron is a great platform which allows companies to explore ideas like automation without a large upfront investment.