I tell anybody that will listen that much of the work I do on growth and analytics for WordPress.com wasn’t really possible until the last couple of years; and that’s almost entirely due to a couple of applications that we use. Of course, some of these have been around for much longer than that. But as a group, these applications (sometimes interoperable) help us to understand our users and their engagement, and the impact that our new features and experiments have.
KISSmetrics
Kissmetrics is a user-level measurement and analytics tool that tells you what things users do on your site, and lets you report on those activities. Internally we’ve been referred to as the ‘Kissmetrics team’, and with good reason: it’s the hub of our data for experimentation. We record dozens of individual key user events, adding up to hundreds of millions every month. It’s also the place that we record properties about users: everything from the device they’re using to the amount of money that they’ve spent on upgrades; it’s how we record details about cohorts that we assign them to.
I think that Kissmetrics solves two big problems that it would be hard for a company of our size to solve ourselves: first, large-scale data storage and retrieval. Second, a great reporting interface that allows us to query and analyze our user data.
Something of a distant third, but still incredibly useful, is the integration with a handful of other testing tools. Which brings me nicely to…
Optimizely
Optimizely is a front-end a/b testing tool that we use for homepage and signup flow tests. It’s great for experiments with logged-out WordPress.com visitors and really easy to use. If you’re scared of code, the easiest way to use Optimizely is to use its drag and drop editing interface. You drop a line of javascript into your page, and Optimizely pretty much figures out all the rest.
Because we’re not scared of code(!) and prefer to keep the code variations in our own files, we just use Optimizely to set cookies for users and then determine which variation (controlled with javascript) to show a given user. Then Optimizely lets Kissmetrics know which variation of an experiment was shown to a given user.
We rarely use Optimizely’s own results for a test. I tend to much more interested in how a change to the homepage affects a user’s likelihood to do something much further down the line: it could be signing up; publishing their first post; maybe even whether or not they’re still engaged a month later. Optimizely can track events besides clicks on that page, but it’s much easier to have the data all in one place, so we use Kissmetrics to determine the winner instead. Because of this, we created one Kissmetrics report that tracked some of our key metrics for early users and we were able to use this same report to measure the impact of successive tests on the homepage.
Qualaroo (formerly KISSinsights)
If you’ve never seen a little popup survey appear on a website, you’ve not traveled the world wide web very much. Many of them are the work of the team at Qualaroo, which makes it incredibly easy to create these little one-off polls that can help to elicit qualitative data from users at specific points in an application.
One of our current uses is for asking users immediately after they sign up for a little detail about what they plan to write about. Just like Optimizely, you can have those responses sent to Kissmetrics. We’ve found pretty interesting insights by cohorting users based on their responses to the survey, and then comparing publishing and other engagement metrics across the cohorts.
Recently, this helped us to understand the increase in signups that we see every new year: we’ve always had to guess what kind of users make up that early January bump; this year we had much richer data on the users, and were able to compare publishing and spending rates across the different cohorts.
Mission Control
MC (for short) is where we keep our internal data. It’s the one true source that we rely on for our internal metrics. I like that the app we use for our experiments (Kissmetrics) is different from where the canonical record is kept. It lets us be fast and scrappy, and not worry too much if we record the wrong piece of data every now and again, or decide (as happened around six months ago) to empty all of our data from the account and start over. Evan provided a more in-depth look at how some of the MC data is visualized, if you’re curious.
Google Analytics
The original… but no longer the best. Google Analytics serves a handful of intermittently useful functions for us:
- Provides great data on audience demographics and technology (both for our publishers and our consumers).
- Allows us to track trends in visitor referral data.
- Provides a sanity check for internal and other app measurements.
These days it’s used mostly as a reference tool: the per-user based tracking that Kissmetrics provides us makes Google Analytics’s Goal Tracking functionality seem pretty old!
Excel
The original… and amazingly, still the best. Kissmetrics reports cover lots of important bases, and are particularly good for at-a-glance trends and understanding funnels. But my most frequent workflow involves exporting their People or Power Report data into csv and then manipulating it in Excel. It’s by far the most comprehensive tool for things like Pivot Tables and quick charts. Hey, if it’s good enough for Nate Silver…
Other websites
isvalid.org is a project built by my former colleague, Evan Solomon. It’s a great tool for measuring significance in test results, especially when combined with the bookmarklet which lets you grab your experiment data from the page you’re on and sends it to isvalid.org.
Tableizer lets you paste data from Excel and then turns it into an html table. Automattic is a completely distributed company, so we communicate with writing a lot (on blogs, of course). But however you work, you probably write up your results, so you may find this handy too.
Clicktale might be my favorite thing for gathering qualitative user data. It allows you to record user sessions and watch them back in real time. The user never knows, so they’re acting under entirely normal conditions.
UserTesting.com I prefer in-person user testing, but if that’s not an option, directed online testing with an app like UserTesting.com is really efficient.
We’ve been playing with some SEO tools as well, I’ll maybe write up a separate post on those another time.
Other products are available
Of course, there are alternatives to all of these, and I’m sure there will be new products in the coming months that may well end up displacing some of the ones that I’ve written about here. It’s also likely that over time (and as the company appreciates more the value of the data in product development) that we’ll build more of this capability ourselves.
Pete's blog 