Lightbody's Corner

BrowserMob Proxy 2.0 beta 8 released!

Sat, 04 May 2013 11:35:00 -0700

It’s been a few months and it was pretty obvious a new release of BrowserMob Proxy was in order. If you’re impatient, you can get it here and skip this announcement.

The most important thing to note is that I’ve forked the project back in to my personal Github repository. In addition, the website has been moved to bmp.lightbody.net. I did this simply because I no longer work at Neustar/Webmetrics and I’m the primary committer to this project, so I figured I should keep it where I know I’ll have control over it.

Now to what’s new:

A brand new test suite so I can finally make sure I’m not breaking anything too badly
Restored Java 6 compatibility, which broke in beta 7
Binary content can now be captured, resulting in base64 encoded content in the HAR body
We no longer modify the User-Agent header (it used to insert “BrowserMob RBU” in to it)
Post data (URL encoded and raw content) is now reliably captured in the HAR

This release also includes some REST API improvements:

Request/response interceptors can now be set via the REST API using a simple JavaScript scripting language (needs documentation still)
Automatic BASIC authentication can now be controlled via REST
Waiting for network activity to stop can now be accessed via REST
Network activity control, such as maxBitsPerSecond, is now accessible via REST
Control over connection timeouts is now available via REST
And more!

PS: Someday I’ll get back in to more interesting blogging, but for now I’m just happy to post something :)

Capturing errors in the end user experience

Mon, 16 Jul 2012 08:00:38 -0700

This year I again had the privilege of delivering a keynote during the Velocity Conference in Santa Clara, CA. Last year I shared my experiences, lessons learned, and cloud services I depended on during the lifetime of BrowserMob - from inception to acquisition (video here).

This year I spent my time on stage discussing the various types of tools DevOps people depend on to maximize performance, availability, and functionality. Stuff like HTTP synthetic monitoring, functional testing, application performance management (APM), browser-based synthetic monitoring, and more recently Real User Measurements. You can find the video here.

RUM was a big topic this year, especially because all the major browsers now support Navigation Timing. But I was surprised how much interest little sidebar on window.onerror got: I had quite a few people coming up to me after to learn more about JavaScript error logging.

Despite window.onerror having been around for years, very few people utilize it to gain insight in to the experience of their real users. For example, of the 60 sponsors at Velocity, only one (Dropbox) was actively capturing JS errors and sending the logs back to a central server.

What’s interesting is that the equivalent thing for mobile apps has recently received quite a bit of attention, mostly due to several competing hot startups:

I never saw this kind of interest in JS error reporting platforms, and I think part of the reason is that people think of JS errors as “non critical” and not at the same level as an app crash. But with web apps as complex as they are today, a single JS error could be just as bad as an app crash, in terms of user experience.

There are few services out there that do JS error logging:

Fire Tower (recently acquired by Crashlytics)
ExceptionHub
Errorception
DamnIT

Most don’t look super professional, with the exception of Fire Tower. And now that it’s part of Crashlytics, it’ll be interesting to see how the combination evolves and grows out the analytics around error reporting. I’m also curious to see if log management services like Splunk, Loggly, Sumo Logic, and Papertrail get in to the game of analyzing error messages and error rates.

10 Tips For Achieving Email Bliss

Thu, 09 Feb 2012 11:20:06 -0800

Over the last few years I’ve learned that the best way to manage my inbox was to follow the Inbox Zero technique, widely advocated by Merlin Mann. As stated on the official website, it’s not about literally having zero items in your inbox (though I do get there), but rather how much of your brainpower is dedicated towards managing your email.

Since I’ve been personally able to achieve a near empty inbox for the last few years, I figured I should share the tricks that work well for me. So here they are!

Tip #1 - Don’t bother with automatic filters

To some this may sound absurd, but I actually believe that setting up advanced rules and filters actually perpetuates the email hell so many people live in.

It’s natural to think that it helps: less stuff flows in to your inbox and instead gets sorted to appropriate folders. The problem is that you now have the constant pressure of checking those folders.

In my experience, anything I automatically filter in to folders or labels ultimately either gets ignored or just causes me to worry more about email, wondering what’s hidden behind that folder - exactly what I’m trying to avoid!

For example, rather than subscribe to email delivery on some Google Groups mailing list, I almost always opt out of email delivery. Instead, I will search the archives and post my question. I’ll then set up a reminder in a few hours or days to check back in via the web interface and see if my question is answered.

For any other distribution list that is important enough for me to be able to respond in real time, I let it flow directly to my inbox. I may still end up archiving or deleting 99% of the emails but if it’s important enough to respond quickly, it’s important enough to see in my inbox.

Tip #2 - Archive like crazy

With storage what it is today, there’s no point in deleting. Even in archaic organizations that give you a puny amount of storage, you should be just shoveling email from Exchange to local storage. The primary benefit to archiving, or course, is that you can easily search for it should you need to pull up an old email that you glanced at and then tossed.

And that’s the real point: don’t spend a lot of time reading email. As long as you don’t delete them, there’s no real risk to just skimming and archiving. Obviously if it’s an email you should respond to, don’t archive it right away, but I bet at least 90% of the email you get can be skimmed and archived in a matter of seconds.

Tip #3 - Be concise

For the 10% of the emails you really do have to reply to, practice the art of keeping your replies concise. This is really hard to do - probably the hardest of these tips. In fact, at first you’ll find that writing shorter emails actually takes longer, since brevity is such an artform.

But it’s also an important skill to learn, so keep practicing. Not only is it beneficial for your own time, it saves the time of those that you’re sending email to. So if you can’t respond in a few sentences, odds are that you should…

Tip #4 - Schedule meetings & reminders for the big stuff

Too many people (myself included, in the past) treat their inbox as some sort of todo list. But the problem with having your inbox serve multiple roles is it then creates stress and anxiety when you look at it. Is this email something I have to do? Is this just a new email? Should I reply now or wait until I do Task X?

All these questions create uncertainty around every email in your inbox. It becomes less clear which emails demand immediate attention and which are associated with longer term tasks. Often the result is that you end up suffering in both directions: tasks that need to get done lag for weeks in your inbox, and emails that require more immediate attention pile up.

Fortunately, it’s easy to avoid this: just make time for Real Work. That means if a request for something that takes an hour comes in by email, quickly reply by email saying “I’ll have it ready by Friday at 3PM” and then block out 2PM to 3PM on Friday in your calendar to get it done. If there is no timeline associated with the task, putting it in your favorite todo list (ex: Flow or Toodo) is fine as well.

Tip #5 - Manage email in bursts

So far we know that we should archive the 90% of the emails that demand very little attention and that we should schedule meetings & reminders for the stuff that demands a lot of attention. But what about all the stuff in between?

That’s where handling your email in bursts helps. For example, while I am always keeping an eye on my inbox throughout the day, I don’t actually concern myself about it most of the time. Instead, I reserve most mornings and afternoons for about 30 minutes blasting through my inbox. It’s at this time than most of my archiving, calendar’ing/todo’ing, and concise replying happens.

For the last week I tracked what my inbox looked like at the start and end of every ~30 minute email session:

You can see that at the end of every session I often have 10 or fewer total emails (this is for all my inboxes - personal and work) and no unread emails. I also, through practice, have learned how to send 30+ emails in a very short amount of time - even if some of them simply say “let’s talk about this on Tuesday” or “I’ll have this by Friday at 3PM”.

Tip #6 - Wait a few days

Another thing to remember: even though the world seems like it’s moving faster than ever, it’s likely that very few emails that you receive really need a reply right away. In fact, even if you can respond right away, there is some value to taking your time before you send back a concise, thoughtful response: by delaying your responses you begin to condition your coworkers that they shouldn’t use email like it’s a realtime conversation tool (for you at least).

In fact, one really easy way to condition your peers is to…

Tip #7 - Be shameless

Be shameless. Wait a few days and then respond and ask: “Do you still need help on this?”. Not only does this help condition people that you will likely take your time responding to non-critical items, but it also sends a message that while you hope/expect that they’ve solved their problem, you’re there to help in case they haven’t.

In my experience, in a healthy team most email topics work themselves out before I really ever get around to them. It’s still good to be kept in the loop and it’s still good to ask if you need to get involved, but by letting issues sit around for a bit and then following up, you can force the right behavior, which is often communicating using realtime instead of with email…

Tip #8 - Get realtime: talk in person and use voice/IM

For some strange reason, millions of professionals have been groomed to send an email to ask a question, even when the recipient is sitting just a few feet away. Sometimes this makes sense, especially when the question requires a little more thought or can be answered asynchronously.

But most of the time people want answers now and still resort to email. Once you’ve conditioned your coworkers that they won’t get prompt responses from you by email, start pushing them towards realtime communication channels like face-to-face conversations (shock!) and using tools like Skype.

Even better, if you’re in an email thread that seems to be getting out of hand, avoid the temptation to keep replying (especially so when it’s a frustrating/confrontational thread). Instead, offer to jump on the phone (RIGHT NOW if necessary) or schedule a meeting to work it out. A ten minute phone call is worth dozens of emails back and forth.

Tip #9 - Use Boomerang

Although you shouldn’t be using email as your todo list, sometimes it’s not practical to keep a todo item to track every little issue/request that you have of your coworkers, especially when you’re sending out emails like “When will you have feature X complete?”.

That’s where a tool like Boomerang comes in. It lets you pick a time window for an email to pop back up in to your inbox if it hasn’t been replied to yet. So you can ask Bob about “feature X” and tell Boomerang to remind you if you Bob hasn’t responded in 3 days.

This is incredibly useful especially for mid-level managers (that’s me!) who have a dozen or so team members and might have half a dozen outstanding questions for each one of them. None of them are important enough to warrant time on my calendar or an item in my todo list, but they are little ways I manage my team. I need to know if I ask a question and it goes unanswered, so Boomerang is a life saver.

Note: there are other competing solutions that work with other mail systems besides Exchange and Gmail, but I’m not aware of any better ones. If you know of some, please leave a note in the comments!

Tip #10 - Use conventions with your coworkers

Lastly, it can only help to use common conventions in your email subjects. For example: AR (Action Required), URGENT, and FYI are three common ones I use frequently. A simple way to encourage your teammates to use these conventions is to start using them yourself.

Over time your team should start picking up the conventions. But if that doesn’t work automatically, don’t be afraid to send out an email asking the team to agree on a few conventions and to understand that you won’t review urgent items unless the subject says URGENT in it.

Final thoughts

It’s important to understand that your inbox shouldn’t control you. It’s one of many tools that is there to help us do our jobs better. But it’s up to us to decide when and how often we’ll use each of these tools.

By following a few of these tricks and conditioning yourself and your coworkers to use email differently, you’ve got a fighting chance at experiencing the same email bliss I do almost every day: often less than 10 emails in all my mailboxes, and a day mostly spent outside of my email client.

Do you have your own email management tips? Please share them in the comments!

Pretty cool video showing Steve Jobs and the original NeXT team...

Mon, 21 Nov 2011 13:20:06 -0800

Pretty cool video showing Steve Jobs and the original NeXT team in an offsite strategy session just as they are launching the company.

From this year’s GTAC: A good overview of Kevin...

Mon, 07 Nov 2011 12:41:05 -0800

From this year’s GTAC: A good overview of Kevin Menard’s vision for checking for visual consistency of look & feel of web pages across multiple browsers. Much of his lessons in this video are reflected in his startup, MogoTest.

A Sister’s Eulogy for Steve Jobs

Sun, 06 Nov 2011 15:22:06 -0800

A Sister’s Eulogy for Steve Jobs:

A touching from Mona Simpson.

Where the heck has Patrick been?

Mon, 24 Oct 2011 09:30:05 -0700

A couple weeks ago, I started blogging again after a hiatus over over two years - my last post was in July 2009. I figured before I got too carried away with any more activity, it might be worth a quick recap of where the heck I’ve been!

Back in November 2008 I announced my second startup, BrowserMob. My goal was to re-invent the world of website load testing by utilizing real browsers (and Selenium) in the cloud. About a year later we launched our second product, a low-cost website monitoring service.

During the summer of 2009 I did an exhaustive search for VC funding, getting several term sheets along the way. Just as an aside: it’s amazing how much the power has shifted to founders in just 24 short months. Today it’s relatively easy to raise a first round of funding on a $3M+ pre-money valuation. Doing so in 2009 was a different story!

But before I closed the financing, I was fortunate enough to have generated enough profits to simply hire two awesome engineers (and friends), Ian White and Rafael Ferreira. We were able to continue to grow organically and profitably, racking up over 500 customers along the way.

And then out of the blue Neustar and a few other companies called to discuss an acquisition. I ultimately decided on selling the company to Neustar for many reasons, which I will write about soon. Needless to say, the experience and lessons learned along the way were life-changing. I met a bunch of fantastic investors, customers, potential acquirers, and founders of startups in the cloud space (ex: Twilio, CloudKick, AppFog, PuppetLabs, PagerDuty, etc).

After the sale I moved from Portland, OR to San Diego to join Neustar and run the product and engineering teams under the Webmetrics product line, (which BrowserMob is part of). I’ve been having a blast since then - both professionally and personally.

Not only do I have a great team that I get to work, I have the backing of a publicly traded company with a $2B+ market cap that is 100% committed to expanding my vision of a simple-but-powerful suite of website performance testing products. And most importantly, since moving to San Diego my wife and I had our first child, a beautiful baby girl named Maggie Rose Lightbody.

Life is good and I’m glad to be stepping back in to the world of blogging, technology, and startups. Part of why I’m blogging again is that things have finally settled down. Between the chaos of the startup, the acquisition, the move, the new job, and the baby it was hard to do much else. But things are a little more calm now and I’m energized to engage with the startup community again and contribute to this blog much more regularly.

Battlefield 3 looks awesome. I was a big fan of Battlefield: Bad...

Fri, 21 Oct 2011 12:40:06 -0700

Battlefield 3 looks awesome. I was a big fan of Battlefield: Bad Company 2. The entire BrowserMob crew - all three of us - played it a bunch. Doubtful I’ll get to play BF3 too much with a 1 month old baby at home, but it’ll probably steal every spare second I can find. Definitely hoping that the boring Call of Duty line finally gets eclipsed by Battlefield this holiday season!

From Steve Jobs: Biography: "I'm going to destroy Android"

Fri, 21 Oct 2011 06:32:00 -0700

From Steve Jobs: Biography: "I'm going to destroy Android":

I love Steve and I love Apple. But it is annoying to see reports of Steve and others claim that Android ripped off the iPhone but few reports of the reverse.

The new iOS 5 notifications are awesome. They also are a total copy of Android’s. It goes both ways.

Read the original story here: From Steve Jobs’ Biography: “I’m going to destroy Android” (shared from Google reader).

Pretty cool Steve Jobs tribute video that walks through the...

Wed, 19 Oct 2011 16:23:06 -0700

Pretty cool Steve Jobs tribute video that walks through the history of the iPhone and all the technologies rolled up in to it (mobile phone, Walkman, internet, OS X, CPU, etc).

Newsflash: people still buy software

Wed, 19 Oct 2011 12:40:06 -0700

I don’t know the exact timing, but somewhere between 2000 and 2007, much of the Silicon Valley lost interest in the idea that a business could build software and that people would buy it. Sure, plenty of startups still went about doing the enterprise software thing - companies I worked for like Jive and Spoke Software.

But the excitement around those kinds of companies pretty much dried up after the bust and the valley re-emerged from it’s hole a few years later. Valley 2.0 didn’t seem interested in selling software. Instead, it was all about social media, advertisements, group buying, gaming (aka ScamVille), etc.

The Independent Software Vendor (ISV) has been forced to stay exactly that: independent. Most investors you talked would tell you plainly that people don’t buy software (or music, no one told Apple). They’d say that your idea isn’t fundable - good luck to you.

I know: both my startups (in the software testing space) were extremely hard to get funding for, which ultimately turned out to be a blessing after I sold BrowserMob to Neustar and retained 100% of the equity. To be fair: My second time around (in 2009) I did begin to notice a difference in attitude and I met some great investors that encouraged me to sell pickaxes over mining for gold.

But something funny happened a few years ago. Maybe it was the launch of the App Store. Maybe the valley just matured a bit. Maybe it just took a few good consumer/prosumer/enterprise startups to break past the Facebook/MySpace/GroupOn hoopla. Whatever it was, the attitude started shifting and making software that people buy became (somewhat) cool again.

And good thing too. The big news this week is DropBox’s whopping $250M round, $4B valuation, and their spurning of Steve Jobs. I’m one of their 50M users and I was just thinking the other day after having uploaded another 500MB of scanned files: “Wow, I bet DropBox knows almost exactly when I’m going to become a paying customer”. And sure enough, they know it:

It’s only going to get better. That 96% of nonpaying customers is throwing their stuff into Dropbox at such a pace that thousands of people each day blow through the free 2 gigabytes of storage, and upgrade to 50 gigs for $10 a month or 100 gigs for $20. Even if Houston doesn’t sign up a single customer in 2012, his sales will double

Quick side note: DropBox, arguably the most successful example of a company that makes money the old fashioned way, applied for YCombinator in April 2007 (a good read, btw). I don’t know Paul Graham, but I can’t imagine many people funding the concept any earlier than that. For reference, by April 2006 (a year earlier) Facebook was all the rage and had already raised over $40M, and before that News Corp had paid nearly $600M for MySpace.

And then there’s Evernote, a service/tool that I’m already happily paying $45/year. I use it, along with my wonderful Fujitsu ScanSnap scanner, to turn my household 100% digital. As TechCrunch put it when reporting on their recent $50M, they want to be the antithesis of Zynga.

Setting aside all the various iPhone and iPad apps I’ve paid for (that’s another post), I’m also happily paying (or have paid) for Pandora, Skype, Flow (before that Things), Reeder, 1Password, SmugMug, and more. So despite open source, despite the Facebookification of the valley, despite all the naysayers - all trends seem to be pointing to a renaissance of the ISV and the simple concept that PEOPLE WILL PAY FOR A QUALITY SERVICE (*gasp*).

Of course, it’s not like the good old days - it’s better! No more CDs or floppy disks. Even the desktop tools, like DropBox, 1Password, Flow, and Reeder, all have nice auto-update utilities so they are as easy to use as a website. But more importantly, both consumer and investor attitude seems to have shifted back towards the idea that it makes sense to pay for a quality service.

And that’s what excites me most about doing another startup.

I know I’ll always be in the business of selling software for greenbacks. My brain simply isn’t wired to be successful at starting a social network, a game company, or a group buying site. While I don’t expect to do another startup in the enterprise/IT space, I am super excited about building enterprise or prosumer applications and services in the coming years.

Knowing that there are 50M people supporting DropBox or 11M+ people supporting Evernote gives me the confidence that I too can build something of value and create a real, long-standing, successful company out of it. I may never know why things shifted back this way, but I’m certainly glad it has.

Amazon Lets You Spin Up A Supercomputer Cluster

Tue, 18 Oct 2011 23:23:28 -0700

Amazon Lets You Spin Up A Supercomputer Cluster:

We (BrowserMob) routinely launch thousands of instances on Amazon EC2 every day. Back in 2008 when I started the company, my bet was that as big as EC2 was then, it was only going to get bigger and cheaper. And boy did it!

Today we’re one of Amazon’s largest users, in terms of peak concurrent instances/CPU cores. We don’t quite do the 30K CPU cores that is referenced in this article, but we get pretty close.

What’s really cool is we do it all serf-service: our customers schedule load tests thatm under the covers, deploy massive amounts of hardware within minutes. It’s really amazing to watch as we launch hundreds of thousands of servers every month.

But as useful as load testing in the cloud is, it’s still admittedly a simplistic use case that probably isn’t changing the world as we know it.

What I’m really excited about are all the people who now can rent supercomputers and do serious science. With Amazon’s recent foray in to GPU compute clusters, I expect to see some really cool stuff come out of biotech and Amazon in the coming years.

Read the original story here: Amazon Lets You Spin Up A Supercomputer Cluster (shared from Google reader).

Predator-Inspired Ammo Backpack Cobbled Together By Soldiers In Afghanistan

Mon, 17 Oct 2011 23:33:15 -0700

Predator-Inspired Ammo Backpack Cobbled Together By Soldiers In Afghanistan:

Read the original story here: Predator-Inspired Ammo Backpack Cobbled Together By Soldiers In Afghanistan (shared from Google reader).

"People think focus means saying yes to the thing you’ve got to focus on. But that’s not what it..."

Mon, 17 Oct 2011 16:26:05 -0700

“People think focus means saying yes to the thing you’ve got to focus on. But that’s not what it means at all. It means saying no to the hundred other good ideas that there are. You have to pick carefully.”

- Steve Jobs

Really awesome video showing off techniques for inserting...

Mon, 17 Oct 2011 12:41:05 -0700

Really awesome video showing off techniques for inserting artificial objects in to photographs of real places. Crazy stuff!

HALL.com Raises $580K From Founder’s Collective And Others To Transform Realtime Collaboration

Mon, 17 Oct 2011 06:18:23 -0700

HALL.com Raises $580K From Founder’s Collective And Others To Transform Realtime Collaboration:

I’ve had the pleasure of watching Brett go through several iterations on his idea and I he’s going to do great with Hall.com.

Despite the “big” players in the space like Jive (my old employer), Yammer, and Salesforce, there is still plenty of room for innovation and disruption in group collaboration, including in the enterprise.

It doesn’t hurt that his investors (Founder’s Collective and PivotNorth) are a great bunch. Good luck Brett!

Read the original story here: HALL.com Raises $580K From Founder’s Collective And Others To Transform Realtime Collaboration (shared from Google reader).

Chronon 2.0 Offers Post Execution Logging

Sun, 16 Oct 2011 22:33:16 -0700

Chronon 2.0 Offers Post Execution Logging:

I’ve seen Chronon and it’s really cool technology. Hoping to try it out on BrowserMob soon!

Read the original story here: Chronon 2.0 Offers Post Execution Logging (shared from Google reader).

frontend SPOF survey

Sun, 16 Oct 2011 21:48:01 -0700

frontend SPOF survey:

Pat Meenan continues to offer up simple and practical advice for anyone looking to build a professional, high performing website.

Here he provides a fantastic example of how a simple thing like Twitter being offline can have disastrous effects on your own website.

Read the original story here: frontend SPOF survey (shared from Google reader).

WSJ Gets Personal With Gravity

Sat, 15 Oct 2011 22:48:09 -0700

WSJ Gets Personal With Gravity:

Really glad to see that people are still trying to figure out how to effectively personalize news.

A few years back I worked on a startup/side project called mioNews. It never quite got full lift off but the intent was to combine your social graph, your reading habits, and your reason feedback (like/dislike) and be able to deliver the top X stories you should read today.

I still think this is a huge opportunity and might eventually try to tackle it again (if someone else doesn’t nail it first).

Read the original story here: WSJ Gets Personal With Gravity (shared from Google reader).

The tribesman who Facebook friended me

Sat, 15 Oct 2011 16:20:06 -0700

I just read a really interesting story about a tribesman from Papua New Guinea, whose tribe just recently discovered that feathers make arrows fly better, re-connecting with an old friend over Facebook. The author asks:

Should we mourn the passing of a phase in our history when bands of human minds still lived in isolation, or rejoice that we are finally all on the same page?

My answer: we should rejoice that we’re all able to connect, as I truly do believe that a more connected world is a less dangerous world (ex: it’s unlikely we’ll ever see WWIII due to our global economy).

But we should also mourn the fact that Facebook is the tool they’re using, as it tends to appeal to the lowest common denominator. My personal contribution to the world is to try to avoid Facebook as much as possible these days!