HoneyByte: Make a Beeline Toward Observability Just Like DEVs Molly Struve

When things broke, Molly explained, youre mad scramblingjumping from website to website to website, trying to put the pieces together. Molly was able to use Honeycomb to fix things up: It makes my job easier as an SRE.

.ltag__user__id__119473 .follow-action-button { background-color: #61122F !important; color: #FFFFFF !important; border-color: #61122F !important; }

Molly Struve (she/her)

International Speaker Runner Always Ambitious. Never Satisfied. I ride 's IRL

molly_struve mstruve https://www.mollystruve.com

Getting started with Honeycomb doesnt require a lot of work: at dev.to, they used the Ruby Beeline to get it going: I didnt do that much, she said. I feel like I cheated! We invite you along on their journey, and we'll share precisely what lines of code dev.to used to get started.

Service Fatigue

When I started at DEV, there was very little in the way of monitoring, in terms of observability, said Molly. We had six or seven little tools, each doing one little piece.

While each tool had initially served a purpose for the team, the fact that there were so many became a pain point during incidents. Its a familiar experience for SREs or anyone who owns code in production.

Youd look at one service, youd see a spike, but that was all the information you'd get. So then youd have to go to the other services and try to line up when the spike happened.

Even during normal site operations, thats enough to fatigue most people. With the added stress of an outage it becomes painful, error-prone, or both. Thats why Molly took the initiative to consolidate DEVs use of monitoring services as much as possible. Whether facing an outage or just wanting to see how the site was being used, she wanted all that information in one place.

Mollys initial goal was to consolidate her tooling, but what she found with Honeycomb were benefits beyond easing the immediate pain of service fatigue.

Honeycomb really gives us some great insight into how users are interacting with our platform, she said. Its been super helpful not only for debugging issues, but allowing me to find places in the application where slowdowns are happening, were hitting the database too many times, etc.

Its no coincidence that this was Mollys first project after joining the DEV team. She gave a talk at SREcon19 last year on the benefits of an easy-to-use, centralized monitoring system. The ability to have all that information in one spot and then be able to really drill down is super convenient, she said. Having it all in one place has been super helpful.

Additionally, streamlining their tooling has simplified onboarding engineers at DEV. They no longer have to learn five, six, seven different tools, she explained. When they come on, I can say 'Heres Honeycomb, this is how you use it,' and thats it.

Sometimes people think when youre an SRE that we just magically know, but we dont, said Molly. Half of what were doing is literally just throwing in queries, hitting stuff. We look at a graph, Ah theres nothing there. Okay lets look at a different one, Ah theres nothing there.

Molly makes videos for her teammates demonstrating how she uses Honeycomb to help them learn the tool, and seeing her exploration process has helped them understand that its more about mindset than expertise.

I think that was a big aha moment for some people, she said. Theres no shortcuts to what we do, its basically getting all the data and finding the pieces that stick out.

Honeycomb enables first-principles debugging. That's when you dont have to be an expert in the system, you just need to know where to start digging. Demystifying debugging as a process can empower an entire team.

Getting Up and Running

Molly described her experience adding Honeycombs Beeline gem for Ruby. It was simpler than she anticipated.

I didnt do that much, she said. I feel like I cheated!

I literally just took your documentation, slapped it into the project and it was like, Boom! theres all of your data, she said. It was pretty awesome. DEV is open-source, so you can see the exact change she made in her pull requeston GitHub:

• add honeycomb-beeline gem to Gemfile
• add Honeycomb API key to secrets
• generate Honeycomb configuration using the Beelines Rails generator: bundle exec rails generate honeycomb YOUR_API_KEY --dataset rails

Once Mollys PR got merged, the DEV team was sending data to Honeycomb and gaining new insights.

It was super straightforward, and honestly that got us probably 80% of the way there, said Molly. Any tool where you can get up and running so quickly is huge.

By using the Ruby Beeline first, Molly reduced the overhead of starting out compared to instrumenting the entire application from scratch. If you can get value out of it before you even have to customize? Thats a pretty big deal. If you have to go into the weeds before you can get value? Thats a blocker for a lot of people.

Once she used Honeycomb with production data, Molly thought of specific areas she wanted to instrument in more detail. Observability isnt all-or-nothing. The incremental approach is a lot less scary.

Using the Tool

Honeycomb has several features that encourage users to dive in and start querying. The suggested queries on the home page help users get oriented.

Thats one of the first things I start most of the videos with, Molly said, referring to the instructional videos she makes for her teammates. Looking at the home page, Molly said, something sticks out and then I kind of dive into it.

Another feature of Honeycomb that Molly appreciates is the query builder, in particular, the autocomplete. That helps immensely in trying to figure out what fields you want to query for.

Rather than having users type into a blank query editor, the query builder reduces friction by breaking up clauses into separate fields. Theres no bespoke query language to learn, its SQL but more visual and with suggestions.

(Read more about the query builder.)

With the query builder you dont have to memorize all the fields the app is sending to Honeycomb. This evens the playing field among team members.

Molly described to her coworkers the mindset she has when investigating interesting data in Honeycomb: Sometimes Im not even quite sure how Im gonna filter down into what I want, but I just start typing and I start scrolling and I start looking.

BubbleUp is another unique feature in Honeycomb that Molly finds super helpful. Let me tell you how many times Ive been on other platforms where all I want is to just narrow down that graph with a drag and drop, and you cant.

Rather than trying to correlate system behavior and line up time periods across separate graphs, BubbleUp does the work of comparing the baseline to the outlier data youve selected.

Shared History

Another Honeycomb feature Molly loves is the saved queries. Usually Im opening like, tab after tab after tab after tab trying to save all my queries. I go down one path and it doesnt return anything, so then I have to go all the way back.

Honeycomb automatically saves your query history and displays them in the sidebar. Having those saved queries, you can literally just go over to your sidebar, you can easily go back, you can go down a rabbit hole and come back to your starting point in seconds. Thats one feature thats really stood out to me in Honeycomb that I havent really seen anywhere else.

The shared history and query permalinks make it easy to collaborate and get a second set of eyes, or to prevent multiple people from hitting the same dead ends. If youre having an incident you can slap the link to people. They can see your history as well so then they can recreate what you did and possibly find issues with it, possibly take it down their own path.

Level Up Your Team

If you're a Rails developer, you can add the Beeline gem to your app and start sending data today. Go to honeycomb.io/signup to start for free.