Grafana @ eBay - Moving Monitoring Visualizations from custom UIs to Grafana Plugins - Vijay Samuel Github: @vjsamuel Twitter: @vjsamuel_ - GrafanaCon
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Grafana @ eBay - Moving Monitoring Visualizations from custom UIs to Grafana Plugins Vijay Samuel Github: @vjsamuel Twitter: @vjsamuel_ © 2019 eBay. All rights reserved.
Many moons ago…
Monitoring and Logging UIs were custom
Logs Metrics
Built Custom Log Viewer
tools No different from logs
Developers would build Users instrument code with internal
custom interfaces using frameworks to ship metrics.
HTML CSS and JS
libraries.
Custom UIs allowed users to view their
Users instrument code with metrics and set up alerts.
internal frameworks to ship
logs. Quality of UIs were greatly dependent
on the person building
The Custom UIs allowed
users to search and view
behaviors of their
applications.I’m not a frontend guy … How my frustrations with building UIs led to a PoC on Grafana!
A First Attempt At Grafana! The Hack Modify the OpenTSDB data source on Grafana HEAD to understand our internal TSDB’s request/ response formats Dashboards Build out scripted dashboards for one of our internal monitoring components. Distribute the dashboards to folks who do on call to see if it really helps
What was missing? It wasn’t clean The PoC was used within the team and it was a dirty hack to modify source code directly. It wasn’t maintainable. Missing Features Basic features like drawing a graph and aliases were present. Still lacked support for features like templating, annotations.
The PoC was lost in the sands of time. Until…
Folks from DB Ops loved Grafana so much that they decided to help out! Thank you Steven West, Auston McReynolds!!!
How did they help? How I took it and ran with
Clean things it!
up Rebase often
What was a dirty hack was No more dirty changes.
now a proper Grafana data Rebase often and roll out
source plugin. new containers with most
recent version of Grafana.
Plugin was still a modified
grunt generated file J Also had Kubernetes specs
Add Docker for folks to easily deploy.
Support Build a community
Created a Dockerfile Grafana was still a pet
and build scripts to project at this point
create a container with
the data source loaded.
Evangelize the product
and gain more
adoption.The big break through… SRE decides to revamp all their monitoring dashboards with the help of Grafana! Thank you Satish Sambasivan!!!
SRE takes it to the next
level
Overlay data Contribute code
Overlay application metrics SRE not only increased
with change executions in adoption to the system but
the system. also submitted PRs to
enhance the plugin.
Requests Monitoring
(Still a grunt generated file)
Provides hosted team to support
dashboards Grafana!
Hosted dashboards Life has now become
with key indicators for so easy for SRE that
site applications they want the
provided as templated monitoring team to
dashboards for product officially support
developers by SRE. Grafana!© 2019 eBay. All rights reserved.
We said YES!
A whole new make over
Throw away that JS
file! Provide a hosted solution
Remove grunt generated Move away from a model
code and do it right this where folks spin up their
time. own in favor of a single
hosted solution that is
managed by the monitoring
team.
Add widgets for logs/
events Build out more features
Provide a single Use Grafana’s custom
interface for logs/ apps to build out
events/metrics on alerting experiences
Grafana with the help of based on internal
internal widgets. alerting APIs.
Log viewer plugin Implement annotations,
provides a tail like eBay specific
experience. authentication etc.What happened in the
backend
Move away from custom
APIs Same would apply for metrics
Instead of building custom Instrument code with well
APIs to push logs/metrics/ known libraries like
events we decided to use Prometheus.
cloud native mechanisms.
The platform would scrape
metrics and store them!
Do all of this as a community
A newbie can log to file player!
Anyone should be able Contribute to various
to write logs to either projects like Elastic
stdout or to a log file. Beats, Flink etc. with
Our platform would features and bug fixes.
collect and ship the
data.
Keep custom code in
plugins.
This allows new hires to
onboard faster.Lessons learned…
Be closer to
community Cloud Native is everything
Never resort to in house Moving away from
code changes unless proprietary APIs and into a
working with community cloud native model helped
fails. onboard use-cases that
weren’t conventionally
possible.
Give code back to the
Creating dashboards is
community when new
features are built.
We should’ve adopted Grafana
easy sooner!
The cost of building out This was almost 4
data source plugins is years in the making.
far cheaper than
building out entire UIs.
This should’ve been a
no brainer J
Rome was not built in a
day but our custom
data source was!Thank You © 2019 eBay. All rights reserved.
You can also read
