Mat Schaffer

+81-70-1629-9509

A builder of approachable and reliable systems that handle unrelenting internet-scale traffic.

Work Experience

October 2022 - Present

Principal Software Engineer

Roblox

Japan - Remote

  • Build Golang automation for managing a fleet of 20+ Kubernetes clusters
  • Ensure workload portability between Rancher, Tencent Cloud, and AWS k8s distributions
  • Test full functionality of Spark & Trino/Presto data stack for EMR & EKS
  • Coordinate Airflow migrations and Grafana/VictoriaMetrics/Loki observability for thousands of company-wide data tasks
  • Optimize cost for storage and transfer of 100+PB data lake

July 2021 - October 2022

Principal Software Engineer

Elastic

Japan - Remote

  • Developed new features for Elastic's Observability stack
  • Triaged and fixed functional and performance issues in Golang (beats) data ingest tools and React (Kibana) UI
  • Diagnosed customer issues with Elasticsearch for cloud and on-prem clusters
  • Built full stack test plans with Phantom.js and Selenium to ensure smooth upgrades across multiple releases

January 2019 - July 2021

Team Lead - Cloud Observability

Elastic

Japan - Remote

  • Managed for a global 7-person team that handles TBs/day of incoming observability data
  • Prioritized incoming work and negotiate feature requests with the larger Elastic organization
  • Built cross-team tech demos for knowledge sharing and aligning longer term strategies
  • Made compensation decisions and coordinated team schedules
  • Prototyped new techniques and technologies for increasing team scale and productivity

August 2016 - January 2019

Principal SRE

Elastic

Japan - Remote

  • Built and launched the first Google Cloud Platform regions for Elastic Cloud
  • Introduced Terraform for management of Infrastructure as Code. Grew codebase beyond 50k LOC
  • Created fio-based disk performance test suite and used it with AWS engineers to certify i3 instances for customer use
  • Deployed and maintained Elasticsearch container management across 4 cloud providers (AWS, GCP, Azure, IBM), 50+ regions
  • Maintained and extended Scala APIs and Golang clients for orchestrating thousands of customer Elasticsearch deployments

August 2015 - August 2016

Reliability Engineer

Freelance

Japan - Remote

  • Improved AWS site reliability through automation using Chef, Consul, Terraform and CloudFormation
  • Reduced downtime by creating new insights into key performance metrics using Grafana, Graphite, Prometheus, and CloudWatch
  • Exercised backup and recovery plans for data storage systems on PostgreSQL and Cassandra
  • Optimized cloud costs through performance analysis and full-stack testing

December 2014 - July 2015

Reliability Engineer

Stellar Development Foundation

Japan - Remote

  • Developed and maintained infrastructure and delivery pipelines for large open source projects
  • Improved operational insight using the latest in monitoring tools
  • Implemented cutting edge deployment scenarios leveraging Docker, Packer and Ansible

April 2013 - December 2014

Reliability Engineer

Netflix

San Jose, CA

  • Built new tools and visualizations using the NetflixOSS stack to increase system reliability
  • Analyzed deep causes of production failures and provided reports for the entire company
  • Improved company-wide processes and tools for in-depth monitoring and automated continuous delivery
  • Led calls and guide engineering teams through triage and resolution of complex production issues that affected millions of customers
  • Educated new and veteran teams on constantly evolving best practices when running Java web technologies such as Tomcat, Netty, Guice and Jersey

September 2012 - May 2013

Adjunct Faculty

University of Pennsylvania

Philadelphia, PA

  • Designed and taught CIS 196 - Web Development with Ruby on Rails
  • Coached students through development of custom web applications
  • Mentored TAs through coursework evaluation and grading

September 2010 - February 2013

Co-Founder, Developer

Mashion, LLC

Philadelphia, PA

  • Automated provisioning and deployment of large scale systems with Chef
  • Developed custom Rails and Sinatra applications running on Heroku and EC2
  • Managed business development, sales pipeline, accounting and payroll for a growing consultancy

March 2010 - September 2010

Senior Developer

Hoopla Software

Glen Mills, PA

  • Managed product backlog and requirements using Agile practices and Pivotal Tracker
  • Optimized development workflow using Rake, ERB, Node.js and jQuery
  • Coordinated provisioning of Rails, MySQL and Redis using Chef

April 2007 - March 2010

Senior Software Engineer

Comcast Interactive Media

Philadelphia, PA

  • Developed rich in-browser applications such as MyDVR and TiVo Scheduler using jQuery as well as standards-based HTML and CSS
  • Organized daily scrums, product backlog and sprint planning sessions as the Scrum Master for the cross-platform product team
  • Implemented prototypes for build-time JavaScript optimizations using Ruby and Maven

August 2005 - May 2007

Programmer Analyst

Comcast

Philadelphia, PA

  • Developed a beta-tester community portal using Linux, Apache, MySQL, and PHP
  • Worked with vendors including Cisco, Netgear, RCA and Nintendo to fix TCP issues using tools such as Wireshark and Cacti

June 2004 - July 2005

English Teacher

Aeon

Ichinomiya, Japan

  • Designed lesson plans, and taught English classes for students ranging from beginner to advanced speaking levels
  • Interviewed prospective students and counseled existing students on study and organizational techniques

November 2003 - May 2004

Product Tester

Comcast

Philadelphia, PA

  • Deployed server software such as Bugzilla and WS-FTPd to expedite testing processes
  • Designed testing strategies to be used in product certification for products ranging from cable modems to video email systems
  • Educated coworkers on bug tracking and test script creation practices

September 2000 - December 2002

Systems Engineer

Comcast Business Communications

Philadelphia, PA

  • Researched and implemented a Linux-based small business server product containing Squid, Samba and Sendmail systems
  • Managed distributed development in Perl and PHP with developers from Sun Microsystems
  • Built an email system using Sendmail and Bash to handle the large scale @Home network migration without losing customer email

Volunteering

July 2014 - Present

Volunteer SRE

Safecast

Japan - Remote

  • Maintain data ingest and storage pipeline for nearly 200 million environmental measurements
  • Train new volunteers on software development and operations
  • Research and implement new approaches for high-scale geographical data processing

January 2019 - Present

Volunteer Firefighter

9th Ward

Hokuto, Yamanashi, Japan

  • Readiness drills and equipment testing
  • Community outreach and fire watch patrol

Speaking

2021

RubyKaigi Online

10 years of Ruby-powered citizen science

2017

RubyKaigi Hiroshima

Mapping your world with Ruby

2017

ElasticON Tokyo

Elastic Cloud Deep Dive (Japanese)

2014

SRECon

Disaster Preparedness and Recovery

2012-2013

UPenn

Web development with Ruby on Rails

2011, 2012

Wharton Web Conf

Ruby on Rails, Infrastructure as code with Chef

Education

BS, Computer Science

December, 2003

Drexel University

Philadelphia, PA

Tools & Skills

  • Elasticsearch
  • Kibana
  • NetflixOSS
  • AWS & GCP
  • Golang
  • TypeScript/JavaScript
  • React
  • Java
  • Scala
  • Docker
  • PostgreSQL
  • Ruby
  • Python
  • Bash/ZSH
  • Nginx
  • Git
  • Ansible
  • Graphite
  • Linux
  • REST
  • HTML/CSS
  • TDD/BDD
  • Agile
  • DevOps

OSS Contributions

Kibana
developing new features and fixes
Elasticsearch
maintenance and planning for new features
Grafana
CloudWatch metrics and map rendering
PlantUML
diagramming Elastic Stack components
JRuby
C extension support

Samples available online at github.com/matschaffer.