Central Component for #datadrivenarchitecture: Real-time Insights powered by Reactive Programming #netflix

Can help with:

  • testing
  • debugging
  • security

Solution:

  • log everything

Problems:

  • so much data
  • so many devices
  • not feasible to save to elasticsearch first (real time!)

Stream analysis with reactive programming

In a data driven architecture the processors for the high performance message bus benefit from being written in Rx.

Who use es it?

users of rxjs: netflix, google, facebook and more

Existing Solutions:

Differences

  • Mantis is designed for operational use cases where message guarantee levels vary by jobs. So, some jobs can choose at-most once guarantees while others choose at least once guarantees via Kafka. We are able to saturate the NIC on the servers for operational use case with very little CPU usage.
  • Bulit-in back pressure that allows Mantis to seamlessly switch between push, pull or mixed modes based on the type of data sources
  • Support a mix of long running perpetual analysis jobs along with user triggered short lived queries in a common cluster
  • Since the volume of data to be processed at Netflix varies tremendously by time of day being able to autoscale workers in a job based on resource consumption & the ability to scale the cluster as a whole was a key requirement. None of the existing streaming frameworks provided such support.
  • We wanted more control over how we schedule the resources so we can do smarter allocations like bin packing etc. (that also allows us to scale the jobs)
  • deep integration with Netflix ecosystem that allows filtering event stream at the source of data.
    among others

Sources:

Fixing homebrew permissions after osx update

The problem:

[code language=”bash”]
brew update
error: unable to unlink old ‘.gitignore’ (Permission denied)
error: unable to unlink old ‘.yardopts’ (Permission denied)
error: unable to unlink old ‘README.md’ (Permission denied)
error: Your local changes to the following files would be overwritten by merge:
.gitignore
README.md
Please, commit your changes or stash them before you can merge.
Aborting
Error: Failure while executing: git pull –quiet origin refs/heads/master:refs/remotes/origin/master
[/code]

The solution:

[code language=”bash”]
#sudo chown denbox:admin /usr/local;
sudo chown $(USER):admin /usr/local
[/code]

Sources:
https://github.com/Homebrew/homebrew/issues/41665
http://osxdaily.com/2013/04/23/change-file-ownership-mac-os-x/

BETA [WIP] Execution Framework for Daily Workload

As I described I think we all need to be agile and pivoting like startups to make the best out of our resource. When we do we have the chance to change the world. One small task at a time.

The following presentation gives you an overview how J.D. Meier, a project manager at microsoft adapted Agile to a productivity method.

http://www.30daysofgettingresults.com/2011/10/day-7-setup-boundaries-and-buffers.html

I made the workflow somewhat more concrete:

For me the core value is that a task should create something tangible. Only code in the repository exist. (No speed limits!)

  1. COLLECT everything into an Inbox: This is very important as it allows you to have an Inbox Zero, stops your from the task of keeping things in your mind but just dump it into your Inbox (For me this is an Evernote Folder INBOX).
  2. ORGANIZE:
    1. If it is not related to a task delete it. Or if possible don’t even put it into the inbox. The management of an repository of information is to expensive. Use social networks.
    2. If it can be done in less than 2 min do immediately and archived item in BACKLOG (add a reminder and instantly mark as done).
    3. Else create a project for it and answer two questions What is my goal? Why is this important to me? If it is not important it I add an reminder and directly goes into backlog. If it is important I tag it with importantx as well as the area where it belongs.
    4. Monday Vision: Check your Hierarchy of GOAL & Backlog what is relevant for this week, and select 3 result for this week: “If this were Friday, what are three results I want under my belt?”
    5. Daily Win: Check the Monday Vision for 3 result of today: ““What are the three things I really want to accomplish today?”
  3. EXECUTE: Set of values, experiences and best practices.
  4. REVIEW: Daily scrum, friday reflection

The most important part are the VALUES of the doing & the executing to apply:

  • Only tangible results count (“Writing code”)
  • It is about feedback loops and iteration: If you give the product to the customer he will give you feedback. If you don’t progress you either not produce or you produce nothing good.
  • Fail fast & Done is better then perfect