Central Component for #datadrivenarchitecture: Real-time Insights powered by Reactive Programming #netflix

Can help with:

  • testing
  • debugging
  • security

Solution:

  • log everything

Problems:

  • so much data
  • so many devices
  • not feasible to save to elasticsearch first (real time!)

Stream analysis with reactive programming

In a data driven architecture the processors for the high performance message bus benefit from being written in Rx.

Who use es it?

users of rxjs: netflix, google, facebook and more

Existing Solutions:

Differences

  • Mantis is designed for operational use cases where message guarantee levels vary by jobs. So, some jobs can choose at-most once guarantees while others choose at least once guarantees via Kafka. We are able to saturate the NIC on the servers for operational use case with very little CPU usage.
  • Bulit-in back pressure that allows Mantis to seamlessly switch between push, pull or mixed modes based on the type of data sources
  • Support a mix of long running perpetual analysis jobs along with user triggered short lived queries in a common cluster
  • Since the volume of data to be processed at Netflix varies tremendously by time of day being able to autoscale workers in a job based on resource consumption & the ability to scale the cluster as a whole was a key requirement. None of the existing streaming frameworks provided such support.
  • We wanted more control over how we schedule the resources so we can do smarter allocations like bin packing etc. (that also allows us to scale the jobs)
  • deep integration with Netflix ecosystem that allows filtering event stream at the source of data.
    among others

Sources:

DOING THE REACT TUTORIAL | LIVE CODING #1 | BUILDTHEFUTURE

Modern application are delivered with web technologies – no discussion. Congratulations Javascript you have won. And with React and it’s virtual DOM the performance increased to a level that is close to native. So I decided to get my hands dirty and do the official react tutorial. Hope you enjoy this guys.

Reduce Poverty in Inner Cities

An Overview of which factors are making cities smarter and why, what’s possible and how and what we can expect, and when.

Smarter cities are providing a myriad of benefits for their people, businesses and governments:
*an improved quality of life,
*sustained economic vitality,
*and intense communication through social media.


Michael Dixon leads IBM’s strategy and its implementation for Smarter Cities around the world. Prior to a brief period as the Managing Partner for IBM’s consulting and systems integration business (GBS) in Central/Eastern Europe in 2011, Michael was the Vice President of Public Sector for GBS in the Growth Markets, leading teams focused on supporting client initiatives in Government, Healthcare Payers and Providers and Education across Asia Pacific, the Middle East, Africa, Central / Eastern Europe and Latin America. Client success in numerous projects resulted with IBM providing strategic advice, business change and process transformation to improve services and deliver cost efficiencies.
Before joining IBM Michael completed a BBSc at La Trobe University, an MSc at the University of Melbourne and a PhD at the University of Alberta. Michael is an Australian and spends his time in Melbourne, New York and all parts of the world.

BETA [WIP] Execution Framework for Daily Workload

As I described I think we all need to be agile and pivoting like startups to make the best out of our resource. When we do we have the chance to change the world. One small task at a time.

The following presentation gives you an overview how J.D. Meier, a project manager at microsoft adapted Agile to a productivity method.

http://www.30daysofgettingresults.com/2011/10/day-7-setup-boundaries-and-buffers.html

I made the workflow somewhat more concrete:

For me the core value is that a task should create something tangible. Only code in the repository exist. (No speed limits!)

  1. COLLECT everything into an Inbox: This is very important as it allows you to have an Inbox Zero, stops your from the task of keeping things in your mind but just dump it into your Inbox (For me this is an Evernote Folder INBOX).
  2. ORGANIZE:
    1. If it is not related to a task delete it. Or if possible don’t even put it into the inbox. The management of an repository of information is to expensive. Use social networks.
    2. If it can be done in less than 2 min do immediately and archived item in BACKLOG (add a reminder and instantly mark as done).
    3. Else create a project for it and answer two questions What is my goal? Why is this important to me? If it is not important it I add an reminder and directly goes into backlog. If it is important I tag it with importantx as well as the area where it belongs.
    4. Monday Vision: Check your Hierarchy of GOAL & Backlog what is relevant for this week, and select 3 result for this week: “If this were Friday, what are three results I want under my belt?”
    5. Daily Win: Check the Monday Vision for 3 result of today: ““What are the three things I really want to accomplish today?”
  3. EXECUTE: Set of values, experiences and best practices.
  4. REVIEW: Daily scrum, friday reflection

The most important part are the VALUES of the doing & the executing to apply:

  • Only tangible results count (“Writing code”)
  • It is about feedback loops and iteration: If you give the product to the customer he will give you feedback. If you don’t progress you either not produce or you produce nothing good.
  • Fail fast & Done is better then perfect

3d printable cloth

Downloadable, printable clothing may be coming to a closet near you. What started as designer Danit Peleg’s fashion school project turned into a collection of 3D-printed designs that have the strength and flexibility for everyday wear. “Fashion is a very physical thing,” she says. “I wonder what our world will look like when our clothes will be digital.”

Why I will focus more on the human in the future of industrie & living

In the last years and month I kept my eye on the areas of Industrie 4.0 or Industrial Internet. While I will continue this journey, and will refocus more on the area around lifestyle and healthcare and consumer products. Why?

I definitely believe that there is large potential in Industrie 4.0, I personally am quite surprised with the experience how long it takes until an idea makes it to pilot or even production in the enterprise it. In the consumer area the cycle times are much lower.

But there are two more important reasons. First I enjoy helping people, and when I thought about my dream when I was a little boy, I dreamed to help people make the world a better place by making them more happy and helping them to be healthy. Well back then I dream of healing every diseases and fixing all the world problems form climate change to mass hunger. Most likely this will stay a dream, but I love the idea to contribute a small small piece to something meaningful.

I always believe I could do something meaningful, and this does not mean to revolutionise the world, only few of us are luck enough to experience this. But do something meaningful means to putting your head to a meaningful challenge and throwing punches to wear down the challenge for a heavy hitter to punch it out.

But do something meaningful means to putting your head to a meaningful challenge and throwing punches to wear down the challenge for a heavy hitter to punch it out.

So we come to the third reason: I am currently working at IBM, and one reasons I chose to work for IBM was that they brand themselves as a company that contributes something meaningful the the solutions of large problems. Currently I integrate systems at big banks, while this is important, it seams quite meaningless compared to curing cancer or alzheimer’s or any other disease. But as it seems we IBM has not given up on making meaningful contributions. When they announced in April 2015 that “healthcare will be our moonshot”, I knew that I have to be a part of this. Therefore I will refocus on this area, and learn what is important about this topic and how it can be combined with the Internet of Things. I see a large overlap between Industrie 4.0 and the consumer side. I think we will get a much larger acceptance for Industrie 4.0 to be not just another buzzword, when we focus on the customer and identify areas where we can generate immediate benefit to the people.

What this will mean for my focus areas, well I will have to keep learning to develop and maintain apps on a cloud platform. This includes connecting things to the cloud with Arduino & Node.JS. Those Applications will follow a modern agile Microservice architecture. But modern application need also some analytics to generate value from data. To do this I will useApache Spark, Java, Python, R and Statistics.

This will be continued!