Central Component for #datadrivenarchitecture: Real-time Insights powered by Reactive Programming #netflix

Can help with:

  • testing
  • debugging
  • security

Solution:

  • log everything

Problems:

  • so much data
  • so many devices
  • not feasible to save to elasticsearch first (real time!)

Stream analysis with reactive programming

In a data driven architecture the processors for the high performance message bus benefit from being written in Rx.

Who use es it?

users of rxjs: netflix, google, facebook and more

Existing Solutions:

Differences

  • Mantis is designed for operational use cases where message guarantee levels vary by jobs. So, some jobs can choose at-most once guarantees while others choose at least once guarantees via Kafka. We are able to saturate the NIC on the servers for operational use case with very little CPU usage.
  • Bulit-in back pressure that allows Mantis to seamlessly switch between push, pull or mixed modes based on the type of data sources
  • Support a mix of long running perpetual analysis jobs along with user triggered short lived queries in a common cluster
  • Since the volume of data to be processed at Netflix varies tremendously by time of day being able to autoscale workers in a job based on resource consumption & the ability to scale the cluster as a whole was a key requirement. None of the existing streaming frameworks provided such support.
  • We wanted more control over how we schedule the resources so we can do smarter allocations like bin packing etc. (that also allows us to scale the jobs)
  • deep integration with Netflix ecosystem that allows filtering event stream at the source of data.
    among others

Sources:

NPM vs Gulp – Do I need a build javascript system besides npm scripts

#NPM vs Gulp – Do I need a build javascript system besides npm scripts

First when I look at npm vs gulp I was like, well I can do this with npm scripts, why to learn another tool.
So I tried to make it happen with just npm scripts, this worked great until I tried to automatically create
responsive images with imagemagick. This would have required me to write a complete bash script.

Advantage npm script:
– I just write my build steps as shell commands and I am independent of the lifecycle of certain tools like gulp. As the bash script is portable.
– I minimize the tooling I need to learn. No need to install an additional command-line tool to run your build.
– They don’t have the complexity of a full build tool.

Advantage gulp:
– A tool like Gulp gives you the full power of Node and JavaScript.
– They provide consistent APIs optimized for composing build tasks.
– They offer huge ecosystems of plugins for almost anything you can imagine.
– They avoid cross-platform compatibility problems.
– Gulp is javascript and no additional DSL.
– Gulp increases the performance by doing the build steps in parallel.

So I see that gulp is not so much more to learn as I don’t have to learn a tool with it’s own language. So my decision is I use the best tool for the job if it is in java or javascript.

My solution:

  1. I learn gulp and write my build tasks in javascript.
  2. I include gulp in my npm scripts with the following code into my ‘package.json’:
{
  "scripts": {
    "build:prod": "gulp build --prod",
    "build:dev": "gulp build"
  }
}

Setting up your MEAN.js Develeoper Environment with Vagrant, Chef and Berkshelf

First you have to get started developing your Infrastructure Environment. An Up to Date Guide can be found at safaribooksonline.com. The following procedure will get your local workstation configured and ready to develop cookbooks with Test Kitchen, Vagrant and Chef-zero. Note that this is not a tutorial on how to write cookbooks.

This is just a short discription on the progress I made and not complete. Since I switch from the MEAN.js stack to the jHistper Stack (Java (Spring) in the Backend, AngularJS and Bootstrap in the Frontend, and a variety of databases) I won't continue this. Hope this still helps someone.

The Tools

Vagrant – We use Vagrant and Virtualbox to test our cookbook code and run our tests on our local workstations. The creator of Vagrant wrote the definitive guide for us all.

Test Kitchen – Test Kitchen is the tool we use to manage our Vagrant configurations, configure our environment to mirror Production in a meaningful way and run our tests.

Chef-Zero – Chef-Zero is a lightweight local Chef Server that our testing VMs use to mimic our live Chef environment.

ChefDK – The Chef Development Kit provided by Opscode bundles various Chef tools together. The beauty of ChefDK is that it comes with an embedded Ruby environment and the necessary Ruby Gems required by the tools. This is a great improvement if you’ve ever had to deal with pre-ChefDK local Ruby/RVM configurations, fluctuating gem dependancies and breaking changes between tools.

Getting Vagrant, ChefDK and Tools

ChefDK Installation

First, download and install ChefDK from https://downloads.getchef.com/chef-dk. ChefDK comes with chef-client, Berkshelf, Test Kitchen, ChefSpec, Foodcritic and more.

Vagrant Installation and Configuration

Next, install Vagrant. Head over to https://www.vagrantup.com/downloads.html and download and install the appropriate package for your system.

Once you have Vagrant installed you’ll run the following commands in your (bash) shell to install the supporting plug-ins that we need to integrate Vagrant and Chef.

vagrant plugin install vagrant-berkshelf
vagrant plugin install vagrant-omnibus
vagrant plugin install vagrant-chef-zero
vagrant plugin install vagrant-vbguest

After installing these, you should logout of your shell and relaunch it to pickup the new environment variables.

Virtualbox Installation and Setup

Next, download and install VirtualBox from https://www.virtualbox.org/wiki/Downloads.

Follow the directions for your platform but skip the VMWare portion and go to the Test Your Setup section below if you’ve opted to use Virtualbox.

Alternately, VMWare installation and Setup

(only do this if you use VMWare rather than Virtualbox)

Download and install VMWare from the web site and have your Vagrant VMWare license handy. You need to purchase the Vagrant VMWare integration license from HashiCorp (the company that makes Vagrant).

Copy your Vagrant VMWare license to ~/.vagrant.d/vmware-license.lic then follow the steps for your specific setup:

For VMWare Workstation, add the following to your ~/.bashrc file ( or ~/.bash_profile on OS X):

export VAGRANT_DEFAULT_PROVIDER=vmware_workstation

Then run:

vagrant plugin install vagrant-vmware-workstation
vagrant plugin license vagrant-vmware-workstation ~/.vagrant.d/vmware-license.lic

For VMWare Fusion, add the following to your ~/.bashrc file ( or ~/.bash_profile on OS X):

export VAGRANT_DEFAULT_PROVIDER=vmware_workstation

Then run:

vagrant plugin install vagrant-vmware-workstation
vagrant plugin license vagrant-vmware-workstation ~/.vagrant.d/vmware-license.lic

lly, everyone using VMWare should reboot their machine and launch VMWare for the first time, accepting the license agreement and allowing it to initialize itself once. Do this before running Vagrant. FYI: Vagrant has plans to consolidate Fusion and Workstation at some point in the near future.

Finally, everyone using VMWare should reboot their machine and launch VMWare for the first time, accepting the license agreement and allowing it to initialize itself once. Do this before running Vagrant. FYI: Vagrant has plans to consolidate Fusion and Workstation at some point in the near future.

Prepare Chef Tools

As you work more with Chef, you will eventually run into repositories that depend on Chef recipes that require your a broad range of tools to be available and configured on your workstation. Run through the following steps now so they are available when you need them.

Configure Knife

(do this only if you don’t have a knife configuration already)
Setup Knife and accept the defaults:

knife configure

You must place your validation key in the following location before generating instance data with Knife:

/etc/chef-server/chef-validator.pem

Create .pem File

Create a .pem for Chef Zero to use when faking Chef server for local configuration.

openssl genrsa 2048 > ~/.chef/<username from knife configure>.pem

Test Your New Setup

You should now create a temporary directory locally for Vagrant VMs and then test that you can create a new Vagrant project. There are many GitHub projects with Vagrant file templates. I recommend that you read up on Vagrant and create your own simple configuration file but if you want to test you can use the one in the next step.

Create an empty directory (~/vagrant_vm, say) for the configuration. This location will contain a Vagrantfile and will eventually contain the running VM as well.

you@workstation$ mkdir ~/vagrant_vm
you@workstation$ cd ~/vagrant_vm

Preparing your Vagrantfile

Copy a Vagrant file from a GitHub repo that suits you and put it in a new file called Vagrantfile in the ~/vagrant_vm directory. (You must name the file Vagrantfile. You cannot always leave it the same name as it is in GitHub.) You can find some generic Vagrantfile examples and the ssh key for the images here: https://github.com/patrickdlee/vagrant-examples.

Or you can use the example one below:

#Quick vagrant file for testing
#
Vagrant::Config.run do |config|
  config.vm.box = 'precise32'
  config.vm.box_url = 'http://files.vagrantup.com/precise32.box'
end

Using the Virtual Machine

Next, launch the VM from the ~/vagrant_vm directory using vagrant up:

you@workstation$ vagrant up

There will be lots of output and once the machine has booted you can ssh to it.

you@workstation$ vagrant ssh

Vagrant exports ~/vagrant_vm from your local your local machine to the VM. Check that you see the the Vagrantfile and .vagrant directories in the VM by running:

sbo@linux:~$ ls -la /vagrant
total 16drwxrwxr-x  1 sbo  sbo  4096 Oct 10 16:03 .
drwxr-xr-x 23 root root 4096 Oct 10 16:05 ..
drwxrwxr-x  1 sbo  sbo  4096 Oct 10 13:56 .vagrant
-rw-r--r--  1 sbo  sbo  1150 Oct 10 16:03 Vagrantfile

Exit from the VM

sbo@linux:~$ exit

Source: http://stackengine.com/devops-delight-chef-dk-chef-zero-vagrant/

Authoring a Cookbook

Each of the individual parts is now installed and shown to work. So lets configure a machine using chef. To do this we will:

  • Write a cookbook to install some of our favorite packages that should be on every machine
  • Run Foodcritic

Scaffolding a Cookbook with Berkshelf

Berkshelf is the “package manager” for Chef cookbooks. It is also the standard way to scaffold a cookbook with ChefDK.

Generate a Berksfile in a pre-existing cookbook
$ cd cookbooks
$ berks init .

https://www.youtube.com/watch?v=0sPuAb6nB2o