Raka's

Introducing Glazplatova 1.0

NEWSFLASH
---------
We delve into the architecture of Glazplatova in another video, here: Attn. Programmers: know some devops stuffs. It's good for you.
---------

Introducing Glazplatova 1.0 (Глаз Платова 1.0), a twitter crawling-machine I wrote and dub as "adaptive-reactive crawler" (yea, jargon-ridden). But it's true, it's adaptive in the sense that its architecture allows processor plugged-in to change the queries in real-time (adapting to latest finding in the data accumulated so far). Reactive in the sense that..., well, I use Reactive Programming techniques which enables us to create some nifty scheduling, quite elegantly. I use RxJS. This video is just to give a feel of what it is (we are using it for our media-monitoring and social-network analysis). Later I'll try to find an opportunity to dive down into the architecture. Uses: Mongo, RabbitMQ, Redis, RxJS.

Introducing Glazplatova 1.0 (Глаз Платова 1.0) from Cokorda Raka Angga Jananuraga on Vimeo.

What should we expect from a DevOps?

I was wondering.

First of all, supposedly a DevOps is this magic person who resides in the goldilock zone :) I mean, an intersection between programmer (who knows the code of the system she's managing), QA (really necessary?), and system administration (biggest portion of her works would be dealing with shits like hadoop, microservices, database, forward-proxy... shits like that).

I'm asking this question (or have been asking this question), because until I know I'm basically all-the-things (UX expert, product designer, software architect, programmer, QA (barely, not much time left)..., and including DevOps... I decide all the mix of stuffs we have in the platform (hadoop, druid, mongo, nginx, postgre, spark, loopback), define their dockers, including now writing a web application to automate the creation of those dockers... you know, PaAS-style).

NEWSFLASH
---------
I also made the continuation of this blogpost, here: Attn. Programmers: know some devops stuffs. It's good for you.
---------

Now, the troubleshooting. What happened today; the dashboard is not showing numbers of this day.

Checked the front-end, all seems ok, no error in the browser.
Checked the database, we have data arriving.
Checked the batch process (that performs aggregation), it performs correctly its job; it pulls the data out of the database, transform them, and send them off to Hadoop (the file-system, not the map-reduce cluster).
Checked the Druid, it's up and running. But..., turns out it fails when it tries to send cube-building tasks to Hadoop.
Checked the connection between Druid node and Hadoop; seems fine, I could ping both ways.
Checked again in Druid's, more detail into the log, and it says "failed to communicate with Hadoop service on port 8032".
Checked inside Hadoop's node, check if there is a running service that binds to 8032, netstat -plnt. Nope, it's down. 8032 is YARN.
From that point it was clear what to do: identify why it went down (space issue), and bring it back up. Problem solved.

Now.., I was able to do that without much difficulties because of two things:

My familiarity with linux things, at least the basic system administration tasks. But that we can / should expect from any DevOps. That's the very least we should expect from them. So that shouldn't be an issue.
My knowledge of the system; how the things connect and interact (hadoop, druid, mongo, batch-data-transformation, etc). Should we expect that knowledge from the DevOps (who doesn't write the code of all those things)? In this case, it was pretty simple (or maybe not), from the log of Druid, that it wasn't able to communicate with YARN of Hadoop. But what about less obvious factors, which requires deeper knowledge of the log, or even some knowledge of how the code works?

Maybe the answer is:

Sharing the code with the DevOps (so we should really expect with some experience as programmer)?

And if the DevOps is in the team during development, maybe involve her in code-review? Or at least design-review?

Introduction to Reactive Programming in JavaScript with RxJS

Slides "Visual Programming with Blockly", GdlJS 28 Feb 2017

Download here: Slides "Visual Programming with Blockly"

Preliminary video:

Expression Builder using Blockly from Cokorda Raka Angga Jananuraga on Vimeo.

Starting series on reactive micro services system, using EventSourcing and CQRS techniques

Just started a couple of months ago my new journey on EventSourcing and CQRS, through this blog I will share the progress of it. Here are two videos on the topic I have created, another video on the way in a week or two.

The first one is about the foundation of EventSourcing and CQRS framework in Scala (using Actor programming model). It was presented in Guadalajara Java User Group ( http://www.meetup.com/gdljug/events/230324875/ ). Source code of this presentation is available here.