Skip to main content

Posts

Apache Kafka - Integration with Storm and Spark

Integration With Storm In this chapter, we will learn how to integrate Kafka with Apache Storm. About Storm Storm was originally created by Nathan Marz and team at BackType. In a short time, Apache Storm became a standard for distributed real-time processing system that allows you to process a huge volume of data. Storm is very fast and a benchmark clocked it at over a million tuples processed per second per node. Apache Storm runs continuously, consuming data from the configured sources (Spouts) and passes the data down the processing pipeline (Bolts). Com-bined, Spouts and Bolts make a Topology. Integration with Storm Kafka and Storm naturally complement each other, and their powerful cooperation enables real-time streaming analytics for fast-moving big data. Kafka and Storm integration is to make easier for developers to ingest and publish data streams from Storm topologies. Conceptual flow A spout is a source of streams. Fo

Automated Identification and Graphing of SQL Dependencies

All of your queries are running, and your SQL-defined charts are returning results, but when it’s maintenance time, you don’t have a clear picture of the cascade of views, tables, and CSV files that your analysis depends on. At least it’s lucky that we’re good analysts, and we don’t make mistakes like circular references … right. How many of you ran straight for the Git repo? You can sit down and start coding directed acyclical graphs (DAGs) in something like  Airflow , but that’s a laborious task that involves reading through tons of SQL files and recording dependencies. There must be a better way! I searched far and wide and found nothing, so I decided to dust off my Python skills from a previous life. This blog post will walk you through what I wrote, and leave you with a (mostly) working mapper, so you too can identify, clean up, and maintain your SQL ETL. If you want a copy of the script to run on your own set of SQL, email us at  [email protected]  and we’ll send it right o

JavaScript conditions

JavaScript If...Else Statements Conditional statements are used to perform different actions based on different conditions. Conditional Statements Very often when you write code, you want to perform different actions for different decisions. You can use conditional statements in your code to do this. In JavaScript we have the following conditional statements: Use  if  to specify a block of code to be executed, if a specified condition is true Use  else  to specify a block of code to be executed if the same condition is false Use  else if  to specify a new condition to test, if the first condition is false Use  the switch  to specify many alternative blocks of code to be executed The if Statement Use the  if  statement to specify a block of JavaScript code to be executed if a condition is true. Syntax if ( condition ) {     block of code to be executed if the condition is true } Note that  it  is in lowercase letters. Uppercase

JavaScript Overview

JavaScript Overview JavaScript is a lightweight, interpreted programming language with object-oriented capabilities that allows you to build interactivity into otherwise static HTML pages. The general-purpose core of the language has been embedded in Internet Explorer, Firefox, and other web browsers The ECMA-262 Specification defined a standard version of the core JavaScript language. JavaScript is: JavaScript is a lightweight, interpreted programming language Designed for creating network-centric applications Complementary to and integrated with Java Complementary to and integrated with HTML Open and cross-platform Client-side JavaScript: Client-side JavaScript is the most common form of the language. The script should be included in or referenced by an HTML document for the code to be interpreted by the browser. It means that a web page need no longer be static HTML, but can include programs that interact with the user, control the browser, and d