Adopting an architecture that meets specific user requirements during setup, you can guarantee optimal performance from your Amazon Redshift cluster. Let us take a look at some of the architectural choices that are available to manage workload and steer clear of outages.
(more…)Big Data
Building Covid-19 Twitter Data Aggregation Platform with PySpark CLI
In a previous blog post, we explained how you can use PySpark CLI to jumpstart your PySpark projects. Here, I’ll explain how it can be used to build an end-to-end real-time streaming application.
(more…)
IoT Analytics: The New Frontier in Business Intelligence
The data engineering that precedes analytics was covered in our previous post, Data Engineering: The Heavy Lifting Behind IoT.
Among the many sobriquets that the Internet of Things has acquired, none is more expressive than the term “Internet of Insights.” Incontestably, industrial IoT’s claim to fame is the visibility it brings to previously inaccessible phenomena. The combination of IoT and cutting-edge analytics has pushed the frontiers of business intelligence, emboldening business organizations with better contextual awareness and predictive capabilities.
Data Engineering: The Heavy Lifting Behind IoT
This post is part of our continuing blog series on the Internet of Things. In our previous posts, we discussed sensors, wireless technologies in IoT, and Connected Operations: 3 IoT Scenarios.
Smart cities, self-driving cars, intelligent machines—the IoT market is exploding with “Things.” The ease with which they cross over from sci-fi to real life makes it look like a breeze, thanks in part to data engineers who do the heavy lifting behind the scenes.
Database Use Cases: Finding the Perfect Fit for Applications
The connected world may be shrinking by the day, but the digital universe is expanding at a mind-boggling rate. Organizations now handle data in the range of terabytes and petabytes. This data looks nothing like what RDBMS traditionally dealt with. New distributed databases, known by the umbrella term NoSQL, help in the efficient handling of this unstructured and scaling data.
Processing Big Data Part II: Orchestrating Spark Jobs with Azure Data Factory
In Part 1 of this series, we learnt how to set up a Hadoop cluster on Azure HDInsight and run a Spark job to process huge volumes of data. In most practical scenarios, however, such jobs are executed as part of an orchestrated process or workflow unless the need is for a one-time processing. In our specific use case, we had to derive different metrics related to error patterns and usage scenarios from the log data and report them on a daily basis.
Processing Big Data with Azure HDInsight and Spark: Part I
Before we delve into the interesting part, let me set the context first. The problem we had in hand was to do some data crunching on the log data for one of our client applications, to analyze and report on the various client-defined metrics from the application logs. The application under consideration had a user base of more than 100K users, which meant millions of rows of data to process on a daily basis. Clearly, we were dealing with “big data.” Considering the volume of data involved, we decided to go with Spark running on an Azure HDInsight cluster to benefit from the increased performance offered by Spark’s in-memory RDDs (Resilient Distributed Datasets).
Medical Wearables: Creating a World of Convenience for Patients and Physicians
In the face of growing healthcare challenges such as an aging population, chronic diseases, and high cost of hospitalization, wearable patient monitoring (WPM) systems create new opportunities for improving patient care.
From modish wearables that track general fitness, these systems have matured to medical-grade devices that can monitor chronic diseases and other medical conditions. Wearables fitted with advanced biosensors and integrated with a robust IoT platform for analysis and communication constitute a potential solution for early detection of clinical deterioration, timely response by medical staff, and appropriate medical intervention. (more…)
Pagination in D3 Tree Layouts
Hierarchical data visualized in a collapsible tree format can grow rapidly and fill the screen with nodes, compromising on readability. Particularly so when a node has hundreds of child nodes. This article proposes a pagination mechanism where a fixed number of nodes are displayed at a time, and the user is allowed to move between previous and subsequent nodes. It describes the implementation of pagination in D3.js.
Living the Automated Life in Your Home [Infographic]
The Internet of Things is about all things around us connected and communicating to make our lives simpler and efficient. You can control your home using your mobile phone even from the other end of the world. This infographic depicts some of the cool things that happen in a smart home.
(more…)