In Part 1 of this series, we learnt how to set up a Hadoop cluster on Azure HDInsight and run a Spark job to process huge volumes of data. In most practical scenarios, however, such jobs are executed as part of an orchestrated process or workflow unless the need is for a one-time processing. In our specific use case, we had to derive different metrics related to error patterns and usage scenarios from the log data and report them on a daily basis.
Processing Big Data with Azure HDInsight and Spark: Part I
Before we delve into the interesting part, let me set the context first. The problem we had in hand was to do some data crunching on the log data for one of our client applications, to analyze and report on the various client-defined metrics from the application logs. The application under consideration had a user base of more than 100K users, which meant millions of rows of data to process on a daily basis. Clearly, we were dealing with “big data.” Considering the volume of data involved, we decided to go with Spark running on an Azure HDInsight cluster to benefit from the increased performance offered by Spark’s in-memory RDDs (Resilient Distributed Datasets).
What Every Developer Needs to Know About GDPR Compliance
When was the last time you went through a privacy policy update? When everyone, including the likes of Google and Twitter, starts sending you updates, it’s worth having a look. Because, we have a new rule in place, the General Data Protection Regulation or GDPR. It is going to affect the way we think about a project and the way we code!
Medical Wearables: Creating a World of Convenience for Patients and Physicians
In the face of growing healthcare challenges such as an aging population, chronic diseases, and high cost of hospitalization, wearable patient monitoring (WPM) systems create new opportunities for improving patient care.
From modish wearables that track general fitness, these systems have matured to medical-grade devices that can monitor chronic diseases and other medical conditions. Wearables fitted with advanced biosensors and integrated with a robust IoT platform for analysis and communication constitute a potential solution for early detection of clinical deterioration, timely response by medical staff, and appropriate medical intervention. (more…)
Blockchain Technology & Bitcoin Cryptocurrency: Part 3
Some smart guy once said the devil is in the details, and it is the details we’re getting down to in this post, the last in our series on Blockchain Technology & Bitcoin Cryptocurrency.
We shall go through the following:
- What a Bitcoin transaction looks like
- Scripting languages used
- Data structure
Blockchain Technology & Bitcoin Cryptocurrency: Part 2
We went through the building blocks of blockchain technology in Part 1. There are a few other concepts that are integral to blockchain and Bitcoin. Let’s go through them one by one in this second post on Blockchain Technology and Bitcoin Cryptocurrency.
- Decentralization
- Distributed Consensus Protocol
- Double-Spend Attack
- Proof of Work
- Incentive Mechanisms (more…)
Blockchain Technology & Bitcoin Cryptocurrency: Part 1
In a three-part series on Blockchain technology and Bitcoin Cryptocurrency, our bloggers unravel the technology underlying Bitcoin, which is currently hogging the headlines globally. Part 1 explores the basics of blockchain technology. Part 2 dwells on the protocols and concepts in Bitcoin. Part 3 takes a closer look at the concepts covered in Part 1 and 2. A minimal understanding of cryptography—its basic concepts and terms—will make your journey through these posts effortless.
Five Digital Technologies for Your Enterprise in 2018
The precipitous rise in digital data and the subsequent growth of automated machine intelligence has engendered the latest digital business transformation. As we enter 2018, enterprises find themselves at different ends of the transformation spectrum. While some have become pathfinders, more staid enterprises are still waiting at the threshold.
Many are plagued by questions such as, What digital technologies to embrace in 2018 and how to tap them effectively?
I reached out to some of our senior architects and engineers and what follows is their take on what enterprises will need in their arsenal in the new year. (more…)
Go SAFe. Be SAFE.
Years ago when I joined, QBurst was a start-up with just 100 employees. Being fast-paced in technology adoption and service delivery, we adopted agile methodology early on—specifically Scrum. Now, as the company has grown to over a thousand people, we are in need of scaling agile. And that is what brings me to Scaled Agile Framework, aka SAFe! (more…)
Deploying Microservices on AWS with Infrastructure as Code
One of the major IT trends in recent years has been the increase in automation and a simultaneous decrease in the need for manual intervention. Another big hit is ‘microservices’.
Recently, my team came up with the idea of utilizing these key technological developments to build a flawless infrastructure that can be continuously improved and monitored with minimal manual effort.
While numerous IT groups are still dependant on manual steps, custom scripts and obsolete tools, the DevOps team at QBurst took to automating infrastructure management. With this, we were able to oversee the infrastructure, reduce errors, and perform seamless deployments and monitoring.
In this blog, I would like to introduce the concept of Infrastructure as Code (IaC) using the example of deploying a microservice to Amazon ECS. (more…)