Apache Spark

How to process real-time data with Apache tools

Simon Crosby — Fri, 28 Feb 2020 08:03:00 +0000

How to process real-time data with Apache tools Simon Crosby Fri, 02/28/2020 - 03:03

Open source is leading the way with a rich canvas of projects for processing real-time events.

In the "always-on" future with billions of connected devices, storing raw data for analysis later will not be an option because users want accurate responses in real time…

How to analyze log data with Python and Apache Spark

djsarkar — Tue, 14 May 2019 07:02:00 +0000

How to analyze log data with Python and Apache Spark djsarkar Tue, 05/14/2019 - 03:02

Case study with NASA logs to show how Spark can be leveraged for analyzing data at scale.

In part one of this series, we began by using Python and Apache Spark to process and wrangle our example web logs into a format fit for analysis, a vital technique considering…

How to wrangle log data with Python and Apache Spark

djsarkar — Tue, 14 May 2019 07:01:00 +0000

How to wrangle log data with Python and Apache Spark djsarkar Tue, 05/14/2019 - 03:01

Case study with NASA logs to show how Spark can be leveraged for analyzing data at scale.

One of the most popular and effective enterprise use-cases which leverage analytics today is log analytics. Nearly every organization today has multiple systems and…

20 innovative Apache projects

sallykhudairi — Tue, 26 Mar 2019 07:01:00 +0000

20 innovative Apache projects sallykhudairi Tue, 03/26/2019 - 03:01

As the Apache Software Foundation turns 20, let's celebrate by recognizing 20 influential and up-and-coming Apache projects.

As the world's largest and one of the most influential open source foundations, the Apache Software Foundation (ASF) is home to more than 350 community-led projects and…

jimjag

An introduction to data processing with Cassandra and Spark

dtrapezoid — Wed, 11 May 2016 07:01:00 +0000

An introduction to data processing with Cassandra and Spark dtrapezoid Wed, 05/11/2016 - 03:01

This article is co-authored by Jon Haddad. There's been a huge surge of interest around the Apache Cassandra database due to the increasing uptime and performance demands of…

A guide to Apache's Spark Streaming

arush — Thu, 23 Apr 2015 07:00:00 +0000

A guide to Apache's Spark Streaming arush Thu, 04/23/2015 - 03:00

Apache Spark is an open source cluster computing framework. In contrast to Hadoop’s two-stage disk-based MapReduce paradigm, Spark’s in-memory primitives provide performance…

Spark ignites at ApacheCon

Jen Wike Huger — Mon, 06 Apr 2015 09:00:00 +0000

Spark ignites at ApacheCon Jen Wike Huger Mon, 04/06/2015 - 05:00

ApacheCon is coming up, and within that massive conference there will be a glimmering gem: a forum dedicated to Spark. Reynold Xin is organizing it, and he shared some…

Using Spark DataFrames for large scale data science

rxin — Thu, 26 Mar 2015 11:00:00 +0000

Using Spark DataFrames for large scale data science rxin Thu, 03/26/2015 - 07:00

When we first open sourced Spark, we aimed to provide a simple API for distributed data processing in general-purpose programming languages (Java, Python, Scala). Spark…

World record set for 100 TB sort by open source and public cloud team

rxin — Thu, 15 Jan 2015 10:00:00 +0000

World record set for 100 TB sort by open source and public cloud team rxin Thu, 01/15/2015 - 05:00

In October 2014, Databricks participated in the Sort Benchmark and set a new world record for sorting 100 terabytes (TB) of data, or 1 trillion 100-byte records. The team used…