## glossary

What is Big Data?

Big Data refers to datasets whose volume, velocity or variety exceed the processing capacity of conventional database tools. It is not just "a lot of data" — it is the point where the traditional approach stops working and a distributed architecture becomes necessary.

The distinction is practical: if your data fits comfortably in a well-indexed PostgreSQL on a reasonable machine, you do not have a Big Data problem — you have a database problem, which is much simpler to solve.

The five Vs

Volume — the scale of the data, from terabytes to petabytes.
Velocity — the rate of generation and the need to process in real time, or close to it.
Variety — structured, semi-structured and unstructured data living side by side: tables, JSON, text, image, video, logs.
Veracity — the degree of reliability; data at scale brings inconsistency, duplication and gaps.
Value — the most important and the most forgotten: if it does not lead to better decisions, it is just storage cost.

Why traditional tools are not enough

A relational database on a single machine runs into the physical limits of disk, memory and CPU. Big Data's answer is to scale horizontally: distribute data and processing across dozens or hundreds of commodity machines, instead of buying an ever-bigger one.

Distributed processing: the same logic, across many machines

# PySpark: the operation is declared once and
# executed in parallel by the cluster
from pyspark.sql import functions as F

eventos = spark.read.parquet("s3://dados/eventos/2026/*")

resumo = (
    eventos
    .filter(F.col("tipo") == "compra")
    .groupBy("categoria", F.to_date("data").alias("dia"))
    .agg(
        F.sum("valor").alias("receita"),
        F.countDistinct("usuario_id").alias("clientes"),
    )
    .orderBy(F.desc("receita"))
)

resumo.write.mode("overwrite").parquet("s3://dados/resumo/")

The technology ecosystem

Hadoop opened the era with distributed storage (HDFS) and the MapReduce model. Spark replaced it in most cases by processing in memory and being far faster. For continuously streaming data, Kafka became the ingestion standard. And modern cloud data warehouses brought massive scale with a familiar SQL interface, a topic covered in depth in what a data warehouse is.

Data lake, warehouse and lakehouse

The data lake stores raw data in its original format, cheap and flexible, but at risk of turning into a "swamp" without governance. The data warehouse holds cleaned, modeled data for analysis, with consistent performance. The lakehouse tries to unite the two, offering transactions and schema on top of the lake's cheap storage. The resulting analytical applications appear in application areas of data science.

## faq

Frequently asked questions

At what size does data become Big Data?

There is no fixed threshold, and be suspicious of anyone who gives you a number. The real criterion is functional: when conventional tools can no longer keep up within acceptable time. In practice, most companies that claim to have Big Data would solve everything with a well-modeled relational database.

What is the difference between Big Data and Data Science?

Big Data is about the infrastructure and techniques for storing and processing large volumes. Data Science is about extracting knowledge from data through statistics and modeling, regardless of size. They are complementary disciplines, not synonyms.

Do I need Hadoop to work with large volumes?

Less and less. Hadoop lost ground to Spark for processing and to cloud object storage in place of HDFS. Today it is more common to find Spark over S3 or a managed data warehouse than a traditional Hadoop cluster.