Requirements
To follow this course, it is required to know following …
•
No Requirements
Big Data
Big data is not defined by one fixed file size. The practical question is whether the volume, arrival rate, or variety of data exceeds what the current tools can process reliably within the required time.
Follow the data from collection to storage, processing, and use. Check its schema, units, timestamps, and quality before adding more machines. Distributing an incorrect calculation only produces incorrect results across more servers.
Batch Processing
A batch job processes a bounded set of data, such as yesterday’s orders. MapReduce divides work into map tasks and groups intermediate values before reduce tasks combine them. Parallel work still has costs: transferring, sorting, and redistributing data can dominate a job. Decide what must be grouped together before distributing the calculation.
Stream Processing
A streaming job processes incoming data as it arrives. Event time and processing time can differ, so late or out-of-order events need an explicit policy. Spark Structured Streaming can process data in micro-batches; streaming does not necessarily mean one event is handled at a time. Recovery also depends on checkpoints and compatible sources and sinks, not just on keeping a process running.
