Kinesis

Posted on Posted in AWS

Kinesis enables to collect and process streams of data records in real time.

What you can you do:

  • accelerated log and data feed intake and processing -ie large amount of applicaiton logs, market data feeds, web clickstream data, social media
  • real-time metrics and reporting
  • real-time data analytics
  • Complex stream processing

High level architecture

Terminologies:

  • Stream – ordered sequence of data
  • Data records – unit of data stored in kinesis (contains sequence number, partition key, data blob) up to 1mb
  • Retention period – configurable in hourly increments from 24 to 168 hours (1 to 7 days), default 24 hrs
  • Producers – those who create the stream
  • Consumers – stream consumers
  • Stream application
  • Shard –  group of data records in a stream
  • partition key – segregates data records
  • sequence number

Leave a Reply